There is no doubt that SQL databases play an important role in modern applications. In an ideal world, a single database can handle hundreds of incoming connections from multiple clients and scale to accommodate the related transactions. However the world is not ideal and databases are often a cause of major headaches when applications need to scale to accommodate more connections, transactions, or both.
In order to overcome scaling issues, application developers often resort to administrative acrobatics, also known as database sharding. Sharding helps to improve application performance and throughput by splitting the database into two or more shards. Unfortunately, this practice also requires application developers to code transactional consistency into their applications. Getting transactional consistency across multiple SQL database shards can prove to be very difficult.
Sharding requires developers to think about things like rollbacks, constraints, and referential integrity across tables within their applications when these types of concerns are best handled by the database. It also makes other common operations such as joins, searches, and memory management very difficult. In short, the very solution implemented to overcome throughput issues becomes a bottleneck in and of itself.
What if database sharding was no longer required to scale your application? Let me explain. For the past several months I have been following and writing about NuoDB, a hot new SQL database technology out of Cambridge, MA. NuoDB is officially out of beta and they have recently released their first release candidate so I decided to dig into the database in a little more detail. Their architecture is very interesting and exciting because it completely eliminates the need to shard a database to achieve higher throughput.
Each NuoDB database consists of at least three or more processes that enable a single database to run across multiple hosts. These processes include a Broker, a Transaction Engine and a Storage Manager. Brokers are responsible for connecting client applications to Transaction Engines and maintain a global view of the network to keep track of the multiple Transaction Engines available at any time. Transaction Engines are in-memory processes that client applications connect to for processing SQL transactions. Storage Managers are responsible for persisting data to disk and serving up records to the Transaction Managers if they don’t exist in memory.
The secret to NuoDB’s approach to solving the sharding problem is that it is a truly distributed, peer-to-peer, SQL database. Each of its processes can be deployed across multiple hosts. When client applications need to connect to a Transaction Engine, the Broker will automatically route the request to the most available process. Since multiple Transaction Engines and Storage Managers running across multiple host machines represent a single logical database, you never have to resort to sharding to get the throughput your application requires.
NuoDB is a new pioneer in the SQL database world. They are making database scalability simple by eliminating the need for acrobatics such as sharding, and they are also making general administration of the database simpler as well.  Their distributed database appears to you as a user like a single SQL Server database. With their RC1 release they have also provided a web based administrative console that they call NuoConsole. This tool makes it extremely easy to deploy and manage NuoDB processes across one or multiple hosts with the click of a mouse button. See for yourself by downloading NuoDB here.
Reference:Â Pinal Dave (https://blog.sqlauthority.com)
6 Comments. Leave new
Hi Pinal,
Nice article!
Apart from switching to a new technology (NuoDB), is there any better solution within SQL Server World than Database Sharding ?
Pinal,
Great article. Just to make sure that I heard you right ; you mentioned that NuooDB is a peer-to-peer software. Is this the same thing as open source software? Would using the cloud option be an alternative to database sharding, given the almost infinite storage you get from this kind of storage?
Thanks,
Justin
Hi Pinal. I’ve been following NuoDB for about a year and have been impressed with their approach to this problem. I am optimistic about this product and other “NewSQL” such as VoltDB. They offer scale-out performance without throwing data modeling back 20 years, like some of the NoSQL databases do. I’m extremely disappointed that Microsoft and Oracle haven’t already built this into their products.
Justin, cloud storage does not solve the same problems. A cloud server is just a server that lives in a remote data center, right? Cloud storage can certainly be useful but doesn’t, in itself, help a data server scale.
Looks like WordPress lost one of my two comments so here’s a short version. I have followed NuoDB for about a year now and am really excited about what they and some of the other NoSQL vendors are doing – trying to scale of data performance without throwing data modeling back 20 years, like the cool NoSQL databases do. I am also really disappointed that Microsoft and other relational vendors haven’t already built this into their products, as the need has been obvious for a very long time. Building SQL Azure without this kind of functionality was pretty much a big waste of time,in my opinion.Thanks for giving a shout out to NuoDB, Pinal!
Hmm even if this is an old article regarding NuoDB, what if we have too much data right now like over 15gb (but less ram, ~4gb ram). If not sharding is done then how it’s supposed to work with big data ?