Databases and the Cloud

In the early days of DoubleClick, when designing the DART technology, we had to decide on an efficient way to store massive amounts of information, such as number of exposures to a particular advertisement ID.

At first, we stored this information in Oracle, although it was quickly apparent that this was not an ideal long term solution. We eventually would need to store a trillion rows of information with random access to that information.

We built a proprietary, real-time data store for the problem. The system:

• supported the operations get, put, increment
• was very fast and optimized for the problem
• was non-transactional, which was ok for the problem at hand
• was very easy to partition (shard) across multiple machines

Now we are working on the 10gen database, named Mongo. As we work on the design, we keep these issues in mind. For a database for web 2.0 applications today, we want:

• Scalability: a database that easily partitions over any number of commodity-class machines, and grows to any scale.
• Hardware efficiency: it is one thing to scale linearly, another to have a good “constant factor”. We want a system that, in addition to scaling well, supports a high number of requests per second per cpu core, and is fast enough that intermediate caching layers are unnecessary.
• Multimedia capability: we want to store large objects in the DB itself, instead of writing complex code to go find the right thing in an attached file system.
• Ease of programming and depth of functionality.

What are lower priorities for us? We are not optimizing for decision support; rather, we need an operational data store. Also, we are not targeting hardened, banking class transactions and durability – especially if those features come at a large hardware cost.

As we look at the four design goals above, we conclude the ideal technology solution is an object oriented DBMS:

• Scalability: object databases are easier to scale than relational databases; sharding is easier. In a relational database, distributed joins are a complex problem that must be solved if one desires true plug-and-play scalability without limits.
• Hardware efficiency: when used appropriately, object databases can be very fast. All the data for a complex object can be clustered at the same disk location. The data representation on disk can be very close to the program’s in-memory representation, making reads that are file system cache hits fast.
• Multimedia: an object DB can do well storing large, opaque blobs of information.
• Ease of programming: object databases are easy to use when developing software in an object oriented programming language. To maintain portability, existing object persistence interfaces can be mapped to the database.
  

Our approach with Mongo is in some ways similar, and in some ways different, from that of Amazon SimpleDB and Google BigTable. It is similar in that all three are non-relational. It is different in that Mongo is a true object database, rather than a key/value data store.

We also want Mongo to have greater depth of functionality than existing cloud databases. To take a simple example, some of these products have no “order by” feature for queries. We want something that works in the cloud well yet also has the rich functionality developers expect.

An early alpha release of Mongo will be available next week, with a full release targeted for late in the year.