10gen Blog

SDK Updates - Stable for Windows, new snapshots

Geir Magnusson — Wed, 27 Aug 2008 12:35:00 EDT

We rolled out a new app server to production yesterday, so we refreshed the stable SDK builds with those versions. We also added a stable binary build for Windows.

Also, there is a fresh set of snapshot builds for Windows and 32-bit linux and as always, the nightlies have the latest binaries for 64-bit linux, OS X as well as source.

See the SDK page for more information.

Windows Binary SDK Available

Geir Magnusson — Mon, 25 Aug 2008 08:08:00 EDT

This weekend we posted our binary SDK for Microsoft Windows. You can find in on the SDK page on the wiki, or for the impatient, just download it directly.

Note that this is a "snapshot" - we'll do it from time to time, and eventually nightly. Also, we don't have the "stable" build yet for Windows for technical reasons related to the lack of real symbolic links in the Windows filesystem, but that will be resolved later this week.

This has been built and tested with Sun's implementation of Java SE 6. It requires that you have the location of jvm.dll in your PATH as MongoDB uses it for it's in-process JVM. To achieve this, simply add

C:\Program Files\Java\jre1.6.0_07\bin\client

(or whatever is appropriate for your installation) to the PATH.

This is our first version of this, and we expect that problems will be found. We've tested [lightly] on WinXP and Vista, but as we wanted to make this available ASAP, we decided to just push it out and keep testing in parallel.

As always, if you find any issues, please contact us on the public list/forum

http://groups.google.com/group/10gen

or any of the other ways to contact us.

Sandboxing

Dwight Merriman — Wed, 13 Aug 2008 10:34:00 EDT

The 10gen runtime environment uses application layer sandboxing to isolate each client environment running on a physical machine. This method is quite different than using operating system virtual machines.

We like this approach as it is more elastic and fine-grained than 'booting' VM's on demand. We want to abstract away the whole concept of a machine. We want users of the cloud to get away from the work of OS configuration, security hardening, and environment scaling.

The foundation of our sandbox is the Java virtual machine (JVM). Much work has been done on the JVM and security. We are trying to leverage that work, and build additional provisioning and quota management layers atop it.

There is a cost of course: you can only execute code which will run within the JVM. We think it is well worth it for the gain in flexibility. If we encounter a special need for an existing software system that will not run on the platform, that program can be ran on a virtual machine hosted somewhere, and we can communicate with it via web services.

A Very Simple Mongo Insert Speed Test

Dwight Merriman — Mon, 11 Aug 2008 15:58:00 EDT

I ran the test below on my macbook pro - both database and app server running on the same machine. Result is about 19k inserts / second.

The collection being tested has one index on the _id field. In general, the more indexes present, the slower inserts will be. One question here is: will the insert rate slow down once the index is too large to fit in RAM? In general, the answer will be yes of course. However, when indexing on object ID, it may not slow down: the ID's have a timestamp component which makes them increase over time, thus all inserts in the b-tree will occur near the right-hand edge. That 'edge' of the tree will always fit in ram. (That said I did not run such a test -- YMMV.)

~/ed ./runLight.bash ed.js.Shell /tmp/test.js 
...
i: 1000000 19666/sec

~/ed cat /tmp/test.js

// very simple speed test
db=connect("test");
t1 = db.collection2;
t1.ensureIndex({_id:ObjId()});
var start = new Date();
var i = 0;
for( var i = 0; i   1000000; i++ ) {
    t1.save( { i:i, z: "abaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
     } ); 
    if( i % 10000 == 0 ) {
        print(i);
    }
}
// make us wait until db has completed all work:
t1.findOne();
var now = new Date();
var persec = i / ((now-start+1)/1000);
print("i: " + i + " " + persec.toFixed() + "/sec");

Template Technologies: Mix them in your app

Geir Magnusson — Tue, 05 Aug 2008 07:26:00 EDT

The 10gen platform was recently written about in the German "Jaxenter" online magazine:

http://it-republik.de/jaxenter/artikel/10gen-Komplette-Plattform-zum-Ablauf-von-Webanwendungen-1871.html

It's a nice writeup (as far as I can tell from the Yahoo! Babelfish! translation!) and kudos to the author, who I think did a good job - he downloaded the SDK and actually took it for a test drive.

One of the things that I appreciated was that he grokked the idea that you can mix template styles in the app (forgive the Babelfish translation...):

"An interesting realization, which one can gain from this simple example, is those that the different Template technologies can be mixed with one another."

He didn't elaborate further, but I think that this is an important aspect of our technology - the appserver supports the use multiple template technologies seamlessly and simultaneously in an application.

If you look at the "samplesite" demo app that comes with the SDK, you'll see the following [trivial] example of this in controller.jxp where the code calls three different templates and passes each data (template1, template2 and template3 in the templates directory of the application) to render into the output stream:

local.templates.template1({ theCount : data.count})
local.templates.template2({countThis : data.count})
local.templates.template3({ countThat : data.count})

Note that the variable name is changed to show that each template is independent.... What you don't need to worry about as a programmer here is that these three templates are actually of different types. "template1" is a djang10 template, "template2" is a jxp, and "template3" is 10gen html.

Briefly, the 10gen appserver supports:

djang10 : this is a re-implementation of the great template system from the Django web application platform. We really like Django, and are re-implementing it as fast as we can, as accurately as we can.
jxp : similar to the deservedly-maligned JSPs, where you mix markup and actual JavaScript in the same document
html : another 10gen-invented template, it's like jxp with added enhancements for in-line data access via $variable_name. It was the first templating system that we did (we wanted to make it easy to mix data into existing HTML) and if you are starting fresh, we suggest that you use djang10.

So why is this important? Well, I'd never advocate that one goes and starts building an application with multiple template styles out of the gate, but software re-use is important in keeping project construction and maintenance costs under control, and supporting multiple template styles as first class citizens for an application helps with this goal. This will make it easer to reuse app components, irrespective of the template style chosen by the author.

Note also that this part of our architecture allows for easy evolution and customization as this system is pluggable - you can easily implement other template styles, plug them into the appserver, and they can be first class templates as well. To see how this works, go look at

ed.appserver.templates.TemplateEngine.java

in the appserver. (Look in the nightly source download). I'm hoping someone contributes a support for my favorite template engine. ;)

Next time, I'll talk about mixed languages - how your app can use JavaScript and Ruby (and Python, even though Eliot doesn't want me to mention it as it's really not ready. But you can try it. Download the SDK...) code at the same time, in the same app.

Problems with JavaScript

Eliot Horowitz — Wed, 30 Jul 2008 11:00:00 EDT

In my previous post I discussed why we chose JavaScript as our first language for 10gen. While I love building applications with server side JavaScript, JavaScript does have its faults. Here are some of the most frustrating to me personally:

Variables should be local by default, and use a modifier to make them global, similar to python. I very often see mistakes where someone is using the same variable in two places and they are overriding each other. My favorite is when you have a for loop "for (i=0;...") in two functions, and it causes an infinite loop.
Lack of normal class inheritance. Prototype based is very powerful, but you still need normal classes also. However this is fixed in ECMAScript 4 which should be coming out in the fall.
Syntactic Sugar Things
- for loop iteration over arrays. "for ( foo : [ 1 , 2 ] ){}" - one of the things I miss from Java. While you can use forEach, there are some problems with that like breaking out of loops.
- terser lambda expressions. "'(x){ return x + 5; }" no need to type function all the time.
Lack of standard libraries - not really a problem with the language, but is a problem. This is one that we are actively addressing. CoreJS is our base library for JavaScript. This includes some web specific code, but also just has nice JavaScript libraries that can be used in 10gen and other places. One of the nice things about building on top of Java is that we can easily expose any of the extensive Jjava libraries. We have made sure the bridging between JavaScript and Java is very easy to use so you can go back and forth easily.

I can't wait till we add more languages so we can start critiquing those too :)

You Don't Need a File System

Dwight Merriman — Mon, 28 Jul 2008 21:48:00 EDT

We've seen quite a bit of innovation on distributed file systems over the past few years: Google GFS, ZFS, HDFS -- and as a file store, Amazon S3.

What's the next step? We would argue it is time that you stop using file systems and put that data in a database instead.

When we began the 10gen project, we wanted a fast, scalable database. The question then became: if it is fast and truly scalable, why do I need a separate filesystem? If we needed one, that would mean we had not achieved our goals in terms of database performance. It's a great litmus test: take a database, and implement a file system atop it.

Does it work? Yes, it is working quite well. GridFS, the 10gen file system interface, is simply a thin layer atop the Mongo db.

We did have to address a few differences in functionality. First, with files, read and write access to arbitrary ranges is common. Generally with a database, it is difficult to query a data range inside an object or row. To solve this, we simply break large files into chunk objects, which we store in the _chunks collection.

A second issue is that files have an implied order to their data. Consider the problem of log files: we want each log event (row in a text file, traditionally) stored in "insertion order". To do this with a database, we might define a timestamp attribute and index on it, but that will be slower of course. Mongo's natural order feature provides a way to access this data in insertion order quickly, without the overhead of an index. 10gen uses this feature to store log data in the database, instead of in files. This enables a system with a global log "file", instead of logs per server.

SSL Plans

Dwight Merriman — Fri, 25 Jul 2008 08:00:00 EDT

There is now a *.10gen.com SSL certificate on the 10gen cloud. For example, try this.

There are two ways to address a site on the 10gen cloud: either via http://mysite.10gen.com/, or if you own your own domain mysite.com, you can CNAME that in DNS to 10gen.

SSL will now work for mysite.10gen.com, but will not work for www.mysite.com. Thus, this is not a complete solution, but it's a start: for example, you could use www.mysite.com for most of your site's pages, and then link over to https://mysite.10gen.com/ to do things like secure checkout.

One interesting detail worth pointing out: the 10gen app server automatically marks up static object URLs to point to a CDN. The app server is smart enough to not perform this markup on secure pages, as that would cause security warnings.

32-bit Linux Binary SDK Available

Geir Magnusson — Thu, 24 Jul 2008 15:04:00 EDT

Greetings from OSCON. (More cloud here than you might have thought... report later, after recovery.)

Some people have been asking for the 32-bit build of the 10gen SDK for Linux - it's avail now. You can always find the download links at our SDK page or directly :

http://www.10gen.com/~~/f/10gen-linux_i386-nightly.tar.gz

It was built on Ubuntu 7 (in a VM, of course) - let us know if there are any problems. See you in the forums.

Why JavaScript?

Eliot Horowitz — Tue, 22 Jul 2008 08:00:00 EDT

One of the interesting and different things about 10gen is that its first supported language is JavaScript. Most of our libraries are in JavaScript, and a lot of the system itself is in JavaScript. So, why did we decide to use JavaScript instead of one of the other popular server side languages (Ruby or Python)?

- One of the big reasons I like using JavaScript on the server is that you already have to use JavaScript on the client. You can now write a class once, use it for the client, server, and even with our database. No having to marshal between python/ruby and JavaScript, and then into sql. Just one representation for all of your objects from front to back.

- JavaScript is dynamically typed, has lambda expressions, and many of the other features that people are getting excited about in the scripting world. The biggest difference between JavaScript and the other languages is that its prototype based. (a good inheritance example) Using prototypes for inheritance is a bit different than people are used to, but is actually quite powerful. JavaScript 2, which should be out this year, will also support more standard classes.

- An assumption about JavaScript is that it is slow. We saw no reason why javascript should be slow, so set out to create a really fast JavaScript system. What we did to solve it was to write a JavaScript to java compiler, which then is able to leverage javac and java's jit engine. Our JavaScript system is about 3-4 times faster than rhino or spider monkey.

While JavaScript is the first language that 10gen supports, we will be adding other languages shortly. Ruby will be available this year, and python should follow that. We're also going to let you include java code in your applications when we get to it or demand is high.

Since JavaScript is only our first language and we will be supporting others, and I wouldn't want you to think that I don't think JavaScript has faults, stay tuned for the next article on problems with JavaScript.

10gen Platform is Open Source

Geir Magnusson — Mon, 21 Jul 2008 09:45:00 EDT

Today appears to be "announcement day" at 10gen, as we have another - the 10gen platform is open source.

The source code for the 10gen platform available under open source licenses can be found on our SDK page. For the impatient :

http://www.10gen.com/~~/f/10gen-src-nightly.tar.gz

I want to emphasize that this is everything - the good. the bad, and yes, the ugly. There are no stubs or missing pieces. (Ok, we're missing a lot of docs and code comments, but that's due to non-existance rather than omission :) We think that cloud computing should be an open environment - open in many ways - and this is the best thing we can do now to move towards that vision.

I'll be at OSCON this week and I'd love to talk about what we're doing, show you the code, or help you get started with our SDK.

The last few weeks have been a real whirlwind - finishing the financing, hiring, getting some documentation together, the endless nips and tucks that you never expect when putting together an SDK or a source release (and I'll sheepishly note there's still more to do)... so in the next few weeks, we'll be working on supporting the first few users of the SDK and code, and building a community around the codebase. I hail from the ASF school of open source, and I think community matters. If you want to participate, see our contact page for a summary of how you can get involved.

10gen Platform SDK Available

Geir Magnusson — Mon, 21 Jul 2008 09:45:00 EDT

Today is a big day for us at 10gen - in addition to the series A financing that we announced earlier this morning, we're also announcing that our SDK is available for download.

Our new 10gen Platform Software Development Kit (aka "SDK") is available for download starting today. The SDK is available as pre-built binaries for OS X and 64-bit linux and contains our appserver, our database and the core Javascript libraries and modules - the full platform. For more information, please see our SDK page on the wiki. For the impatient :

OS X Nightly:
http://www.10gen.com/~~/f/10gen-macosx-nightly.tar.gz

Linux Nightly: http://www.10gen.com/~~/f/10gen-linux_x86_64-nightly.tar.gz

See SDK page for stable releases. There also is documentation.

Please note this is an early preview release, so there are a lot of sharp edges and pointy corners on the code and the documentation. Now, this code *is* running four sites in production, but still - this is for early adopters and not yet ready for mission critical projects.

Speaking for the team, we look forward to your engagement and feedback.

Financing

Dwight Merriman — Mon, 21 Jul 2008 09:30:00 EDT

Today we announced that 10gen has received $1.5 million in Series A financing from Union Square Ventures. We are very excited to work with Albert Wenger and the rest of the Union Square team. Albert is a thought leader in the platform computing space, check out his blog post on the subject from last year.

Databases and the Cloud

Dwight Merriman — Tue, 15 Jul 2008 12:30:00 EDT

In the early days of DoubleClick, when designing the DART technology, we had to decide on an efficient way to store massive amounts of information, such as number of exposures to a particular advertisement ID.

At first, we stored this information in Oracle, although it was quickly apparent that this was not an ideal long term solution. We eventually would need to store a trillion rows of information with random access to that information.

We built a proprietary, real-time data store for the problem. The system:

supported the operations get, put, increment
was very fast and optimized for the problem
was non-transactional, which was ok for the problem at hand
was very easy to partition (shard) across multiple machines

Now we are working on the 10gen database, named Mongo. As we work on the design, we keep these issues in mind. For a database for web 2.0 applications today, we want:

Scalability: a database that easily partitions over any number of commodity-class machines, and grows to any scale.
Hardware efficiency: it is one thing to scale linearly, another to have a good 'constant factor'. We want a system that, in addition to scaling well, supports a high number of requests per second per cpu core, and is fast enough that intermediate caching layers are unnecessary.
Multimedia capability: we want to store large objects in the DB itself, instead of writing complex code to go find the right thing in an attached file system.
Ease of programming and depth of functionality.

What are lower priorities for us? We are not optimizing for decision support; rather, we need an operational data store. Also, we are not targeting hardened, banking class transactions and durability - especially if those features come at a large hardware cost.

As we look at the four design goals above, we conclude the ideal technology solution is an object oriented DBMS:

Scalability: object databases are easier to scale than relational databases; sharding is easier. In a relational database, distributed joins are a complex problem that must be solved if one desires true plug-and-play scalability without limits.
Hardware efficiency: when used appropriately, object databases can be very fast. All the data for a complex object can be clustered at the same disk location. The data representation on disk can be very close to the program's in-memory representation, making reads that are file system cache hits fast.
Multimedia: an object DB can do well storing large, opaque blobs of information.
Ease of programming: object databases are easy to use when developing software in an object oriented programming language. To maintain portability, existing object persistence interfaces can be mapped to the database.

Our approach with Mongo is in some ways similar, and in some ways different, from that of Amazon SimpleDB and Google BigTable. It is similar in that all three are non-relational. It is different in that Mongo is a true object database, rather than a key/value data store.

We also want Mongo to have greater depth of functionality than existing cloud databases. To take a simple example, some of these products have no 'order by' feature for queries. We want something that works in the cloud well yet also has the rich functionality developers expect.

An early alpha release of Mongo will be available next week, with a full release targeted for late in the year.

Zeus Load Balancer is a Good Product

Dwight Merriman — Tue, 08 Jul 2008 12:27:00 EDT

Zeus ZXTM is a very good layer 7 software load balancer. It has great performance and great features, including ability to cache content (with an optional module). It has only one downside: it is expensive. If you can afford it, highly recommended.

One advantage of platform-as-a-service is cost savings on things like these: even though a ZXTM instance can push a gigabit per second or more of bandwidth, small sites need less than that, yet still have the cost of a full license. Platform computing solutions provide a way to effectively buy fractional licenses for technologies: whether a load balancer, a database license, or something else.

Platform vs. Cloud

Dwight Merriman — Tue, 24 Jun 2008 09:00:00 EDT

We are beginning to see a segmentation within the cloud computing space. I would consider Amazon's web services to be the "traditional" form -- point web services which solve specific problems. However, a new segment has developed in the cloud -- platform computing.

With the traditional cloud model, physical provisioning of hardware goes away. However, the CTO and staff must still design a system architecture, even if running on EC2 servers. Many decisions must be made: what database, what load balancer, what app server. memcached? Apache pool size? System administration work has not disappeared.

On a platform, this work goes away -- someone has already done this work. No more system administration. No more system architecture decisions. The benefits are clear -- huge savings in effort in these areas. The disadvantage: you must buy into the platform as it exists.

As an investor in many startups over the last few years, I have watched the CTOs solving the same problems -- over and over again! With today's methods, we are reinventing the wheel every time we set up a best-in-class system infrastructure for a complex web site. This is silly. This work should be done once, done well, and reused.

That's what platform computing does.

So what is a platform? AWS is not a platform. It's a set of tools to build your own infrastructure. Google App Engine is a platform. Many startups are building platforms too -- some upon EC2.

We are convinced in the future this will be the default model for web site development.

Welcome

Dwight Merriman — Tue, 17 Jun 2008 08:20:00 EDT

Welcome to the 10gen blog. Here you'll find our views and observations on cloud computing, commentary on the 10gen system itself, and interesting things about our company. Stay tuned - our alpha begins in July. To see a real site running 10gen today, please check out Silicon Alley Insider.

OS X Nightly:	http://www.10gen.com/~~/f/10gen-macosx-nightly.tar.gz
Linux Nightly:	http://www.10gen.com/~~/f/10gen-linux_x86_64-nightly.tar.gz