JUL
You Don't Need a File System
We've seen quite a bit of innovation on distributed file systems over the past few years: Google GFS, ZFS, HDFS -- and as a file store, Amazon S3.
What's the next step? We would argue it is time that you stop using file systems and put that data in a database instead.
When we began the 10gen project, we wanted a fast, scalable database. The question then became: if it is fast and truly scalable, why do I need a separate filesystem? If we needed one, that would mean we had not achieved our goals in terms of database performance. It's a great litmus test: take a database, and implement a file system atop it.
Does it work? Yes, it is working quite well. GridFS, the 10gen file system interface, is simply a thin layer atop the Mongo db.
We did have to address a few differences in functionality. First, with files, read and write access to arbitrary ranges is common. Generally with a database, it is difficult to query a data range inside an object or row. To solve this, we simply break large files into chunk objects, which we store in the _chunks collection.
A second issue is that files have an implied order to their data. Consider the problem of log files: we want each log event (row in a text file, traditionally) stored in "insertion order". To do this with a database, we might define a timestamp attribute and index on it, but that will be slower of course. Mongo's natural order feature provides a way to access this data in insertion order quickly, without the overhead of an index. 10gen uses this feature to store log data in the database, instead of in files. This enables a system with a global log "file", instead of logs per server.
Comments
Tony Roche
Wow, the idea of using the DB as a filesystem totally rocks. I mean, if I hadnt used a System/38 or currently run a warehouse full of AS400's, Id be all over it. I guess its not a warehouse anymore, its a 'cloud'.
Wake me up when you have something new and exciting that hasn't been done before.
Max
But that's basically just writing another filesystem. That's what a filesystem IS. So yes, you DO need a filesystem, you just decided the usual ones weren't for you.
dwight
Was trying to say that there is a lot of redundancy to the traditional approach, where one has separate implementations of file system and database: it would be cleaner to have just one, and given database has more functionality, if there is to be just one, it needs to be a database.
People write a lot of code, at times, to manipulate segments of files sitting on a filesystem: with a db, some of that work would go away.
Casper Bang
Semantics... it's obvious that we are going to need (and eventually get) non-hierarchical file system which works by tags and reminds much more of a database system than file system. We've seen this from wiki's, blogs and other recent information heavy applications.
The problem is that of legacy. We can dream up an arbitrary list of wonderful features but when it comes down to it, if the programs we use are based on having a file dialog where you pick out a single path then... we have something of a chicken and egg problem.
I believe this was the major reason why WinFS did not make it. The technology is in place (in Yukon). I would look at Apple for the first to really start evolving from the current ancient hierarchical file systems.
DAR
DB's are for structured data, while file systems have no inherent structure to their data. Yes, you can implement one on the other, but they're 2 different animals with different needs, and different characteristics - and which therefore require different optimization goals and strategies to make them efficient. So yes, you can implement one on the other, but it'll probably be rather slow.
Joshua
All filesystem metadata is structured data. A system of inodes pointing to datablocks is a system of structured data.
It just doesn't index the content by default. It indexes the things the OS needs to use to find a particular file.
There are implementations of filesystems on top of databases (the MySQLfs program for FUSE kernel module comes to mind).
The previous poster was correct. A filesystem is just a specialized database. It could certainly be extended to include indices of the content by trading either time or space (time in that it might index on the fly when looking for an attribute, or space in that the index would be pre-computed at the time of insert or update).
It would be more interesting to have the FS be something other than a strict hierarchy (collections of files based on user preference/activity/intent).