Michael Calabrese is a senior developer with Lunar Logic, a custom software development firm that specializes in custom websites and web-based technologies for businesses, educational institutions, and non-profit organizations. He is also, as noted below, a convert from the relational database world.
I have a confession: I have crossed over. It has finally happenedÃ¢â‚¬â€ù I no longer like to use SQL on products. The joy of simplifying needlessly complex queries is gone. I no longer see problems as tables broken down in third normal form. I am now one of them. I have accepted the world of document databases and, in particular, that of MongoDB.
Growing Up SQL
Like many developers that finished college in the 90s (or before), I grew up with relational databases. I created structures, followed the rules, and removed duplication. I was careful to implement foreign keys and set domains on my data. Data integrity was of the utmost importance.
My databases blossomed with tables, adding order types, product inventory types, and inventory locations. Tables grew like flowers, popping up with each new type of data that I had to track. I would feel the rush of building massive SQL queries to mine the data to build reports, adding join after join.
Cracks in the SQL Faith
But then reality started to set in. It became: ...Just add another table, another join, and it will work out.” The user experience dragged as data grew. We went from thousands of rows to millions. As data requirements changed at management’s whim, domains on the data would not hold.
As the data kept growing and becoming more complex, I had data requirements that didn’t necessarily match historical data. I could no longer depend on the fast inner join. I suffered from table creep and now I needed transactions to manage the updates across tables. Tables grew needlessly, with near duplicate products that only changed names.
I was increasingly building meta systems, like a CMS where I would not know shape of the data being stored. Choices to model the data became complex and unwieldy. More and more, data processing was moving into the application code or stored procedures. It was now standard procedure to break the rules of SQL where the rules are paramount.
The NoSQL Solution
About this time I changed employers, and was introduced to NoSQL. In the beginning, I was confused by the simplicity of just storing a document. What is a document anyway? Over time I came to realize documents are very similar to the serialized objects in my settings tables. That is, an arbitrary structure that can be stored, but still accessed simply, as part of a row/document. I could easily create and access multi-part fields, without having to mess with any sort of serialization.
This was revolutionary. It could solve many of the problems in which I needed arbitrary data structures. In the new meta systems, I could easily store user-created data structures. On top of all that, I could even create indexes for the sub-fields/sub-documents.
From there, document databases and NoSQL just started to make sense. My designs became cleaner. De-normalization became the standard. I was designing schemas for how the data was going to be used, but realized that I was doing that anyway in SQL. With the MongoDB document database, how the data was used became more important than the decomposition of the data.
All necessary information could be stored in a single document, rather than across tables and joins. All similar documents could be housed in a collection, so, for example, an order document included all of the relevant order information: customer information, order detail, shipping. Everything is neatly packaged, to be sent to my applications without much processing.
Living in a Post-SQL World
I thought I would miss transactions, but I don’t. Generally, transactions are needed because in SQL we were forced to divide data between tables to remove duplication. We had to keep data consistent between those tables. Now, I just keep data that must be kept consistent in the same document. This is better anyway, as in most cases, the data that must be consistent is part of a conceptual whole.
My designs are now cleaner and more flexible. This flexibility inspired cleaner, flexible code. All-in-all, I am more productive in design, development and maintenance of my projects. After a year of developing in MongoDB, I no longer look back.