In the News

Big Data's Missing Ingredients

If you've even experimented with building big-data applications or analyses, you're probably acutely aware that the domain has its share of missing ingredients. We've boiled it down to five top wants on the big-data wish list, starting with SQL (or at least SQL-like) analysis options and shortcuts to deployment and advanced analytics and finishing with real-time and network analysis options.

The good news is that people and, in some cases, entire communities, are working on these problems. There are armies of data-management and data-analysis professionals who are familiar with SQL, for example, so organizations naturally want to take advantage of knowledge of that query language to make sense of data in Hadoop clusters and NoSQL databases -- the latter is no paradox, as the "No" in "NoSQL" stands for "not only" SQL. It's not a surprise that every distributor of Apache Hadoop software has proposed, is testing, and has or will soon release an option for SQL or SQL-like analysis of data residing on Hadoop clusters. That group includes Cloudera, EMC, Hortonworks, IBM, MapR and Teradata, among others. In the NoSQL camp, 10gen has improved on the analytics capabilities within MongoDB, and commercial vendor Acunu does the same for Cassandra.

The old days of developers releasing a game -- "put it in a box, pay £50, that's it, move on" -- are over, according to Square Enix's Jim Blackhurst, as gaming moves towards the more "iterative" model pioneered by mobile gaming.

Gaming is becoming a big data problem, as online multiplayer sessions and always-on consoles let developers keep track of players like never before. Square Enix starting doing this in 2007, Jim Blackhurst tells Wired.co.uk, because the company's developers were "interested to see if the decisions we had made as designers were being played out in reality".

By the time Deus Ex: Human Revolution shipped, Square Enix had developed the infrastructure it needed to both absorb all the information that was coming in and to analyse it in a meaningful way. Jelinek said: "Jim was asked to produce some statistics like how many kilometres players drove, and he had to look at all the data we had gathered since the beginning. The kernel ran and it took something like three weeks to run that single query."

Now, that same query can take around two minutes with a MongoDB database hosted on a cloud server. The implications for how Square Enix develops games are intriguing, as it significantly simplifies and speeds up much of a game's development. Something that was initially meant to be a reactive way of tracking player activity became something more proactive, a way of collecting and archiving information on actual gameplay elements. It lets Square Enix's studios create what are essentially templates that its developers know have already been tested and shown to work on other titles.

MongoDB creator 10gen is growing again, this time with the addition of Sydney Carey as the company’s first CFO. She’ll help lay the infrastructure for plans to more than double in size in two years as it eyes an eventual IPO.

10gen, the creator and proprietor of the extremely popular MongoDB NoSQL database, is growing up fast and on Tuesday announced it has hired Sydney Carey as the company’s first-ever chief financial officer. Carey comes from enterprise software company Tibco where she was executive vice president and CFO.

MongoDB 2.4 was recently released with new features such as Text Search, hash-based sharding, better geo-spatial capabilities with GeoJSON support and several performance and tooling improvements. We also discussed with 10gen about what’s next on the roadmap.