Jesus Diaz, Infinithread Corp
February 7 2013
Gyes is an aggregator platform for the Web. Gyes allows you to develop, schedule and troubleshoot data extraction programs (crawlers) that translate html content into structured data you can use later on. In selecting the data model for the platform, several challenges arose due to the lack of structure of the scrapped data, and the need to provide meaninful and efficient access to it. MongoDB was our third rewrite of the Gyes back-end, and by far has exceeded expectations. In this talk, I would like to discuss some of the challenges we faced, and how MongoDB addressed them.