Popular items

 

Onetrail’s choice for MongoDB and Solr

Blogpost by: Eric van Bree
From: Friday 11 April 2014
logo-new-vierkant

Onetrail uses MongoDB for storing every document in every stage of the data interchange process. During this process every log line is stored in MongoDB. Documents and logs are kept for a certain period, due to the tremendous number of records. The documents and logs are accessible for support purposes via the TPN™ Monitor. The Onetrail TPN™ customers have access via the Message Viewer.

MongoDB is a great NoSQL database. However, we have decided to separate the querying to Solr and to use MongoDB only for storage purposes. This reduces the number off indexes to a bare minimum on MongoDB. As a result writes are not slowed down and we have less disk and memory usage.

Moreover, the text index feature would have a large impact on the system due to the slower insertion throughput.

Documents are stored with an Exchange-ID and other meta-data, while logging lines are stored with an uuid and meta data per logging event.

We aggregate the new Mongo entries based on either Exchange-ID or uuid to get relevant data for query purposes and for storage in Solr. Due to the 16Mb JSON limitation we use Map/Reduce to recreate the entire index. This approach enables us to query from the Solr index and to display the results in a split second. We only retrieve the entries from the MongoDB when details are needed. The only disadvantage is that this approach is not real-time. Due to a time-lag this approach is near real-time. However, this delay does not cancel out the advantages.

Finally, the maintainability is improved. Due to the capability to recreate the index on a different Solr instance and when finished switch to a new instance.

« Back to Blogs

Add your reply

Reactieformulier

They chose Onetrail