Yesterday I mentioned on Twitter that I was playing with the MongoDB pub\/sub features and that it worked quite well for my needs.<\/p>\n
What I didn’t mention was that the documentation and blog posts were a bit all over the show and the Ruby examples I saw didn’t actually do what they said they did so I’ll show in this post working code and some basic approaches I took to deal with per consumer destinations etc.<\/p>\n
Actually MongoDB is a really nice database but like most NoSQL databases the thing to know about it is what shortcuts it takes with your data to do it’s magic. Knowing this you have to evaluate its suitability to your specific problem and if it’s not suitable, don’t use it.<\/p>\n
It’s fast and has a flexible query system to search over arbitrary structured JSON data. Yes it has some interesting ideas about data durability but this is well known by now and if your needs match it’s features it’s not bad.<\/p>\n
For shops with needs well suited to MongoDB who might want to add some queueing ability it can be daunting to bring in new tech like RabbitMQ or ActiveMQ because it brings new unknowns requires an investment in more monitoring, training and learning by making mistakes. If you already have a Mongo instance and know its quirks using it for a queue might not be such a terrible thing.<\/p>\n
Additionally MongoDB is really easy to get going and generally I find for my work loads it just works with little maintenance required.<\/p>\n
So my interest in its queueing abilities lies in providing a simpler ‘getting started’ for MCollective. New MCollective has pluggable discovery which works really well when discovering against a MongoDB cache of registration data so it would be nice if a simple starter edition setup could include both the queue and discovery data in one simple bit of software.<\/p>\n
There are other options of course like Redis and I’ll evaluate them but of the various options MongoDB is the only one that comes with both pubsub and searching\/querying capabilities that does what I need, isn’t written in Java and has OS packages for most distros easily available.<\/p>\n
The combination of some of these Cursor flags and Capped Collections enables a kind of tail -f<\/em> behavior that works like a queue.<\/p>\n When you have a collection it usually returns nil when you reached the end of your results as can be seen here:<\/p>\n <\/code><\/p>\n Here we opened a collections and did a find. We moved to the end of the results and fetched the next result which immediately returned a nil<\/em> indicating there’s nothing new.<\/p>\n Lets see how we can change the behavior of this collection that instead of returning immediately it will block for a while waiting for a new document and then return a nil after after a timeout if nothing new was found:<\/p>\n <\/code><\/p>\n Now instead of immediately returning a nil it will wait 2 to 3 seconds at the end of the collection incase new data comes.<\/p>\n<\/p>\n\r\n>> coll = db.collection('commands')\r\n=> Mongo::DB:0x7fa1ae005f58 ....>\r\n>> cursor = coll.find()\r\n=> Mongo::Cursor:0x3fd0d6f61184 ....>\r\n>> cursor.skip(cursor.count)\r\n=> Mongo::Cursor:0x3fd0d6f61184 ....>\r\n>> cursor.next_document\r\n=> nil\r\n<\/pre>\n<\/p>\n\r\n>> cursor = coll.find()\r\n=> Mongo::Cursor:0x3fd0d6f61184 ....>\r\n>> cursor.skip(cursor.count)\r\n=> Mongo::Cursor:0x3fd0d6f61184 ....>\r\n>> cursor.add_option(Mongo::Cursor::OP_QUERY_TAILABLE)\r\n=> 2\r\n>> cursor.add_option(Mongo::Cursor::OP_QUERY_AWAIT_DATA)\r\n=> 34\r\n>> loop { puts \"#{Time.now}> Tailing....\"; p cursor.next_document }\r\nFri Aug 31 13:40:19 +0100 2012> Tailing....\r\nnil\r\nFri Aug 31 13:40:21 +0100 2012> Tailing....\r\nnil\r\nFri Aug 31 13:40:23 +0100 2012> Tailing....\r\nnil\r\n<\/pre>\n