{"id":2640,"date":"2012-06-19T20:42:55","date_gmt":"2012-06-19T19:42:55","guid":{"rendered":"http:\/\/www.devco.net\/?p=2640"},"modified":"2012-06-19T20:55:00","modified_gmt":"2012-06-19T19:55:00","slug":"mcollective-direct-addressing-mode","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2012\/06\/19\/mcollective-direct-addressing-mode.php","title":{"rendered":"MCollective Direct Addressing Mode"},"content":{"rendered":"<p>As mentioned in my <a href=\"http:\/\/www.devco.net\/archives\/2012\/06\/16\/a-look-at-mcollective-2-0-0-and-beyond.php\">first post in this series<\/a> I will be covering new MCollective features that were introduced with version 2.0.0 or later.  Today I&#8217;ll talk about the biggest new feature called <em>Direct Addressing<\/em>.<\/p>\n<p><H3>The Past &#8211; Broadcast Only Mode<\/H3><br \/>\nIn the past MCollective only had one mode of communication.  It would send a broadcast message to all nodes with the target agent in a named group (subcollective) and this message would have a filter attached that nodes will validate to determine if they should run the action.  Basically if I send a message with a filter &#8220;country=uk&#8221; all machines will get it and validate this filter, the ones that match will act on the message.<\/p>\n<p>This mode is the first problem I set out to solve &#8211; a way to have a broadcast based zero config RPC system that can address many machines in parallel with a new style of addressing. I wanted to get the broadcast model right first and I wanted to get the RPC structures right as well before looking at other possibilities.<\/p>\n<p>There were many 1:1 RPC systems before and it&#8217;s not a very hard problem to solve if you have a queue &#8211; but it was not the problem I set out to solve as my first target. MCollective 1.2.1 and older did not have a 1:1 mode.<\/p>\n<p>The parallel mode works fine in many scenarios, specifically this is the only real way to build a central coordinator that degrades well in split brain scenarios since addressing is done by discovery and only discovered nodes are expected to reply.  It&#8217;s a new paradigm &#8211; one thats better suited for distributed applications since failure is inevitable you may as well code your applications to always work in that environment. <\/p>\n<p>I think MCollective solved that problem well in the past but the very nature of that mode of communication means it is not suitable for all use cases. The unsuitable usage include some of the points below but there are of course others:<\/p>\n<ul>\n<li>If you want to deploy to a pre-determined set of hosts you really want to be sure they get your request and get warnings if they dont<\/li>\n<li>The broadcast mode is very fast and parallel, you might want to do rolling restarts<\/li>\n<li>The broadcast only operates with a concept of <em>now<\/em>, you might know a machine is being provisioned and soon as its up you want it to run your command that you sent 10 minutes ago<\/li>\n<li>Your discovery needs might not map onto what MCollective support, like when you present users with a host list they can pick arbitrary hosts from<\/li>\n<\/ul>\n<p>There are many similar problems that was awkward to fit into the MCollective model in the past, all related to either trying to live outside its idea of addressing or about slowing it down to a pace suitable for rolling changes.<\/p>\n<p><H3>Messaging Rewrite<\/H3><br \/>\nAs of 2.0.0 we now have a fairly large rewrite of the messaging subsystem to be more generic and extendable but it also introduce a new mode of addressing that allows you to provide the host list in any way you want.  Rather than doing a broadcast for these requests it will communicate only with the specified nodes.<\/p>\n<p>The history of MCollective is that it used to be a in-house unpublished project that was made pluggable and open sourced.  The initial release did a OK job of it but the messaging had a bunch of badly coupled decisions all over the code base that was a legacy left over.  In 2.0.0 we&#8217;re rewritten all of this and abstracted all the actual communication with the middleware away from MCollective core.  This made it much easier to change how we approach messaging.<\/p>\n<p>Armed with the 2nd mode of communication we were able to apply the very same existing RPC system to a second more traditional style of addressing and we&#8217;re able to mix and match freely between these modes when appropriate. In 2.0.0 this is all kind of under the covers and accessible to the API only but in the new development series &#8211; 2.1.x &#8211; there has already been a bunch of new user facing features added thanks to this new mode.<\/p>\n<p>Best is to show some code, here&#8217;s a traditional discovery based approach to running and action against some machines.  The example will just restart apache on some machines:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nc = rpcclient(\"service\")\r\n\r\nc.fact_filter \"country=uk\"\r\n\r\nprintrpc c.restart(:service => \"httpd\")\r\n<\/pre>\n<p><\/code><\/p>\n<p>This code sets the <em>fact_filter<\/em> which it will get via discovery and then communicates with those hosts.  You&#8217;re basically here at the mercy of the network and current status of those machines for which ones will be affected. <\/p>\n<p>But what if you had a list of hosts that you know you wanted to target like you would if you&#8217;re doing a specific deployment task? You&#8217;d have had to do something like:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nc.identity_filter \/^host1|host2|host3|host4$\/\r\n<\/pre>\n<p><\/code><\/p>\n<p>Instead of the fact filter, not ideal! It would still be doing a discover and should host4 not be around it will not really tell you it can&#8217;t talk to 4.  All it knows is there&#8217;s a regex to match.<\/p>\n<p>Now since MCollective 2.0.0 the situation is hugely improved, here&#8217;s the client again this time supplying custom discovery data:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nc = rpcclient(\"service\")\r\n\r\nc.discover :nodes => File.readline(\"hosts.txt\").map {|i| i.chomp}\r\n\r\nprintrpc c.restart(:service => \"httpd\")\r\n\r\nunless c.stats.noresponsefrom.empty?\r\n   STDERR.puts \"WARNING: No responses from hosts: %s\" % [c.stats.noresponsefrom.join(\", \")]\r\nend\r\n<\/pre>\n<p><\/code><\/p>\n<p>In this example I am reading a text file called <em>hosts.txt<\/em> that should have 1 hostname per line and passing that into the <em>discover<\/em> method.  This switches the MCollective client into Direct Addressing mode and it will attempt to communicate with just the hosts you provided in the host list.<\/p>\n<p>Communication is still via the Message Broker even in direct mode but under the covers this is built using queues.<\/p>\n<p>Now if any of those hosts aren&#8217;t responding in time you will get an actual useful error message that you could handle in your code in whatever way you wish.<\/p>\n<p>Also note that we were not compelled to construct a filter that would match every host like in the past, just giving the list of identities were enough.<\/p>\n<p>This is a lot more suitable for the purpose of building deployment tools or web applications where you might have arbitrary hosts. This also demonstrates that you are effectively doing discovery against a text file and can easily be adapted to communicate with a database or any data you might have on the client side.<\/p>\n<p><H3>Other Possibilities<\/H3><br \/>\nThis mode opens up a whole bunch of possibilities and I&#8217;ll run through a few here &#8211; and there will be follow up posts covering some of these in more detail:<\/p>\n<p><H4>Command Chaining<\/H4><br \/>\nYou can now chain RPC requests via Unix pipes:<\/p>\n<p><code><\/p>\n<pre lang=\"console\">\r\n% mco rpc package status package=foo -j | jgrep data.version=1.2.3 | mco rpc puppetd runonce\r\n<\/pre>\n<p><\/code><\/p>\n<p>This fetches the package version of the <em>foo<\/em> package, filters out only the nodes where the version is 1.2.3 and then does a Puppet run on those nodes.  The Puppet run is using the filtered result set from the first command as a source of discovery information so you do not need to supply any filters or anything like that.<\/p>\n<p><H4>Batched Requests<\/H4><br \/>\nTo avoid affecting all discovered nodes at the same time you can now do things in smaller batches, carving up the total discovered nodes into smaller chunks:<\/p>\n<p><code><\/p>\n<pre lang=\"console\">\r\n% mco rpc service restart service=httpd --batch=2 --batch-sleep=60\r\n<\/pre>\n<p><\/code><\/p>\n<p>This will look on the surface the exact same as before, progress bar and all, but it will progress in groups of 2 and sleep a minute between each group.  It will still be traditional discovery (Unless you use -I), the results will look the same, everything will be the same except it will just affect 2 machines at a time.<\/p>\n<p>You can ^C at any time to bail out and only the batches up to that point will be affected.<\/p>\n<p><H4>Traffic Optimization<\/H4><br \/>\nIf you have a 1000 nodes and you often just communicate with a small amount &#8211; say 10 &#8211; the broadcast mode is not very efficient, the middleware will shunt messages to all 1000 all the time.<\/p>\n<p>Since 2.0.0 the client will switch to Direct Addressing mode if it determines you are communicating with <= 10 hosts, you can configure this threshold to your own needs.\n\nThis has the effect that discovery gets done old style unless you supply a list as above and then it will only communicate with the 10 nodes or less directly thus saving a lot of effort for the middleware layer.\n\n<H4>No discovery for identity only filters<\/H4><br \/>\nIf you are only using the -I option and not supplying regular expressions MCollective will now switch to direct addressing mode and just assume you know what you&#8217;re doing.<\/p>\n<p><code><\/p>\n<pre lang=\"console\">\r\n% mco rpc rpcutil ping -I devco.net\r\n\r\n * [============================================================> ] 1 \/ 1\r\n\r\ndevco.net\r\n   Timestamp: 1340117924\r\n\r\nFinished processing 1 \/ 1 hosts in 62.17 ms\r\n<\/pre>\n<p><\/code><\/p>\n<p>Note there&#8217;s no indication that it&#8217;s doing any discovery &#8211; that&#8217;s because it completely bypassed that step, you can specify multiple <em>-I<\/em> arguments to go out to many machines.<\/p>\n<p>The total runtime here will be very close to 70ms vs the old 2 seconds for discovery and 70ms for the main request.<\/p>\n<p><H4>Pluggable Discovery<\/H4><br \/>\nNow that we can get host lists from anywhere we&#8217;ve made discovery completely pluggable allowing you to decide what is your preferred source of truth that suits your needs best.<\/p>\n<p>Here are some examples:<\/p>\n<p>Query a flatfile:<br \/>\n<code><\/p>\n<pre lang=\"console\">\r\n% mco rpc service restart service=httpd --nodes=hosts.txt\r\n<\/pre>\n<p><\/code><\/p>\n<p>Discover using data kept in PuppetDB:<br \/>\n<code><\/p>\n<pre lang=\"console\">\r\n% mco rpc service restart service=httpd -W country=uk --dm=puppetdb\r\n<\/pre>\n<p><\/code><\/p>\n<p>There are plugins for flatfiles, PuppetDB, MongoDB as built by registration, old style broadcast mode (the default) and more will be written like a recent one by a community member <a href=\"http:\/\/alcy.github.com\/2012\/06\/13\/elasticsearch-discovery-plugin-for-mcollective\/\">that queries Elastic Search<\/a>. Imagine discovering against Zookeeper, Chef Server, Noah or any other data source you can imagine.  Direct Addressing makes all of that possible.<\/p>\n<p>This is a MCollective 2.1.x feature only at the moment so still maturing in the development series.<\/p>\n<p><H4>Message TTLs<\/H4><br \/>\nDirectly addressed messages are implemented using queues, this means they can linger on the network when no-one is there to consume them. Using this feature we can send RPC requests to nodes that do not exist yet &#8211; time band that request and should the node become active during the TTL they will act on that request:<\/p>\n<p><code><\/p>\n<pre lang=\"console\">\r\n% mco rpc service restart service=httpd --ttl 1000 -I some.node\r\n<\/pre>\n<p><\/code><\/p>\n<p>This request will go sit in the queue for <em>some.node<\/em> and if that machine boots up in the next 1000 seconds it will perform the request.  The TTLs are low by default and it does mean your clocks need to be synced.  RPC to non existing hosts though, quite interesting bootstrapping possibilities.<\/p>\n<p><H4>Entirely new styles of application built ontop of MCollective<\/H4><br \/>\nNow that the MCollective messaging has been abstracted out of core it would be fairly easy to apply MCollective to non request\/response style systems.  We can use MCollective libraries to just transport arbitrary data between 2 processes.  This will be done using the security, serialization and connector plugins meaning that you can write generic code and just reuse my libraries to have pluggable security and network capabilities.<\/p>\n<p>The system now supports sending requests and reading the replies elsewhere.  For a web based system this would allow a 100% async model.  You could send your request from the web page and have the replies spool into a temporary storage like a NoSQL database where you show the result sets using paging and traditional web approaches. This combined with arbitrary discovery sources means an entirely new kind of web application can be built using MCollective RPC that&#8217;s very fast, responsive and feature rich<\/p>\n<p><H3>Conclusion<\/H3><br \/>\nThat&#8217;s a quick intro to the new messaging rewrite covering just a few areas it&#8217;s already improved.  I&#8217;ll follow up with more in depth blog posts about some of the items mentioned above.<\/p>\n<p>Having this mode doesn&#8217;t deprecate or invalidate the older broadcast mode, I still believe this is the right approach for zero config systems, still believe its method of degrading is the right way to build a certain kind of application and MCollective will remain suitable to those kinds of application.  It will remain the default mode for a freshly installed MCollective client.  The new mode enhances the existing capabilities.<\/p>\n<p>A side effect of all of this rewriting is that the connectivity plugin is now in full control of how this is implemented paving the way for wider middleware support in the future.  At the moment the only viable way to use this feature is to <a href=\"http:\/\/docs.puppetlabs.com\/mcollective\/reference\/plugins\/connector_activemq.html\">use ActiveMQ<\/a> but we&#8217;ll add more plugins in the future.<\/p>\n<p>Given the above statement Direct Addressing isn&#8217;t yet enabled by default but expect that to change in the next major release.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As mentioned in my first post in this series I will be covering new MCollective features that were introduced with version 2.0.0 or later. Today I&#8217;ll talk about the biggest new feature called Direct Addressing. The Past &#8211; Broadcast Only Mode In the past MCollective only had one mode of communication. It would send a [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[7,1],"tags":[121,85,78,106,13],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2640"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=2640"}],"version-history":[{"count":42,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2640\/revisions"}],"predecessor-version":[{"id":2682,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2640\/revisions\/2682"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=2640"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=2640"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=2640"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}