mcollective.reply.dev1.example.net.999999.0<\/em>.<\/li>\n<\/ul>\nThe data is completely unstructured as far as this message is concerned it just needs to be some text, so base64 encoded is common. All the transport care for is getting this data to its destination with metadata attached, it does not care what’s in the data.<\/p>\n
The reply to this message is almost identical:<\/p>\n
\r\n{\r\n \"data\" => \"... any text ...\",\r\n \"headers\" => {\r\n \"mc_sender\" => \"dev2.example.net\",\r\n \"seen-by\" => [\"dev1.example.net\", \"nats1.example.net\", \"dev2.example.net\", \"nats2.example.net\"],\r\n }\r\n}\r\n<\/pre>\nThis reply will travel via mcollective.reply.dev1.example.net.999999.0<\/em>, we know that the node dev2.example.net<\/em> is connected to nats2.example.net<\/em>.<\/p>\nWe can create a full traceroute like output with this which would show dev1.example.net -> nats1.example.net -> nats2.example.net -> dev2.example.net<\/em><\/p>\nFederated Messages<\/H4>
\nFederation is possible because MCollective will just store whatever Headers are in the message and put them back on the way out in any new replies. Given this we can embed all the federation metadata and this metadata travels along with each individual message – so the Federation Brokers can be entirely stateless, all the needed state lives with the messages.<\/p>\n
With Federation Brokers being clusters this means your message request might flow over a cluster member a<\/em> but the reply can come via b<\/em> – and if it’s a stream of replies they will be load balanced by the members. The Federation Broker Instances do not need something like Consul or shared store since all the data needed is in the messages.<\/p>\nLets look at the same Request as earlier if the client was configured to belong to a Federation with a network called production<\/em> as one of its members. It’s identical to before except the federation<\/em> structure was added:<\/p>\n\r\n{\r\n \"data\" => \"... any text ...\",\r\n \"headers\" => {\r\n \"mc_sender\" => \"dev1.example.net\",\r\n \"seen-by\" => [\"dev1.example.net\", \"nats1.fed.example.net\"],\r\n \"reply-to\" => \"mcollective.reply.dev1.example.net.999999.0\",\r\n \"federation\" => {\r\n \"req\" => \"68b329da9893e34099c7d8ad5cb9c940\",\r\n \"target\" => [\"mcollective.broadcast.agent.discovery\"]\r\n }\r\n }\r\n}\r\n<\/pre>\n\n- it’s is a discovery<\/em> request within the sub collective mcollective<\/em> and would be published via a Federation Broker Cluster called production<\/em> via NATS choria.federation.production.federation<\/em>.<\/li>\n
- it is sent from a machine identifying as dev1.example.net<\/em><\/li>\n
- it’s traveled via a NATS broker called nats1.fed.example.net<\/em>.<\/li>\n
- responses to this message needs to travel via NATS using the target mcollective.reply.dev1.example.net.999999.0<\/em>.<\/li>\n
- it’s federated and the client wants the Federation Broker to deliver it to it’s connected Member Collective on mcollective.broadcast.agent.discovery<\/em><\/li>\n<\/ul>\n
The Federation Broker receives this and creates a new message that it publishes on it’s Member Collective:<\/p>\n
\r\n{\r\n \"data\" => \"... any text ...\",\r\n \"headers\" => {\r\n \"mc_sender\" => \"dev1.example.net\",\r\n \"seen-by\" => [\r\n \"dev1.example.net\",\r\n \"nats1.fed.example.net\", \r\n \"nats2.fed.example.net\", \r\n \"fedbroker_production_a\",\r\n \"nats1.prod.example.net\"\r\n ],\r\n \"reply-to\" => \"choria.federation.production.collective\",\r\n \"federation\" => {\r\n \"req\" => \"68b329da9893e34099c7d8ad5cb9c940\",\r\n \"reply-to\" => \"mcollective.reply.dev1.example.net.999999.0\"\r\n }\r\n }\r\n}\r\n<\/pre>\nThis is the same message as above, the Federation Broker recorded itself and it’s connected NATS server and produced a message, but in this message it intercepts the replies and tell the nodes to send them to choria.federation.production.collective<\/em> and it records the original reply destination in the federation<\/em> header.<\/p>\nA node that replies produce a reply, again this is very similar to the earlier reply except the federation<\/em> header is coming back exactly as it was sent:<\/p>\n\r\n{\r\n \"data\" => \"... any text ...\",\r\n \"headers\" => {\r\n \"mc_sender\" => \"dev2.example.net\",\r\n \"seen-by\" => [\r\n \"dev1.example.net\",\r\n \"nats1.fed.example.net\", \r\n \"nats2.fed.example.net\", \r\n \"fedbroker_production_a\", \r\n \"nats1.prod.example.net\",\r\n \"dev2.example.net\",\r\n \"nats2.prod.example.net\"\r\n ],\r\n \"federation\" => {\r\n \"req\" => \"68b329da9893e34099c7d8ad5cb9c940\",\r\n \"reply-to\" => \"mcollective.reply.dev1.example.net.999999.0\"\r\n }\r\n }\r\n}\r\n<\/pre>\nWe know this node was connected to nats1.prod.example.net<\/em> and you can see the Federation Broker would know how to publish this to the client – the reply-to<\/em> is exactly what the Client initially requested, so it creates:<\/p>\n\r\n{\r\n \"data\" => \"... any text ...\",\r\n \"headers\" => {\r\n \"mc_sender\" => \"dev2.example.net\",\r\n \"seen-by\" => [\r\n \"dev1.example.net\",\r\n \"nats1.fed.example.net\", \r\n \"nats2.fed.example.net\", \r\n \"fedbroker_production_a\", \r\n \"nats1.prod.example.net\",\r\n \"dev2.example.net\",\r\n \"nats2.prod.example.net\",\r\n \"nats3.prod.example.net\",\r\n \"fedbroker_production_b\",\r\n \"nats3.fed.example.net\"\r\n ],\r\n }\r\n}\r\n<\/pre>\nWhich gets published to mcollective.reply.dev1.example.net.999999.0<\/em>.<\/p>\nRoute Records<\/H3>
\nYou noticed above there’s a seen-by<\/em> header, this is something entirely new and never before done in MCollective – and entirely optional and off by default. I anticipate you’d want to run with this off most of the time once your setup is done, it’s a debugging aid.<\/p>\nAs NATS is a full mesh your message probably only goes one hop within the Mesh. So if you record the connected server you publish into and the connected server your message entered it’s destination from you have a full route recorded.<\/p>\n
The Federation Broker logs and MCollective Client and Server logs all include the message ID so you can do a full trace in message packets and logs.<\/p>\n
There’s a PR against MCollective to expose this header to the client code so I will add something like mco federation trace some.node.example.net<\/em> which would send a round trip to that node and tell you exactly how the packet travelled. This should help a lot in debugging your setups as they will now become quite complex.<\/p>\nThe structure here is kind of meh and I will probably improve on it once the PR in MCollective lands and I can see what is the minimum needed to do a full trace.<\/p>\n
By default I’ll probably record the identities of the MCollective bits when Federated and not at all when not Federated. But if you enable the setting to record the full route it will produce a record of MCollective bits and the NATS nodes involved.<\/p>\n
In the end though from the Federation example we can infer a network like this:<\/p>\n
Federation NATS Cluster<\/B><\/p>\n\n- Federation Broker production_a -> nats2.fed.example.net<\/li>\n
- Federation Broker production_b -> nats3.fed.example.net<\/li>\n
- Client dev1.example.net -> nats1.fed.example.net<\/li>\n<\/ul>\n
Production NATS Cluster:<\/B><\/p>\n\n- Federation Broker production_a -> nats1.prod.example.net<\/li>\n
- Federation Broker production_b -> nats3.prod.example.net<\/li>\n
- Server dev2.example.net -> nats2.prod.example.net<\/li>\n<\/ul>\n
We don’t know the details of all the individual NATS nodes that makes up the entire NATS mesh but this is good enough. <\/p>\n
Of course this sample is the pathological case where nothing is connected to the same NATS instances anywhere. In my tests with a setup like this the overhead added across 10 000 round trips against 3 nodes – so 30 000 replies through 2 x Federation Brokers – was only 2 seconds, I couldn’t reliably measure a per message overhead as it was just too small.<\/p>\n
The NATS gem do expose the details of the full mesh though since NATS will announce it’s cluster members to clients, I might do something with that not sure. Either way, auto generated network maps should be totally possible.<\/p>\n
Conclusion<\/H3>
\nSo this is how Network Federation works in Choria. It’s particularly nice that I was able to do this without needing any state on the cluster thanks to past self making good design decisions in MCollective.<\/p>\n
Once the seen-by<\/em> thing is figured out I’ll publish JSON Schemas for these messages and declare protocol versions.<\/p>\nI can probably make future posts about the other message formats but they’re a bit nasty as MCollective itself is not yet JSON safe, the plan is it would become JSON safe one day and the whole thing will become a lot more elegant. If someone pings me for this I’ll post it otherwise I’ll probably stop here.<\/p>\n","protected":false},"excerpt":{"rendered":"
The old MCollective protocols are now ancient and was quite Ruby slanted – full of Symbols and used YAML and quite language specific – in Choria I’d like to support other Programming Languages, REST gateways and so forth, so a rethink was needed. I’ll look at the basic transport protocol used by the Choria NATS […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[1],"tags":[126,78],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3615"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=3615"}],"version-history":[{"count":52,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3615\/revisions"}],"predecessor-version":[{"id":3668,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3615\/revisions\/3668"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=3615"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=3615"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=3615"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}