In my previous post I talked about the need to load test Choria given that I now aim for much larger workloads. This post goes into a few of the things you need to consider when sizing the optimal network size.
Given that we now have the flexibility to build 50 000 node networks quite easily with Choria the question is should we, and if yes then what is the right size. As we can now federate multiple Collectives together into one where each member Collective is a standalone network we have the opportunity to optimise for the operability of the network rather than be forced to just build it as big as we can.
What do I mean when I say the operability of the network? Quite a lot of things:
- What is your target response time on a unbatched mco rpc rpcutil ping command
- What is your target discovery time? You should use a discovery data source but broadcast is useful, so how long do you want?
- If you are using a discovery source, how long do you want to wait for publishes to happen?
- How many agents will you run? Each agent makes multiple subscriptions on the middleware and consume resources there
- How many sub collectives do you want? Each sub collective multiply the amount of subscriptions
- How many federated networks will you run?
- When you restart the entire NATS, how long do you want to wait for the whole network to reconnect?
- How many NATS do you need? 1 can run 50 000 nodes, but you might want a cluster for HA. Clustering introduces overhead in the middleware
- If you are federating a global distributed network, what impact does the latency cross the federation have and what is acceptable
So you can see that to a large extend the answer here is related to your needs and not only to the needs of benchmarking Choria. I am working on a set of tools to allow anyone to run tests locally or on a EC2 network. The main work hose is a Choria emulator that runs a 1 000 or more Choria instances on a single node so you can use a 50 node EC2 network to simulate a 50 000 node one.
Middleware Scaling Concerns
Generally for middleware brokers there are a few things that impact their scalability:
- Number of TCP Connections – generally a thread/process is made for each
- TLS or Plain text – huge overhead in TLS typically and it can put a lot of strain on single systems
- Number of message targets – queues, topics, etc. Different types of target have different overheads. Often a thread/process for each.
- Number of subscribers to each target
- Cluster overhead
- Persistence overheads like storage and ACKs etc
You can see it’s quite a large number of variables that goes into this, anywhere that requires a thread or process to manage 1 of it means you should get worried or at least be in a position to measure it.
NATS uses 1 go routine for each connection and no additional ones per subscription etc, its quite light weight but there are no hard and fast rules. Best to observe how it grows by needs, something I’ll include in my test suite.
How Choria uses NATS
It helps then to understand how Choria will use NATS and what connections and targets it makes.
A single Choria node will:
- Maintain a single TCP+TLS connection to NATS
- Subscribe to 1 queue unique to the node for every Subcollective it belongs to
- For every agent – puppet, package, service, etc – subscribe to a broadcast topic for that agent. Once in every Subcollective. Choria comes default with 7 agents.
So if you have a node with 10 agents in 5 Subcollectives:
- 50 broadcast subjects for agents
- 5 queue subjects
- 1 TCP+TLS connection
So 100 nodes will have 5 500 subscriptions, 550 NATS subjects and 100 TCP+TLS connections.
Ruby based Federation brokers will maintain 1 subscription to a queue subject on the Federation and same on the Collective. The upcoming Go based Federation Brokers will maintain 10 (configurable) connections to NATS on each side, each with these subscriptions.
Conclusion
This will give us a good input into designing a suite of tools to measure various things during the run time of a big test, check back later for details about such a tool.
You can read about the emulator I wrote in the next post.
