The typical Puppet use case is to run the daemon every 30 minutes or so and just let it manage your machines. Sometimes though you want to be able to run it on all your machines as quick as your puppet master can handle.
This is tricky as you generally do not have a way to cap the concurrency and it’s hard to orchestrate that. I’ve extended the MCollective Puppet Agent to do this for you so you can do a rapid run at roll out time and then go back to the more conservative slow pace once your window is over.
The basic logic I implemented is this:
- Discover all nodes, sort them alphabetically
- Count how many nodes are active now, wait till it’s below threshold
- Run a node by just starting a –onetime background run
- Sleep a second
This should churn through your nodes very quickly without overwhelming the resources of your master. You can see it in action here, you can see it started 3 nodes and once it got to the 4th 3 were already running and it waited for one of them to finish:
% mc-puppetd -W /dev_server/ runall 2 Thu Aug 05 17:47:21 +0100 2010> Running all machines with a concurrency of 2 Thu Aug 05 17:47:21 +0100 2010> Discovering hosts to run Thu Aug 05 17:47:23 +0100 2010> Found 4 hosts Thu Aug 05 17:47:24 +0100 2010> Running dev1.one.net, concurrency is 0 Thu Aug 05 17:47:26 +0100 2010> dev1.one.net schedule status: OK Thu Aug 05 17:47:28 +0100 2010> Running dev1.two.net, concurrency is 1 Thu Aug 05 17:47:30 +0100 2010> dev1.two.net schedule status: OK Thu Aug 05 17:47:32 +0100 2010> Running dev2.two.net, concurrency is 2 Thu Aug 05 17:47:34 +0100 2010> dev2.two.net schedule status: OK Thu Aug 05 17:47:35 +0100 2010> Currently 3 nodes running, waiting Thu Aug 05 17:48:00 +0100 2010> Running dev3.two.net, concurrency is 2 Thu Aug 05 17:48:05 +0100 2010> dev3.two.net schedule status: OK |
This is integrated into the existing mc-puppetd client script you don’t need to roll out anything new to your servers just the client side.
Using this to run each of 47 machines with a concurrency of just 4 I was able to complete a cycle in 8 minutes. Doesn’t sound too impressive but my average run time is around 40 seconds on every node with some being 90 to 150 seconds. My puppetmaster server that usually sits at a steady 0.2mbit out were serving a constant 2mbit/sec for the duration of this run.
