Today I am very pleased to release something I’ve been thinking about for years and actively working on since August.
After many POCs and thrown away attempts at this over the years I am finally releasing a Playbook system that lets you run work flows on your MCollective network – it can integrate with a near endless set of remote services in addition to your MCollective to create a multi service playbook system.
This is a early release with only a few integrations but I think it’s already useful and I’m looking for feedback and integrations to build this into something really powerful for the Puppet eco system.
The full docs can be found on the Choria Website, but below you can get some details.
Overview
Today playbooks are basic YAML files. Eventually I envision a Service to execute playbooks on your behalf, but today you just run them in your shell, so they are pure data.
Playbooks have a basic flow that is more or less like this:
Discover named Node Sets
Validate the named Node Sets meet expectations such as reachability and versions of software available on them
Run a pre_book task list that lets you do prep work
Run the main tasks task list where you do your work, around every task certain hook lists can be run
Run either the on_success or on_fail task list for notification of Slacks etc
Run the post_book task list for cleanups etc
Today a task can be a MCollective request, a shell script or a Slack notification. I imagine this list will grow huge, I am thinking you will want to ping webhooks, or interact with Razor to provision machines and wait for them to finish building, run Terraform or make EC2 API requests. This list of potential integrations is endless and you can use any task in any of the above task lists.
A Node Set is simply a named set of nodes, in MCollective that would be certnames of nodes but the playbook system itself is not limited to that. Today Node Sets can be resolved from MCollective Discovery, PQL Queries (PuppetDB), YAML files with groups of nodes in them or a shell command. Again the list of integrations that make sense here is huge. I imagine querying PE or Foreman for node groups, querying etcd or Consul for service members. Talking to random REST services that return node lists or DB queries. Imagine using Terraform outputs as Node Set sources or EC2 API queries.
In cases where you wish to manage nodes via MCollective but you are using a cached discovery source you can ask node sets to be tested for reachability over MCollective. And node sets that need certain MCollective agents can express this desire as SemVer version ranges and the valid network state will be asserted before any playbook is run.
Playbooks do not have a pseudo programming language in them though I am not against the idea. I do not anticipate YAML to be the end format of playbooks but it’s good enough for today.
Example
I’ll show an example here of what I think you will be able to achieve using these Playbooks.
Here we have a web stack and we want to do Blue/Green deploys against it, sub clusters have a fact cluster. The deploy process for a cluster is:
Gather input from the user such as cluster to deploy and revision of the app to deploy
Discover the Haproxy node using Node Set discovery from PQL queries
Discover the Web Servers in a particular cluster using Node Set discovery from PQL queries
Verify the Haproxy nodes and Web Servers are reachable and running the versions of agents we need
Upgrade the specific web tier using:
Tell the ops room on slack we are about to upgrade the cluster
Disable puppet on the webservers
Wait for any running puppet runs to stop
Disable the nodes on a particular haproxy backend
Upgrade the apps on the servers using appmgr#upgrade to the input revision
Do up to 10 NRPE checks post upgrade with 30 seconds between checks to ensure the load average is GREEN, youโd use a better check here something app specific
Enable the nodes in haproxy once NRPE checks pass
Fetch and display the status of the deployed app – like what version is there now
Enable Puppet
Should the task list all FAIL we run these tasks:
Call a webhook on AWS Lambda
Tell the ops room on slack
Run a whole other playbook called deploy_failure_handler with the same parameters
Should the task list PASS we run these tasks:
Call a webhook on AWS Lambda
Tell the ops room on slack
This example and sample playbooks etc can be found on the Choria Site.
Status
Above is the eventual goal. Today the major missing piece here that I think MCollective needs to be extended with the ability for Agent plugins to deliver a Macro plugin. A macro might be something like Puppet.wait_till_idle(:timeout => 600), this would be something you call after disabling the nodes and you want to be sure Puppet is making no more changes, you can see the workflow above needs this.
There is no such Macros today, I will add a stop gap solution as a task that waits for a certain condition but adding Macros to MCollective is high on my todo list.
Other than that it works, there is no web service yet so you run them from the CLI and the integrations listed above is all that exist, they are quite easy to write so hoping some early adopters will either give me ideas or send PRs!
UPDATE: Since posting this blog I had some time and added: Terraform Node Sets, ability to create GET and POST Webhook requests and the much needed ability to assert and wait for remote state.
I started a project called Choria that aimed to massively improve the deployment UX and yield a secure and stable MCollective setup for those using Puppet 4.
The aim is to make installation quick and secure, towards that it seems a common end to end install from scratch by someone new to project using a clustered NATS setup can take less than a hour, this is a huge improvement.
Further I’ve had really good user feedback, especially around NATS. One user reports 2000 nodes on a single NATS server consuming 300MB RAM and it being very performant, much more so than the previous setup.
It’s been a few months, this is whats changed:
The module now supports every OS AIO Puppet supports, including Windows.
Documentation is available on choria.io, installation should take about a hour max.
The PQL language can now be used to do completely custom infrastructure discovery against PuppetDB.
Many bugs have been fixed, many things have been streamlined and made more easy to get going with better defaults.
Event Machine is not needed anymore.
A number of POC projects have been done to flesh out next steps, things like a very capable playbook system and a revisit to the generic RPC client, these are on GitHub issues.
Meanwhile I am still trying to get to a point where I can take over maintenance of MCollective again, at first Puppet Inc was very open to the idea but I am afraid it’s been 7 months and it’s getting nowhere, calls for cooperation are just being ignored. Unfortunately I think we’re getting pretty close to a fork being the only productive next step.
For now though, I’d say the Choria plugin set is production ready and stable any one using Puppet 4 AIO should consider using these – it’s about the only working way to get MCollective on FOSS Puppet now due to the state of the other installation options.
I share a ton of links, I have a thing that harvests my twitter account and over a few years it has collected over 4 500 links. Twitter is great for that but also it’s hard to add a bit of annotation or comment to those shared links in the constrained space. Of course I also curate my Free for Dev list which has grown quite huge, so I come across a lot of interesting stuff.
Everyone these days seem to have a news letter and I’ve toyed with the idea for a long time. The idea would be to share the random things I come across, no real theme, goal or set frequency, just a list of stuff I find with some short comments whenever I feel I have a good selection of things to share. I could stick them up here as blog posts but I don’t want one of those sad blogs where the entire frontpage is weekly link lists ๐
Well I’ve decided to give it a go, I signed up with Mail Chimp and set something up there. Last week I tweeted twice about it and tried to get some early adopters on board for the first mail in order to get some feedback and so forth. This has been quite positive thanks a lot to those who took the time to send me a note.
So if you’d like to subscribe please head over to https://www.devco.net/ and sign up at the bar above, it’s the usual Mail Chimp thing so you’ll get a confirmation email to double opt-in with. If you want to see what you’re getting yourself into the archive can be found at Mail Chimp, there is one mail there already.
The next one should go out tomorrow or so, looking forward to trying this out!
For a few releases now PuppetDB had a new query language called Puppet Query Language or PQL for short. It’s quite interesting, I thought a quick post might make a few more people aware of it.
Overview
To use it you need a recent PuppetDB and as this is quite a new feature you really want the latest PuppetDB. There is nothing to enable when you install it the feature is already active. The feature is marked as experimental so some things will change as it moves to production.
PQL Queries look more or less like this:
nodes { certname ~ 'devco'}
nodes { certname ~ 'devco' }
This is your basic query it will return a bunch of nodes, something like:
There are a bunch of in-built relationships between say a node and it’s facts and inventory, so queries can get quite complex:
inventory[certname]{
facts.osfamily = "RedHat" and
facts.dc = "linodeldn" and
resources {type = "Package" and
title = "java" and
parameters.ensure = "1.7.0"}}
inventory[certname] {
facts.osfamily = "RedHat" and
facts.dc = "linodeldn" and
resources {
type = "Package" and
title = "java" and
parameters.ensure = "1.7.0"
}
}
This finds all the RedHat machines in a particular DC with Java 1.7.0 on them. Be aware this will also find machines that are deactivated.
I won’t go into huge details of the queries, the docs are pretty good – examples, overview.
So this is quite interesting, this finally gives us a reasonably usable DB to do queries that mcollective discovery used to be used for – but of course its not a live view nor does it have any clue what the machines are up to but as a cached data source for discovery this is interesting.
Using
CLI
You can of course query this stuff on the CLI and I suggest you familiarise yourself with JQ.
This is in ~/.puppetlabs/client-tools/puppetdb.conf which is a bit senseless to me since there clearly is a standard place for config files, but alas.
Once you have this and you installed the puppet-client-tools package you can do queries like:
$ puppet query "nodes { certname ~ 'devco.net' }"
$ puppet query "nodes { certname ~ 'devco.net' }"
Puppet Code
Your master will have the puppetdb-termini package on it and this brings with it Puppet functions to query PuppetDB so you do not need to use a 3rd party module anymore:
At the recent PuppetConf Puppet announced that their enterprise tool puppet job supports using this as discovery, if I remember right it’s something like:
These queries are a bit special in that they must return just the certname as here, I’ll document this up. The reason for this is that they are actually part of a much larger query done in the Choria discovery system (that uses PQL internally and is a good intro on how to query this API from code).
Here’s an example of a complex query – as used by Choria internally – that limits nodes to ones in a particular collective, our PQL query, which ones have mcollective installed and running. You can see you can nest and combine queries into quite complex ones:
nodes[certname, deactivated]{# finds nodes in the chosen subcollective via a fact(certname in inventory[certname]{
facts.mcollective.server.collectives.match("\d+") = "mcollective"}) and
# does the supplied PQL query, note the specific certname match(certname in nodes[certname]{
certname ~ 'devco.net'}) and
# limited to machines with mcollective installed(resources {type = "Class" and title = "Mcollective"}) and
# who also have the service started(resources {type = "Class" and title = "Mcollective::Service"})}
nodes[certname, deactivated] {
# finds nodes in the chosen subcollective via a fact
(certname in inventory[certname] {
facts.mcollective.server.collectives.match("\d+") = "mcollective"
}) and
# does the supplied PQL query, note the specific certname match
(certname in nodes[certname] {
certname ~ 'devco.net'
}) and
# limited to machines with mcollective installed
(resources {
type = "Class" and title = "Mcollective"
}) and
# who also have the service started
(resources {
type = "Class" and title = "Mcollective::Service"
})
}
Conclusion
This is really handy and I hope more people will become familiar with it. I don’t think this quite rolls off the fingers easily – but neither does SQL or any other similar system so par for the course. What is amazing is that we can get nearer to having a common language across CLI, Code, Web UIs and 3rd party tools for describing queries of our estate so this is a major win.
