With the release of mcollective recently I’ve been able to work a bit on a deploy problem I’ve had at a client, I was able to build up the following by combining mcollective, cucumber and the open source mcollective plugins.
The cucumber exploring is of course a result of @auxesis‘s brilliant cucumber talk at Devops Days recently.
Note: I’ve updated this from the initial posting, showing how I do filtering with mcollective discovery and put it all into one scenario.
Feature: Update the production systems
Background:
Given the load balancer has ip address 192.168.1.1
And I want to update hosts with class roles::dev_server
And I want to update hosts with fact country=de
And I want to pre-discover how many hosts to update
Scenario: Update the website
When I block the load balancer
Then traffic from the load balancer should be blocked
When I update the package mywebapp
Then the package version for mywebapp should be 4.2.6-3.el5When I unblock the load balancer
Then traffic from the load balancer should be unblocked
Feature: Update the production systems
Background:
Given the load balancer has ip address 192.168.1.1
And I want to update hosts with class roles::dev_server
And I want to update hosts with fact country=de
And I want to pre-discover how many hosts to update
Scenario: Update the website
When I block the load balancer
Then traffic from the load balancer should be blocked
When I update the package mywebapp
Then the package version for mywebapp should be 4.2.6-3.el5
When I unblock the load balancer
Then traffic from the load balancer should be unblocked
This is completely like any other test driven scenario based system, if it fails to block the firewall deploy will bail out. If it fails to update the package it will bail and finally only if those worked will it unblock the firewall.
Thanks to mcollective this is distributed and parallel over large numbers of machines. I can also apply filters to update just certain clusters using mcollective’s discovery features.
Everything’s outcome is tested and cucumber will only show the all clear when everything worked on all machines in a consistent way.
This is made possible in part because the mcollective plugins use the Puppet providers underneath the hood, so package and service actions are complete idempotent and repeatable, I can rerun this script 100 times and it will do the same thing.
I have other steps not included here to keep things simple but in a real world I would restart the webserver after the update and I would then call NRPE plugins on all the nodes to make sure their load average is in acceptable ranges before the firewall gets opened letting the load balancer in.
This opens up a whole lot of interesting ideas, kudos to @auxesis and his great talk at devopsdays!
Last night I was thinking about writing scripts to manage services, packages etc in a heterogeneous environment. This is hard because the operating systems all behave differently.
This is of course a problem that Puppet solves with it’s providers system, after some chatting with Luke on IRC I now have a pretty nice solution.
Assuming you don’t mind shell scripting in Ruby here’s a pretty decent bit of Ruby to manage a service on many different environments.
require'puppet'
service = ARGV.shift
action = ARGV.shift
ARGV.length>0 ? hasstatus = true : hasstatus = falsebegin
svc = Puppet::Type.type(:service).new(:name=> service,
:hasstatus=> hasstatus).provider
svc.send action
puts("#{service} #{action} status: #{svc.status}")rescueException=> e
puts("Could not #{action} service #{service}: #{e}")end
require 'puppet'
service = ARGV.shift
action = ARGV.shift
ARGV.length > 0 ? hasstatus = true : hasstatus = false
begin
svc = Puppet::Type.type(:service).new(:name => service,
:hasstatus => hasstatus).provider
svc.send action
puts("#{service} #{action} status: #{svc.status}")
rescue Exception => e
puts("Could not #{action} service #{service}: #{e}")
end
You’d probably want to put in support for the pattern parameter to keep certain broken Operating Systems happy, but this is pretty nice and platform independent.
I spoke a bit on the puppet IRC channel about my middleware based systems administration tool, I made a video to demo it below.
The concept is that I use publish / subscribe middleware – ActiveMQ with Stomp in my case – to do one-off administration. Unlike using Capistrano or some of those tools I do not need lists of machines or visit each machine with a request because the network supports discovery and a single message to the middleware results in 10s or 100s or 1000s of machines getting the message.
This means any tasks I ask to be done happens in parallel on any number of machines typically I see ~100 machines finish the task in the same time as 1 machine would and no need for SSH or anything like that.
The app server and client libs I wrote take away all the complexities of the middleware and takes care of crypto signing requests, only responding to requests that has been signed properly etc, serializing and deserialization of data etc.
Discovery is built in and it supports puppet classes and facts and a few other criteria I use for my own systems so there is no need to build any kind of system that keeps track of what machines I have with what version of operating system etc. As long as is on the middleware I can find it.
The bulk – timeout handling and so forth removed – of the ping app you see in the demo can be seen here, client:
client = Stomphost::Client.new(config)
client.sendreq(Time.now.to_f, "echo")
loop do
resp = client.receive
elapsed = (Time.now.to_f - resp[:body]) * 1000
end
And the agent is just this:
module Stomphost
module Agent
class Echo
def handlemsg(msg)
msg[:body]endendendend
module Stomphost
module Agent
class Echo
def handlemsg(msg)
msg[:body]
end
end
end
end
You can see that even data types like the float will flow cleanly through end to end.
Watch the video, I mention my uses cases but it includes distributed Exim administration, package updates, services restarts, iptables management and much more.
UPDATE: This code has now been released as an Open Source Apache 2 licenced project at marionette-collective.org
Continuing to move a few bits of unimportant code into the public I’ve released a ruby library called passmakr that assists in creating passwords in various useful formats.
Mostly it takes some key bits of code from another password library and re-uses it in a new way that has fewer dependencies etc.
It supports creating phonemic passwords – that humans should find easier to remember than random ones – as well as random ones using Ruby’s random function as well a mode for reading /dev/urandom.
Using it is trivial, the code has rdoc in it as well to show you the other modes.
You can pass it modes of :phonemic, :random and :urandom.
There’s also a little cli tool included that works like this:
% passmakr -p-l88 character phonemic password:
Password: ooHivahn
Crypt: gDxSAc/ewkTzg
MD5: $1$TcwcAexC$nNxMbU5CTQTt4I.JSNRhx0
Rot 13: bbUvinua
NATO: oscar oscar Hotel india victor alfa hotel november
% passmakr -p -l 8
8 character phonemic password:
Password: ooHivahn
Crypt: gDxSAc/ewkTzg
MD5: $1$TcwcAexC$nNxMbU5CTQTt4I.JSNRhx0
Rot 13: bbUvinua
NATO: oscar oscar Hotel india victor alfa hotel november
The various modes printed are some often found in Unix systems, I’ll add other modes if people send me requests. Rot 13 is there just for laughs.
This code also extends the normal String class with a to_nato method that can be used on all strings:
"Foo Bar".to_nato=>"Foxtrot oscar oscar Bravo alfa romeo"
"Foo Bar".to_nato
=> "Foxtrot oscar oscar Bravo alfa romeo"
Puppet supports putting nodes in environments, this maps cleanly to your development, qa and production life cycles and it’s a way to hand out different code to different nodes. Before reading this post you should probably read the Puppet Wiki page first for background.
I’ll present here a simple layout that is both a powerful tool for isolating code changes while developing combined with minimizing the headaches with maintaining multiple branches of the same module for an extended period of time.
As with the other posts I did on puppet recently the focus here is simplicity, this will work for small sites and will hopefully give you enough inspiration to successfully model larger sites using these techniques.
To really use environments well I strongly suggest you use modules, I’ve blogged about them before, if you read this post and think “hey! that’s great” and don’t yet use modules, go fix that first.
The puppetmaster tries to find modules using it’s modulepath setting, typically something like /etc/puppet/modules.ย You usually just set this value once in your puppet.conf and that’s it, all done. Environments expand on this and give you the ability to set different settings for different environments, below a simple example.
This is your basic multi environment setup, if you run a client with the development environment set modules will be served from an alternative path.
This is already a huge win, if you use something like below:
class ntp {
file{"/etc/ntpd.conf":
source => ["puppet:///ntp/ntpd.conf.${environment}",
"puppet:///ntp/ntpd.conf"]
}
}
node dev1 {
include ntp
}
class ntp {
file{"/etc/ntpd.conf":
source => ["puppet:///ntp/ntpd.conf.${environment}",
"puppet:///ntp/ntpd.conf"]
}
}
node dev1 {
include ntp
}
If you put the ntp module in both production and development, the right one will be chosen depending on the nodes environment setting. The file ntp.conf will also be served from the right modules files directory. You could now safely make changes to the development one without affecting the rest of the machines.
The major drawback here is that if you have say 30 modules, you need to duplicate the lot into development if you wish to use them there, this is not great at all as it quickly becomes a maintenance nightmare. If you made a change to ntp module in development, you need to remember to go and do the same change in all your other environments. In cases where you give each sysadmin or developer their own environment this rapidly becomes unmaintainable.
The modulepath option has a trick up it’s sleeve though, you can specify several search paths just like the normal Unix PATH environment variable, we can rework our config like this.
Here we’ve added a few more environments, even one for a developer called John Doe. The big thing we did though is create /etc/puppet/manifests/common/modules as a 2nd search path.
production
`-- modules
development
`-- modules
common
`-- modules
`-- ntp
`-- manifests
`-- init.pp
production
`-- modules
development
`-- modules
common
`-- modules
`-- ntp
`-- manifests
`-- init.pp
The idea here is that you’d strive to keep all your modules in the common directory as their stable location but if you wished to do development on your ntp module that you wish to test, you just copy the ntp module into either development or johndoe directories, with that done only machines in those environments will get the new code.
This works especially well if you use version control, I use SVN and treat common as my trunk, whenever I need to work on a module I simply branch it into development, do my work there and when ready to release I merge my changes back to common and get rid of the branched copy so all environments gets the same version.
With this you can easily come up with a simple release testing strategy, do the weeks code in development and test locally, then put the changes you wish to release to production into your QA environment and do tests. Finally when you’re ready merge all the changes into common and just get rid of the development and QA copies, all environments are now running the same code ready for the next cycle.
This gives you the ability to create branches, do testing, isolate developers from each other – just put their desktops in individual environments – and lots of other nifty things but in the end you have only one copy of the code for your stable released versions.
I hope you find this useful, I’ll reiterate that this is a simple setup, you should use this information and come up with a solution that will work well for your use case and teams.
Since I’ve been very happy with Google Code for hosting ruby-pdns I thought I’d start moving some other projects there too, in the process I’ll probably also release some other bits and bobs that I did not previous share the code of.
This was my first major bit of Ruby code and looking back at it now I’m still fairly happy with it but the real proof is in the pudding; my production copy of this code is up over 300 days now, does not leak memory or threads and have served 10s of millions of requests without any problems.
