Select Page

Tutorial: Writing MCollective Agents

by | Jun 27, 2010 | Code

I’ve recorded a screencast that walks you through the process of developing a SimpleRPC Agent, give it a DDL and also a simple client to communicate with it.

The tutorial creates a small echo agent that takes input and return it unmodified. It validates that you are sending a string and has a sample of dealing with intermittent failure.

Once you’ve watched this, or even during, you can use the following links are reference material: Writing Agents, Data Definition Language and Writing Clients.

You can view it directly on blip.tv which will hopefully be better quality.

I used a few VIM Snippets during the demo to boilerplate the agent and DDL, you’ll find these in the tarball for the upcoming 0.4.7 release in the ext/vim directory, they are already on GitHub too.

MCollective Data Definition Language

by | Jun 25, 2010 | Uncategorized

I mentioned in my recent post about mcollective Road Map about the DDL.

The DDL is used to describe agents in a way that is accessible by other programs, web applications, client libraries and so forth to help those various client tools to configure themselves correctly.

An actual example of a DDL file can be found here if you want to have a good look at it and full docs here.

I’ve created a short video showing the DDL and some of the features of the upcoming 0.4.7 release, you probably want to view it full screen to really see what’s going on.

And a quick note about the colors, I know people tend to feel strongly about this kind of thing, you can disable them in the config file of the client ๐Ÿ™‚

This is also my first attempt at using blip.tv, please let me know if you see any problems.

Recent MCollective releases and roadmap.

by | Jun 14, 2010 | Code

I’ve had two successive Marionette Collective releases recently, I was hoping to have one big one but I was waiting for the Stomp maintainers to do a release and it was taking a while.

These two releases are both major feature releases covering major feature sets. See lower down for a breakdown of it all.

We’re nearing feature completeness for the SimpleRPC layer as I am adding a number of features of interest to Enterprise and Large users especially around security and web UIs.

Once we’re at the end of this cycle I’ll do a 1.0.0 release and then from there move onto the next major feature cycle. The next cycle will focus on queuing long running tasks, background scheduling, future scheduling of tasks and a lot of related work. I posted some detail about these plans to the list recently.

Over the new few days or weeks I’ll do a number of Screencasts exploring some of these new features in depth, for now the list of what’s new:

Security

Connectivity

We can use Ruby Gem Stomp 1.1.6 which brings a lot of enhancements:

  • Connection pools for failover between multiple ActiveMQs
  • Lots of tunables about the connection pools such as retry frequencies etc
  • SSL TLS between node and ActiveMQ

Writing Web and Dynamic UIs

  • A DDL that describes agents, inputs and outputs:
    • Creates auto generated documentation
    • Can be used to auto generate user interfaces
    • The client library will only make requests that validate against the DDL
    • In future input validations will move into the DDL and will be done automatically for you
  • Web UI’s can bypass or do their own discovery and use the DDL to auto generate user interfaces

Usability

  • Fire-and-Forget style requests, for when you just want something done but do not care about results, these requests are very quick as they do not do any discovery.
  • Agents can now be reloaded without restarting the daemon
  • A new mc-inventory tool that can be used to view facts, agents and classes for a node
  • Many UI enhancements to the CLI tools

MCollective pgrep

by | May 23, 2010 | Uncategorized

The unix pgrep utility is great, it lets you grep through your process list and find interesting things. I wanted to do something similar but for my entire server group so built something quick ontop of MCollective.

I am using the Ruby sys-proctable gem to do the hard work, it returns a massive amount of information about each process and have written a simple agent on top of this.

The agent supports grepping the process tree but also supports kill and pgre+kill though I have not yet implemented more than the basic grep on the command line. Frankly the grep+kill combination scares me and I might remove it. A simple grep slipup and you will kill all processes on all your machine ๐Ÿ™‚ Sometimes too much power is too much and should just be avoided.

At the moment mc-pgrep outputs a set format but I intend to make that configurable on the command line, here’s a sample:

% mc-pgrep -C /dev_server/ ruby
 
 * [ ============================================================> ] 4 / 4
 
dev1.my.com
       root   9833  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
       root  21608  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
 
dev2.my.com
       root  14568  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
       root  31595  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
 
dev3.my.com
       root   1620  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
       root  14093  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
 
dev4.my.com
       root   3231  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
       root  20557  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
 
   ---- process list stats ----
        Matched hosts: 4
    Matched processes: 8
        Resident Size: 37.264KB
         Virtual Size: 629.578MB

You can also limit it to only find zombies with the -z option.

This has been quite interesting for me, if I limit the pgrep to “.” (the pattern is regex) every machine will send back a Sys::ProcTable hash for all its processes. This is a 50 to 70 KByte payload per server. I’ve so far seen no problem getting his much traffic through ActiveMQ + MCollective and processing it all in a very short time:

% time mc-pgrep -F "country=/uk|us/" .
 
   ---- process list stats ----
        Matched hosts: 20
    Matched processes: 1958
        Resident Size: 1.777MB
         Virtual Size: 60.072GB
 
mc-pgrep -F "country=/uk|us/" .  0.19s user 0.06s system 7% cpu 3.420 total

That 3.4 seconds is with a 2 second discovery overhead client machine in Germany and the filter matching UK and US machines – all the way to the West Coast – my biggest delay here is network and not MC or ActiveMQ.

The code can be found at my GitHub account and still a bit of a work in progress, wiki pages will follow once I am happy with it.

And as an aside, I am slowly migrating at least my code to GitHub if not wiki and ticketing. So far my Plugins have moved, MC will move soon too.

DKIM with CentOS 5 and Exim

by | May 14, 2010 | Uncategorized

DomainKeys Identified Mail – DKIM – is a recent attempt to add some sender verification to email. Read more here, here and in the RFC 4871 to get some background info.

If you’re sending any newsletters you really want to be investigating this, if you’re doing anti spam it’s good to start looking at tracking this and really everyone should have DKIM on their domains. Exim recently – as of 4.70 – have decent support for it but CentOS is still on 4.63 thanks to RHEL.

To get a new Exim on your CentOS machine I suggest just using ATrpms who as of writing has 4.71 packages available for Exim and the other bits you need. I needed:

exim-4.71-40.el5.i386.rpm
exim-mysql-4.71-40.el5.i386.rpm
libspf2_2-1.2.5-5.0.el5.i386.rpm
libsrs_alt1-1.0-3_rc1.0.el5.i386.rpm

As well as the 64bit versions, you can just add ATrpms to your systems but really you should have your own repos and carefully control the packages that goes out to your estate.

Once you have upgraded your stock Exim to these versions – it’s a totally clean and compatible upgrade – configuring Exim to automagically sign outgoing mail with DKIM is pretty easy. We’ll make it so it looks for keys in a specific location based on outgoing mail domain so if you’re a relay for many domains you just need to drop down the certs.

Put the following near the top of our /etc/exim/exim.conf file, this just sets some macros we’ll use later on:

DKIM_DOMAIN = ${lc:${domain:$h_from:}}
DKIM_FILE = /etc/exim/dkim/${lc:${domain:$h_from:}}.pem
DKIM_PRIVATE_KEY = ${if exists{DKIM_FILE}{DKIM_FILE}{0}}

This will use, based on sender domain, a private key in /etc/exim/dkim/sender_domain.pem. By default exim just logs DKIM verification, doesn’t block any incoming mail I won’t cover doing blocks here just sending.

Next find your remote_smtp transport later in the file and change it to look like this:

remote_smtp:
  driver = smtp
  dkim_domain = DKIM_DOMAIN
  dkim_selector = x
  dkim_private_key = DKIM_PRIVATE_KEY
  dkim_canon = relaxed
  dkim_strict = 0

This will make Exim do the DKIM signing on outgoing mail but only if it can find a certificate.

To make the certificates is pretty easy, we’ll use a domain example.com:

$ mkdir /etc/exim/dkim/ && cd /etc/exim/dkim/
$ openssl genrsa -out example.com.pem 1024 
$ openssl rsa -in example.com.pem -out example.com-public.pem -pubout -outform PEM

All that’s left now is to update your dns, sticking to example.com you’d add something like this into your bind zone file the text to add after p= is the stuff you’ll find in the public key called example.com-public.pem in our example:

x._domainkey     IN      TXT     "v=DKIM1\; t=y\; k=rsa\; p=MIGfMA0<snip>AQAB"
_domainkey       IN      TXT     "t=y\; o=~\;"

The x matches up with your dkim_selector in the SMTP transport above. The t=y tells the world you’re still testing your setup so remove that only when you’re all 100% certain it works. The o=~ tells everyone you will sign only some mail. You can make that o=- if all mail from you would be signed.

You can verify your DNS is right like this:

$ dig +short txt x._domainkey.example.com
"v=DKIM1\; k=rsa\; p=MIGfMA0<snip>AQAB"

And finally if you’re sending mail you should now see a header in the mail like this:

DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=example.com; s=x;
	h=From:To:Message-Id:Date; bh=g3zLY<snip>5uGs=; b=fonAB<snip>bceHhQ==;

Finally you can send an email to check-auth@verifier.port25.com and it will reply with all sorts of test output about your domain including DKIM validation details.