MCollective Simple RPC

by R.I. Pienaar | Dec 22, 2009 | Code

MCollective is a framework for writing RPC style tools that talk to a cloud of servers, till now doing that has been surprisingly hard for non ruby coders. The reason for this is that I was focussing on getting the framework built and feeling my way around the use cases.

I’ve now spent 2 days working on simplifying actually writing agents and consumers. This code is not released yet – just in SVN trunk – but here’s a taster.

First writing an agent should be simple, here’s a simple ‘echo’ server that takes a message as input and returns it back.

class Rpctest<RPC::Agent
    # Basic echo server
    def echo_action(request, reply)
         raise MissingRPCData, "please supply a :msg" unless request.include?(:msg)
 
         reply.data = request[:msg]
    end
end

This creates an echo action, does a quick check that a message was received and sends it back. I want to create a few more validators so you can check easily if the data passed to you is sane and secure if you’re doing anything like system() calls with it.

Here’s the client code that calls the echo server 4 times:

#!/usr/bin/ruby
 
require 'mcollective'
 
include MCollective::RPC
 
rpctest = rpcclient("rpctest")
 
puts "Normal echo output, non verbose, shouldn't produce any output:"
printrpc rpctest.echo(:msg => "hello world")
 
puts "Flattened echo output, think combined 'mailq' usecase:"
printrpc rpctest.echo(:msg => "hello world"), :flatten => true
 
puts "Forced verbose output, if you always want to see every result"
printrpc rpctest.echo(:msg => "hello world"), :verbose => true
 
puts "Did not specify needed input:"
printrpc rpctest.echo

This client supports full discovery and all the usual stuff, has pretty –help output and everything else you’d expect in the clients I’ve supplied with the core mcollective. It caches discovery results so above code will do one discovery only and reuse it for the other calls to the collective.

When running you’ll see a twirling status indicator, something like:

  - [5 / 10]

This will give you a nice non scrolling indicator of progress and should work well for 100s of machines without spamming you with noise, at the end of the run you’ll get the output.

The printrpc helper function tries its best to print output for you in a way that makes sense on large amounts of machines.

• By default it doesn’t print stuff that succeeds, you do get a overall progress indicator though
• If anything does go wrong, useful information gets printed but only for hosts that had problems
• If you ran the client with –verbose, or forced it to verbose mode output you’ll get a full bit of info of the result from every server.
• It supports flags to modify the output, you can flatten the output so hostnames etc aren’t showed, just a concat of the data.

The script above gives the following output when run in non-verbose mode:

$ rpctest.rb --with-class /devel/
 
Normal echo output, non verbose, shouldn't produce any output:
 
Forced verbose output, if you always want to see every result:
dev1.your.com                          : OK
    "hello world"
 
dev2.your.com                          : OK
    "hello world"
 
dev3.your.com                          : OK
    "hello world"
 
Flattened echo output, think combined 'mailq' usecase:
hello world
hello world
hello world
 
Did not specify needed input:
dev1.your.com                          : please supply a :msg
dev2.your.com                          : please supply a :msg
dev3.your.com                          : please supply a :msg

Still some work to do, specifically stats needs a rethink in a scenario where you are making many calls such as in this script.

This will be in mcollective version 0.4 hopefully out early January 2010

MCollective Release 0.2.0

by R.I. Pienaar | Dec 1, 2009 | Code

I am pleased to announce the the first actual numbered release of The Marionette Collective, you can grab it from the downloads page.

Till now people wanting to test this had to pull out of SVN directly, I put off doing a release till I had most of the major tick boxes in my mind ticked and till I knew I wouldn’t be making any major changes to the various plugins and such. This release is 0.2.x since 0.1.x was the release number I used locally for my own testing.

This being the first release I fully anticipate some problems and weirdness, please send any concerns to the mailing list or ticketing system.

This has been a while coming, I’ve posted lots on this blog about mcollective, what it is and what it does. For those just joining you want to watch the video on this post for some background.

I am keen to get feedback from some testers, specifically keen to hear thoughts around these points:

• How does the client tools behave on 100s of nodes, I suspect the output format might be useless if it just scrolls and scrolls, I have some ideas about this but need feedback.
• On large amount of hosts, or when doing lots of requests soon after each other, do you notice any replies going missing.
• Feed back about the general design principals, especially how you find the plugin system and what else you might want pluggable. I for example want to make it much easier to add new discovery methods.
• Anything else you can think of

I’ll be putting in tickets on the issue system for future features / fixes I am adding so you can track there to get a feel for the milestones toward 0.3.x.

Thanks goes to the countless people who I spoke to in person, on IRC and on Twitter, thanks for all the retweets and general good wishes. Special thanks also to Chris Read who made the debian package code and fixed up the RC script to be LSB compliant.