Tutorial: Writing MCollective Agents

by R.I. Pienaar | Jun 27, 2010 | Code

I’ve recorded a screencast that walks you through the process of developing a SimpleRPC Agent, give it a DDL and also a simple client to communicate with it.

The tutorial creates a small echo agent that takes input and return it unmodified. It validates that you are sending a string and has a sample of dealing with intermittent failure.

Once you’ve watched this, or even during, you can use the following links are reference material: Writing Agents, Data Definition Language and Writing Clients.

You can view it directly on blip.tv which will hopefully be better quality.

I used a few VIM Snippets during the demo to boilerplate the agent and DDL, you’ll find these in the tarball for the upcoming 0.4.7 release in the ext/vim directory, they are already on GitHub too.

Recent MCollective releases and roadmap.

by R.I. Pienaar | Jun 14, 2010 | Code

I’ve had two successive Marionette Collective releases recently, I was hoping to have one big one but I was waiting for the Stomp maintainers to do a release and it was taking a while.

These two releases are both major feature releases covering major feature sets. See lower down for a breakdown of it all.

We’re nearing feature completeness for the SimpleRPC layer as I am adding a number of features of interest to Enterprise and Large users especially around security and web UIs.

Once we’re at the end of this cycle I’ll do a 1.0.0 release and then from there move onto the next major feature cycle. The next cycle will focus on queuing long running tasks, background scheduling, future scheduling of tasks and a lot of related work. I posted some detail about these plans to the list recently.

Over the new few days or weeks I’ll do a number of Screencasts exploring some of these new features in depth, for now the list of what’s new:

Security

• New SSL based security system
• New Authorization system [blog post] and sample plugins, allowing for fine grained control over every request

Connectivity

We can use Ruby Gem Stomp 1.1.6 which brings a lot of enhancements:

• Connection pools for failover between multiple ActiveMQs
• Lots of tunables about the connection pools such as retry frequencies etc
• SSL TLS between node and ActiveMQ

Writing Web and Dynamic UIs

• A DDL that describes agents, inputs and outputs:
• Creates auto generated documentation
• Can be used to auto generate user interfaces
• The client library will only make requests that validate against the DDL
• In future input validations will move into the DDL and will be done automatically for you
• Web UI’s can bypass or do their own discovery and use the DDL to auto generate user interfaces

Usability

• Fire-and-Forget style requests, for when you just want something done but do not care about results, these requests are very quick as they do not do any discovery.
• Agents can now be reloaded without restarting the daemon
• A new mc-inventory tool that can be used to view facts, agents and classes for a node
• Many UI enhancements to the CLI tools

Authorization plugins for MCollective SimpleRPC

by R.I. Pienaar | Apr 11, 2010 | Code

Till now The Marionette Collective has relied on your middleware to provide all authorization and authentication for requests. You’re able to restrict certain middleware users from certain agents, but nothing more fine grained.

In many cases you want to provide much finer grain control over who can do what, some cases could be:

• A certain user can only request service restarts on machines with a fact customer=acme
• A user can do any service restart but only on machines that has a certain configuration management class
• You want to deny all users except root from being able to stop services, others can still restart and start them

This kind of thing is required for large infrastructures with lots of admins all working in their own group of machines but perhaps a central NOC need to be able to work on all the machines, you need fine grain control over who can do what and we did not have this will now. It would also be needed if you wanted to give clients control over their own servers but not others.

Version 0.4.5 will have support for this kind of scheme for SimpleRPC agents. We wont provide a authorization plugin out of the box with the core distribution but I’ve made one which will be available as a plugin.

So how would you write an auth plugin, first a typical agent would be:

module MCollective
    module Agent
         class Service<RPC::Agent
             authorized_by :action_policy
 
             # ....
         end
    end
end

The new authorized_by keyword tells MCollective to use the class MCollective::Util::ActionPolicy to do any authorization on this agent.

The ActionPolicy class can be pretty simple, if it raises any kind of exception the action will be denied.

module MCollective
    module Util
         class ActionPolicy
              def self.authorize(request)
                  unless request.caller == "uid=500"
                      raise("You are not allow access to #{request.agent}::#{request.action}")
                  end
              end
         end
    end
end

This simple check will deny all requests from anyone but Unix user id 500.

It’s pretty simple to come up with your own schemes, I wrote one that allows you to make policy files like the one below for the service agent:

policy default deny
allow   uid=500 *                    *                *
allow   uid=502 status               *                *
allow   uid=600 *                    customer=acme    acme::devserver

This will allow user 500 to do everything with the service agent. User 502 can get the status of any service on any node. User 600 will be able to do any actions on machines with the fact customer=acme that also has the configuration management class acme::devserver on them. Everything else will be denied.

You can do multiple facts and multiple classes in a simple space separated list. The entire plugin to implement such policy controls was only 120 – heavy commented – lines of code.

I think this is a elegant and easy to use layer that provides a lot of functionality. We might in future pass more information about the caller to the nodes. There’s some limitations, specifically about the source of the caller information being essentially user provided so you need to keep that mind.

As mentioned this will be in MCollective 0.4.5.