Ruby Plugin Architectures

by R.I. Pienaar | Dec 1, 2009 | Uncategorized

Most of the applications I write in Ruby are some kind of Framework, ruby-pdns takes plugins, mcollective takes plugins, my nagios notification bot takes plugins etc, yet I have not yet figured out a decent approach to handling plugins.

Google suggests many options, the most suggested one is something along these lines.

class Plugin
    def self.inherited(klass)
        PluginManager << klass.new
    end
end
 
class FooPlugin<Plugin
end

Where PluginManager is some class or module that stores and later allows retrieval, when the FooPlugin class gets created it will trigger the hook in the base class.

This works ok, almost perfectly, except that at the time of the trigger the FooPlugin class is not 100% complete and your constructor will not be called, quite a pain. From what I can tell it calls the constructor on either Class or Object.

I ended up tweaking the pattern a bit and now have something that works well, essentially if you pass a String to the PluginManager it will just store that as a class name and later create you an instance of that class, else if it’s not a string it will save it as a fully realized class assuming that you know what you did.

The full class is part of mcollective and you can see the source here but below the short version:

I am quite annoyed that including a module does not also include static methods in Ruby, its quite a huge miss feature in my view and there are discussions about changing that behavior. I had hopes of writing something simple that I can just do include Pluggable and this would set up all the various bits, create the inherited hook etc, but it’s proven to be a pain and would be littered with nasty evals etc.

module PluginManager
   @plugins = {}
 
   def self.<<(plugin)
         type = plugin[:type]
         klass = plugin[:class]
 
         raise("Plugin #{type} already loaded") if @plugins.include?(type)
 
         if klass.is_a?(String)
             @plugins[type] = {:loadtime => Time.now, :class => klass, :instance => nil}
         else
             @plugins[type] = {:loadtime => Time.now, :class => klass.class, :instance => klass}
         end
   end
 
   def self.[](plugin)
       raise("No plugin #{plugin} defined") unless @plugins.include?(plugin)
 
       # Create an instance of the class if one hasn't been done before
       if @plugins[plugin][:instance] == nil
            begin
                 klass = @plugins[plugin][:class]
                 @plugins[plugin][:instance] = eval("#{klass}.new")
            rescue Exception => e
                raise("Could not create instance of plugin #{plugin}: #{e}")
            end
       end
 
       @plugins[plugin][:instance]
   end
end
 
class Plugin
   def self.inherited(klass)
      PluginManager << {:type => "facts_plugin", :class => klass.to_s}
   end
end
 
class FooPlugin<Plugin
end

For mcollective I only ever allow one of a specific type of plugin so the code is a bit specific in that regard.

I think late creating the plugin instances is quite an improvement too since often you’re loading in plugins that you just don’t need like client apps would probably not need a few of the stuff I load in and creating instances is just a waste.

I am not 100% sold on this approach as the right one, I think I’ll probably refine it more and would love to hear what other people have done.

This has though removed a whole chunk of grim code from mcollective since I now store all plugins and agents in here and just fetch them as needed. So already this is an improvement to what I had before so I guess it works well and should be easier to refactor for improvements now.

The Marionette Collective

by R.I. Pienaar | Nov 4, 2009 | Uncategorized

Some time ago I posted a blog post and screencast about my middleware solution to concurrent systems administration, I had a lot of interest in this and in the end relented and released the thing as Open Source. You should go watch the screencast to really grasp what is going on and what this achieves.

I asked around on Twitter for name suggestions and got many great suggestions, in the end I settled on mcollective short for The Marionette Collective, yeah it sounds like an Emo band, I don’t care. It’ll be easy to google and I got a matching .org domain, that helps over certain other ungoogleable project names!

The code for the core service is released under the Apache License version 2 and is available at google code: http://marionette-collective.org/

There’s a wiki getting started guide and also a quick guide on writing agents and CLI tools to talk to it.

By default the code includes the following plugins:

• Security by means of a Pre Shared Key
• Communications with Stomp compatible servers
• Facts about machines via a simple YAML file
• Discovery

You can drop in replacement agents for all of the above and perhaps write something to use full SSL encryption etc. No actual agents to do any real work is included in the core Apache Licensed code.

A second project was started where I’m dropping a bunch of my own agents, the projects are separate because the agents might have different licensing from the core app server, for example there’s an agent to use Facter from Reductive Labs for facts but this means the code has to be GPL.

Agents available right now lets you use facter for facts, manage services using the puppet providers and also do distributed url benchmarks, check out the plugins project: http://code.google.com/p/mcollective-plugins/

I’d encourage others to put agent on Github or wherever and to announce it on the users list or just to drop me a mail and I’ll link to it from the project wiki – you can also join one of the projects and just commit your code yourself.

It’s still early days – I was accused of being too perfectionist in how I like to release code, so this is very much an early and often approach to releasing! The entire code base is about 3 weeks old and I spent mostly some free time hacking it up, so there’s much improvement to be made still, I do however use a version of it in production and find it very stable and reliable so far.

I am looking for early testers to give me feedback about the code, structure of the project etc. If you’re stuck grab me on freenode my nick is Volcane and I’ll see if I can help you get it going.