Uncategorized | R.I.Pienaar

Thinking about monitoring frameworks

by R.I. Pienaar | Mar 19, 2011 | Uncategorized

I’ve been Tweeting a bit about some prototyping of a monitoring tool I’ve been doing and had a big response from people all agreeing something has to be done.

Monitoring is something I’ve been thinking about for ages but to truly realize my needs I needed mature discovery based network addressing and ways to initiate commands on large amounts of hosts in a parallel manner. I have this now in the MCollective and I feel I can start exploring some ideas of how I might build a monitoring platform.

I won’t go into all my wishes, but I’ll list a few big ones as far as monitoring is concerned:

Current tools represent a sliding scale, you cannot look at your monitoring tool and ever know current state. Reported status might be a window of 10 minutes and in some cases much longer.
Monitoring tools are where data goes to die. Trying to get data out of Nagios and into tools like Graphite, OpenTSDB or really anywhere else is a royal pain. The problem get much harder if you have many Nagios probes. NDO is an abomination as is storing this kind of data in MySQL. Commercial tools are orders of magnitude worse.
Monitoring logic is not reusable. Today with approaches like continuous deployment you need your monitoring logic to be reusable by many different parties. Deployers should be able to run the same logic on demand as your scheduled monitoring does.
Configuration is a nightmare of static text, or worse click driven databases. People mitigate this with CM tools but there is still a long turn around time from node creation to monitored. This is not workable in modern cloud based and dynamic systems.
Shops with skilled staff are constantly battling decades old tools if they want to extend it to create metrics driven infrastructure. It’s all just too ’90s.
It does not scale. My simple prototype can easily do 300+ checks a second, including processing replies, archiving, alert logic and feeding external tools like Graphite. On a GBP20/month virtual machine. This is inconceivable with most of the tools we have to deal with.

I am prototyping some ideas at the moment to build a framework to build monitoring systems with.

There’s a single input queue on a middleware system, I expect an event in this queue – mine is a queue distributed over 3 countries and many instances of ActiveMQ.

The event can come from many places maybe from a commit hook at GitHub, fed in from Nagios performance data or by MCollective or Pingdom, the source of data is not important at all. It’s just a JSON document that has some structure – you can send in any data in addition to a few required fields, it’ll happily store the lot.

From there it gets saved into a capped collection on MongoDB in its entirety and gets given an eventid. It gets broken into its status parts and its metric parts and sent to any number of recipient queues. In the case of Metrics for example I have something that feeds Graphite, you can have many of these all active concurrently. Just write a small consumer for a queue in any language and do with the events whatever you want.

In the case of statusses it builds a MongoDB collection that represents the status of an event in relation to past statusses etc. This will notice any state transition and create alert events, alert events again can go to many destinations – right now I am sending them to Angelia, but there could be many destinations with different filtering and logic for how that happens. If you want to build something to alert based on trends of past metric data, no problem. Just write a small plugin, in any language, and plug it into the message flow.

At any point through this process the eventid is available and should you wish to get hold of the original full event its a simple lookup away – there you can find all the raw event data that you sent – stored for quick retrieval in a schemaless manner.

In effect this is a generic plugable event handling system. I currently feed it from MCollective using a modified NRPE agent and I am pushing my Nagios performance data in real time. I have many Nagios servers distributed globally and they all just drop events into a their nearest queue entry point.

Given that it’s all queued and persisted to disk I can create really vast amount of alerts using MCollective – it’s trivial for me to create 1000 check results a second. The events have the timestamp attached of when the check was done and even if the consumers are not keeping up the time series databases will get the events in the right order and right timestamps. So far on a small VM that runs Puppetmaster, MongoDB, ActiveMQ, Redmine and a lot of other stuff I am very comfortably sending 300 events a second through this process without even tuning or trying to scale it.

When I look at a graph of 50 servers load average I see the graph change at the same second for all nodes – because I have an exact single point in time view of my server estate, and what 50 servers I am monitoring in this manner is done using discovery on MCollective. Discovery is obviously no good for monitoring in general – you dont know the state of stuff you didn’t discover – but MCollective can build a database of truth using registration – correlate discovery against registration and you can easily identify missing things.

A free side effect of using an async queue is that horizontal scaling comes more or less for free, all I need to do is start more processes consuming the same queue – maybe even on a different physical server – and more capacity becomes available.

So this is a prototype, its not open source – I am undecided what I will do with it, but I am likely to post some more about its design and principals here. Right now I am only working on the event handling and routing aspects as the point in time monitoring is already solved for me as is my configuration of Nagios, but those aspects will be mixed into this system in time.

There’s a video of the prototype receiving monitor events over mcollective and feeding Loggly for alerts here.

Debugging MCollective Agents

by R.I. Pienaar | Mar 9, 2011 | Uncategorized

While developing agents for The Marionette Collective it’s often quite difficult to debug weird behavior in agents, I wrote a debugger that makes development easier.

Agents tend to run in a thread inside a daemon connected through middleware, this all makes it really hard. The debugger is a harness that runs an agent standalone allowing you to trace calls, set breakpoints and all the other goodies you expect in a good debugger.

To use it you need to grab the source from GitHub and also install the ruby-debug Gem.

Since I am using the normal ruby-debug debugger for the hard work you can read any of the many tutorials and screencasts about its use and apply what you lean in this environment, the screencast below shows you a quick tour through the features and usage. The screencast is high quality so feel free to full-screen it.

As the screencast mention there’s still some tweaking to do so that the ruby-debug will notice code changes without restarting and I might tweak the trace option a bit. Already though this is a huge improvement for anyone writing an Agent.

Using MCollective 1.1.3 Subcollectives for Security

by R.I. Pienaar | Mar 5, 2011 | Uncategorized

We’ll be releasing The Marionette Collective version 1.1.3 on Monday which will bring with it a major new feature called Subcollectives. This feature lets you partition your collective into multiple isolated broadcast zones much like a VLAN does to a traditional network. Each node can belong to one or many collectives at the same time.

An interesting side effect of this feature is that you can create subcollectives to improve security of your network. I’ll go through a process here for providing untrusted 3rd parties access to just a subset of your servers.

The image above demonstrates a real world case where a customer wanted to control their machines using the abilities exposed by MCollective on a network hosting servers for many customers.

The customer has a CMS that creates accounts on his backend systems, sometimes he detects abuse from a certain IP and need to be able to block that IP from all his customer facing systems immediately. We did not want to give the CMS access to SSH as root to the servers to we provided a MCollective Agent that expose this ability using SimpleRPC.

Rather than deploy a new collective using different daemons we use the new Subcollectives features to let the customer machines belong to a new collective called custcollective while still belonging to the existing collective. We then restrict the user at the middleware layer and set his machines up to allow him access to them via the newly created collective.

To properly secure this setup we give the customer their own username on the ActiveMQ server and secure it so it can only communicate with his subcollective:

<simpleAuthenticationPlugin>
  <users>
    <authenticationUser username="customer" password="secret" groups="customer,everyone"/>
  </users>
</simpleAuthenticationPlugin>
 
<authorizationPlugin>
  <map>
    <authorizationMap>
        <authorizationEntries>
          <authorizationEntry topic="custcollective.>" write="mcollectiveusers,customer" read="mcollectiveusers,customer" admin="mcollectiveusers,genzee" />
        </authorizationEntries>
    </authorizationMap>
  </map>
</authorizationPlugin>

This sets up the namespace for the custcollective and give the user access to it, we only give him access to his collective and no others.

Next we have to configure the customers servers to belong to the new collective in addition to the existing collective using their server.cfg:

collectives = collective,custcollective
main_collective = collective

And finally we give the customer a client.cfg that limits him to this collective:

collectives = custcollective
main_collective = custcollective
 
plugin.stomp.pool.user1 = customer
plugin.stomp.pool.password1 = secret

Due to the restrictions on the middleware level even if the customer were to specify other collective names in his client.cfg he simply would not be able to communicate with those hosts.

We now setup Authorization to give the user access to just the agents and actions you authorize him to communicate with. A sample policy file using the Action Policy Authorization Plugin can be seen below, it lets the user use the iptables agent block action on just his machines while allowing me to use all actions on all machines:

policy default deny
allow   cert=rip        *       *               *
allow   cert=customer   block   customer=acme   *

And finally thanks to the Auditing built into MCollective the clients actions are fully logged:

2011-03-05T21:03:52.598552+0000: reqid=ebf3c01fdaa92ce9f4137ad8ff73336b: 
reqtime=1299359032 caller=cert=customer@some.machine agent=iptables 
action=block data={:ipaddr=>"196.xx.xx.146"}

The customer is happy as he was able to build a real time IDS that reacts to events throughout his network, he can interact with it from CLI, automated IDS and even his web systems.

Using this technique and combining it with the existing AAA in MCollective we as an ISP were able to expose a subset of machines to an untrusted 3rd party in a way that is safe, secure and audited without having to give the 3rd party elevated or even shell access to these machines.

MCollective 1.0.1 demo using AWS CloudFormation

by R.I. Pienaar | Feb 26, 2011 | Uncategorized

Amazon is keeping things ticking along nicely by constantly adding features to their offerings. I’m almost more impressed at the pace and diversity of innovation than the final solution.

During the week they announced AWS CloudFormation. Rather than add to the already unbearable tedium of “it’s not a Chef or Puppet killer” blog posts I thought I’d just go ahead and do something with it.

Till now people who wanted to evaluate MCollective had to go through a manual process of starting first the ActiveMQ instance, gathering some data and then start a number of other instances supplying user data for the ActiveMQ instance. This was by no means a painful solution but CloudFormation can make this much better.

I’ve created a new demo using CloudFormation and recorded a video showing how to use it etc, you can read all about it here.

The demo has been upgraded with the latest production MCollective version that came out recently. This collective has the same features as the previous demos, registration, package and a few other bits.

Impact
Overall I think this is a very strong entry into the market, it needs work still but its a step in the right direction. I dislike typing JSON about as much as I dislike typing XML but this isn’t an Amazon problem to fix – that’s what frameworks and API clients are for.

It’s still markup aimed at machines and the following pretty much ensures user error as much as XML does:

 "UserData" : { 
            "Fn::Base64" : { 
                "Fn::Join" : [ ":", [ 
                    "PORT=80", 
                    "TOPIC=", {
                        "Ref" : "logical name of an AWS::SNS::Topic resource" 
                    }, 
                    "ACCESS_KEY=", { "Ref" : "AccessKey" },
                    "SECRET_KEY=", { "Ref" : "SecretKey" } ] 
                ] 
            } 
         },

CloudFormation represents a great opportunity for the Framework builders like Puppet Labs and Opscode as it can enhance their offerings by a long way especially for Puppet a platrform wide view is something that is very desperately needed – not to mention basic Cloud integration.

Tools like Fog and its peers will no doubt soon support this feature so will knife as a side effect.

Issues
I have a few issues with the current offering, it seems a bit first-iteration like and I have no doubt things will improve. The issues I have are fairly simple ones and I am surprised they weren’t addressed in the first release really.

Fn::GetAtt is too limited
You can gain access to properties of other resources using the Fn::GetAtt function, for instance say you created a RDS database and need its IP address:

"Fn::GetAtt" : [ "MyDB" , "Endpoint.Address"]

This is pretty great but unfortunately the list of resources it can access is extremely limited.

For example, given an EC2 image you might want to find out it’s private IP address – what if it’s offering an internal service like my ActiveMQ instances does to MCollective? You can’t get access to this the only attribute that is available is the public IP address. This is a problem, if you talk to that you will get billed even between your own instances! So you have to do a PTR lookup on the IP and then use the public DNS name or do another lookup and rely on the Amazon split horizon DNS to direct you to the private IP. This is unfortunate since it ads a lot of error prone steps to the process.

This situation is repeated for more or less every resource that CloudFormation can manage. Hopefully this situation will improve pretty rapidly.

Lack of options
When creating a stack it seems obvious you might want to create 2 webservers, at the moment – unless I am completely blind and missed something – you have to specify each instance completely rather than have a simple property that instructs it to make multiples of the same resource.

This seems a very obvious omission, it’s such a big one that I am sure I just missed something in the documentation.

Some stuff just don’t work
I’ll preface this by again saying I might just not be getting something. You’re supposed to be able to create a Security Group that allows all machines in another Security Group to communicate with it.

The documentation says you can set SourceSecurityGroupName and then you don’t have to set SourceSecurityGroupOwnerId.

Unfortunately try as I might I can’t get it to stop complaining that SourceSecurityGroupOwnerId isn’t set when I set SourceSecurityGroupName which is just crazy talk since there’s no way to look up the current Owner ID in any GetAtt property.

Additionally it claims the FromPort and ToPort properties are compulsory but all the docs on the APIs says you cannot set those if you set the SourceSecurityGroupName in the individual APIs

I’ve given up making proper security group the way I want them for the purpose of my demo but I am fairly sure this is a bug.

Slow
If you have a Stack with 10 resources it will do some ordering based on your ref’s, for example if 5 other instances requires the public IP of a 6th it will create the right one first.

Unfortunately even though there then is no reason for things to happen in a given order it will just sit and create the resources one by one in a serial manner rather than start all the requests in parallel.

With even a reasonably complex Stack this can be very tedious, starting a 6 node Stack can take 15 minutes easily. It then also shuts the stack down in series so just booting and shutting it wastes 30 minutes on 6 nodes!

Definitely some room for improvement here, I’d like to give developers a self service environment they won’t enjoy sitting waiting for 1/2 hour before they can get going.

Shoddy Documentation
I’ve come across a number of documentation inconsistencies that can be really annoying. Little things like PublicIP vs PublicIp that makes writing the JSON files a error prone cycle of try and try and try. This is a very easy thing to fix it’s only worth mentioning in relation to the next point.

Given how slow it is to create/tear down stacks if you got something wrong this problem can really hurt your productivity.

The AWS Console
Given that the docs are a bit bad you’ll be spending some time in the AWS console or the CLI tool. I didn’t try to the ClI tools in this case but the console is not great at all.

I am seeing weird things where I upload a new template into a new Formation and it gets an older one at first. Looking at how it works – saving the JSON to a bucket under your name all with unique names I chalked this down to user error. But then I tried harder not to mess it up and it does seem to keep happening.

I’m still not quite ready to blame the AWS console for this though, might be some browser caching or something else to blame, either way it makes the whole thing a lot more frustrating to use.

What I am 100% ready to blame it for is just general unfriendlyness with error messages and I am guessing the feedback you’d get from the API is equally bad:

"Invalid template resource property IP"

Needless to say the string ‘IP’ isn’t even anywhere to be found in my template.

When I eventually tracked this down it was due to the caching mentioned above working on an old JSON file and a user error on my side, but I didn’t see it because it wasn’t clear it was using a old JSON file and not the one I set on my browser upload.

So my recommendation is just not to use the AWS console while creating stacks, it’s an end user tool and excels at that, when building stacks use the CLI as it includes tools to do local validations of the JSON and avoid annoying browser caches etc.

Testing MCollective Agents In Development

by R.I. Pienaar | Jan 6, 2011 | Uncategorized

When developing plugins for MCollective testing can be a pain since true testing would mean you need a lot of virtual machines and so forth.

I wrote something called the MCollective Collective Builder that assist in starting up a Stomp server and any number of mcollectived processes on one machine which would let you develop and test the parallel model without needing many machines.

I’ve found it to be very useful in developing plugins, I recently wrote a new encrypting security plugin and the entire development was done over 10 nodes all on a single 512MB RAM single CPU virtual machine with no complex plugin synchronization issues etc.

You can see how to use it below, as usual it’s best to view the video full screen.

Git hook for redmine tickets

by R.I. Pienaar | Dec 28, 2010 | Uncategorized

I use git and like to have a ticket for every bit of work I do on my projects. I also have a preferred way to prefix commit messages, something like:

123 - Fix this broken thing
 
Description of commits here

Where Fix the broken thing is the ticket subject in RedMine.

It’s not sophisticated or complex but it shows a nice one-liner in GitHub which I like.

To make sure I always stay consistent I have a simple git hook to tweak the commit message:

#!/usr/bin/env ruby
 
require 'rubygems'
require 'httparty'
 
REDMINEURL="http://your.redmine"
 
msg = File.read(ARGV[0])
 
# If the msg starts with number we've been here already
# and no doubt someone is amending existing log entries
exit if msg =~/^\d/
 
begin
    if msg =~ /On branch (\d+)(.+?)\n/
        branch = $1
        ticketurl = "#{REDMINEURL}/issues/#{branch}.json"
        puts "Fetching #{ticketurl}"
        headers = {"User-Agent" => "meh"}
 
        ticket = HTTParty.get(ticketurl, {:headers => headers})
 
        msg = "%d - %s\n\n%s" % [ branch, ticket["issue"]["subject"], msg ]
 
        File.open(ARGV[0], "w") {|f| f.print msg}
    end
rescue Exception => e
    puts "git hook failed: #{e.class}: #{e}"
    sleep 2
    exit
end

It requires that I name my branches something like <ticket number>_anything_else which might be too limited but works for me.

This will prepare my commit message with the ticket subject before I commit. I don’t do much off-line coding so generally don’t have to worry about connectivity checks.