![](\"http:\/\/www.devco.net\/images\/um_puppet_metrics.png\")
The end result is that my previously mentioned event system<\/a> is now getting messages sent the moment Puppet does anything. And I have callbacks at the event system layer so I can now instantly react to change anywhere on my network. <\/p>\nIf a new webserver in a certain cluster comes up I can instantly notify the load balancer to add it – perhaps via a Puppet run of of its own. This lets me build reactive cross node orchestration. Apart from the callbacks I also get run metrics in Graphite as seen in the image.<\/p>\n
The best way I found to make Puppet do this is by creating your own transaction reports processor that is the client side of a normal puppet report, here’s a shortened version of what I did. It’s overly complex and a pain to get working since Puppet Pluginsync still cannot sync out things like Applications.<\/p>\n
The code overrides the finalize_report<\/em> method to send the final run metrics and the add_resource_status<\/em> method to publish events for changing resources. Puppet will use the add_resource_status<\/em> method to add the status of a resource right after evaluating that resource. By tapping into this method we can send events to the network the moment the resource has changed.<\/p>\n <\/code><\/p>\n Finally as I really have no need for sending reports to my Puppet Masters I created a small Application that replace the standard agent. This application has access to the report even when reporting is disabled so it will never get saved to disk or copied to the masters. <\/p>\n This application sets up the report using the class above and creates a log destination that feeds the logs into it. This is more or less exactly the Puppet::Application::Agent<\/em> so I retain all my normal CLI usage like --tags<\/em> and --test<\/em> etc.<\/p>\n <\/code><\/p>\n And finally I can now create a callback in my event system this example is over simplified but the gist of it is that I am triggering a Puppet run on my machines with class roles::frontend_lb<\/em> as a result of a web server starting on any machine with the class roles::frontend_web<\/em> - effectively immediately causing newly created machines to get into the pool. The Puppet run is triggered via MCollective so I am using it's discovery capabilities to run all the instances of load balancers.<\/p>\n <\/code><\/p>\n Doing this via a Puppet run demonstrates to me where the balance lie between Orchestration and CM. You still need to be able to build a new machine, and that new machine needs to be in the same state as those that were managed using the Orchestration tool. So by using MCollective to trigger Puppet runs I know I am not doing anything out of bounds of my CM system, I am simply causing it to work when I want it to work.<\/p>\n","protected":false},"excerpt":{"rendered":" I’ve wanted to be notified the moment Puppet changes a resource for ages, I’ve often been told this cannot be done without monkey patching nasty Puppet internals. Those following me on Twitter have no doubt noticed my tweets complaining about the complexities in getting this done. I’ve now managed to get this done so am […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[1],"tags":[121,85,21,13,96],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2165"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=2165"}],"version-history":[{"count":23,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2165\/revisions"}],"predecessor-version":[{"id":2188,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2165\/revisions\/2188"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=2165"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=2165"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=2165"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<\/p>\n\r\nrequire 'puppet\/transaction\/report'\r\n\r\nclass Puppet::Transaction::UnimatrixReport < Puppet::Transaction::Report\r\n def initialize(kind, configuration_version=nil)\r\n super(kind, configuration_version)\r\n\r\n @um_config = YAML.load_file(\"\/etc\/puppet\/unimatrix.yaml\")\r\n end\r\n\r\n def um_connection\r\n @um_connection ||= Stomp::Connection.new(@um_config[:user], @um_config[:password], @um_config[:server], @um_config[:port], true)\r\n end\r\n\r\n def um_newevent\r\n event = { ... } # skeleton event, removed for brevity\r\n end\r\n\r\n def finalize_report\r\n super\r\n\r\n metrics = um_newevent\r\n sum = raw_summary\r\n \r\n # add run metrics from raw_summary to metrics, removed for brevity\r\n\r\n Timeout::timeout(2) { um_connection.publish(@um_config[:portal], metrics.to_json) }\r\n end\r\n\r\n def add_resource_status(status)\r\n super(status)\r\n\r\n if status.changed?\r\n event = um_newevent\r\n\r\n # add event status to event hash, removed for brevity\r\n\r\n Timeout::timeout(2) { um_connection.publish(@um_config[:portal], event.to_json) }\r\n end\r\n end\r\nend<\/pre>\n<\/p>\n\r\nrequire 'puppet\/application'\r\nrequire 'puppet\/application\/agent'\r\nrequire 'rubygems'\r\nrequire 'stomp'\r\nrequire 'json'\r\n\r\nclass Puppet::Application::Unimatrix < Puppet::Application::Agent\r\n run_mode :unimatrix\r\n\r\n def onetime\r\n unless options[:client]\r\n $stderr.puts \"onetime is specified but there is no client\"\r\n exit(43)\r\n return\r\n end\r\n\r\n @daemon.set_signal_traps\r\n\r\n begin\r\n require 'puppet\/transaction\/unimatrixreport'\r\n report = Puppet::Transaction::UnimatrixReport.new(\"unimatrix\")\r\n\r\n # Reports get logs, so we need to make a logging destination that understands our report.\r\n Puppet::Util::Log.newdesttype :unimatrixreport do\r\n attr_reader :report\r\n\r\n match \"Puppet::Transaction::UnimatrixReport\"\r\n\r\n def initialize(report)\r\n @report = report\r\n end\r\n\r\n def handle(msg)\r\n @report << msg\r\n end\r\n end\r\n\r\n @agent.run(:report => report)\r\n rescue => detail\r\n puts detail.backtrace if Puppet[:trace]\r\n Puppet.err detail.to_s\r\n end\r\n\r\n if not report\r\n exit(1)\r\n elsif options[:detailed_exitcodes] then\r\n exit(report.exit_status)\r\n else\r\n exit(0)\r\n end\r\n end\r\nend\r\n<\/pre>\n
<\/p>\n\r\nadd_callback(:name => \"puppet_change\", :type => [\"archive\"]) do |event|\r\n data = event[\"extended_data\"]\r\n\r\n if data[\"resource_title\"] && data[\"resource_type\"]\r\n if event[\"name\"] == \"Service[httpd]\"\r\n # only when its a new start, this is short for brevity you want to do some more checks\r\n if data[\"restarted\"] == false && data[\"tags\"].include?(\"roles::frontend_web\")\r\n # does a puppet run via mcollective on the frontend load balancer\r\n UM::Util.run_puppet(:class_filter => \"roles::frontend_lb\")\r\n end\r\n end\r\n end\r\nend\r\n<\/pre>\n