{"id":2095,"date":"2011-05-28T11:10:06","date_gmt":"2011-05-28T10:10:06","guid":{"rendered":"http:\/\/www.devco.net\/?p=2095"},"modified":"2011-08-03T18:22:24","modified_gmt":"2011-08-03T17:22:24","slug":"pluggable_extlookup_for_puppet","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2011\/05\/28\/pluggable_extlookup_for_puppet.php","title":{"rendered":"Pluggable Extlookup for Puppet"},"content":{"rendered":"<p><B>NOTE:<\/B> This ended up being a proof of concept for a more complete system called <a href=\"http:\/\/www.devco.net\/archives\/tag\/hiera\">Hiera<\/a> please consider that instead.<\/p>\n<p>Back in 2009 I wrote the first <a href=\"http:\/\/www.devco.net\/archives\/2009\/08\/31\/complex_data_and_puppet.php\">implementation of extlookup for Puppet<\/a> later on it got merged &#8211; after a much needed rewrite &#8211; into Puppet mainstream.  If you don&#8217;t know what extlookup does please go and read that post first.<\/p>\n<p>The hope at the time was that someone would make it better and not just a hacky function that uses global variables for its config.  I was exploring some ideas and showing how rich data would apply to the particular use case and language of Puppet but sadly nothing has come of these hopes. <\/p>\n<p>The complaints about extlookup fall into various categories:<\/p>\n<ul>\n<li>CSV does not make a good data store<\/li>\n<li>I have a personal hate for the global variable abuse in extlookup, I was hoping Puppet config items will become pluggable at some point, alas.<\/li>\n<li>Using functions does not let you introspect the data usage inside your modules for UI creation<\/li>\n<li>Other complaints fall in the &#8216;Not Invented Here&#8217; category and the &#8216;TL; DR&#8217; category of people who simply didn&#8217;t bother understanding what extlookup does<\/li>\n<\/ul>\n<p>The complaint about using functions to handle data not being visible to external sources is valid.  Puppet has not made introspection of classes and their parameters easy for ENCs yet so this just seems to me like people who don&#8217;t understand that extlookup is simply a data model not a prescription for how to use the data.  In a follow up post I will show an extlookup based <a href=\"http:\/\/docs.puppetlabs.com\/guides\/external_nodes.html\">ENC<\/a> that supports parametrized classes and magical data resolution for those parametrized classes using the exact same extlookup data store and precedence rules.<\/p>\n<p>Not much to be done for the last group of people but as @jordansissel said &#8220;haters gonna hate, coders gonna code!&#8221;.   <\/p>\n<p>I have addressed the first complaint now by making an extlookup that is pluggable so you can bring different backends.<\/p>\n<p>First of course, in bold defiance of the Ruby Way, I made it backward compatible with older versions of extlookup and gave it a 1:1 compatible CSV backend. <\/p>\n<p>I addressed my global variable hate by adding a config file that might live in <em>\/etc\/puppet\/extlookup.yaml<\/em>.  <\/p>\n<h2>Status<\/h2>\n<p>I wrote this code yesterday afternoon, so you should already guess that there might be bugs and some redesigns ahead and that it will most likely destroy your infrastructure.  I will add unit tests to it etc so please keep an eye on it and it will become mature for sure.  <\/p>\n<p>I have currently done backends for CSV, YAML and Puppet manifests.  A JSON one will follow and later perhaps one querying Foreman and other data stores like that.<\/p>\n<p>The code lives at <a href=\"https:\/\/github.com\/ripienaar\/puppet-extlookup\">https:\/\/github.com\/ripienaar\/puppet-extlookup<\/a>.<\/p>\n<h2>Basic Configuration<\/h2>\n<p>Configuration of precedence is a setting that applies equally to all backends, the config file should live in the same directory as your <em>puppet.conf<\/em> and should be called <em>extlookup.yaml<\/em>.  Examples of it below.<\/p>\n<h2>CSV Backend<\/h2>\n<p>The CSV backend is backward compatible, it will also respect your old style global variables for configuration &#8211; but the other backends wont.  To configure it simply put something like this in your config file:<\/p>\n<p><code><\/p>\n<pre lang=\"yaml\">\r\n---\r\n:parser: CSV\r\n:precedence:\r\n- environment_%{environment}\r\n- common\r\n:csv:\r\n  :datadir: \/etc\/puppet\/extdata\r\n<\/pre>\n<p><\/code><\/p>\n<h2>YAML Backend<\/h2>\n<p>The most common proposed alternatives to extlookup seem to be YAML based.   The various implementations out there though are pretty weak and seemed to get bored with the idea before reaching feature parity with extlookup.  With a plugable backend it was easy enough for me to create a YAML data store that has all the extlookup features. <\/p>\n<p>In the case of simple strings being returned I have kept the extlookup feature that parses variables like <em>%{country}<\/em> in the result data out from the current scope &#8211; something mainline puppet extlookup actually broke recently in a botched commit &#8211; but if you put hash or array data in the YAML files I don&#8217;t touch the data.<\/p>\n<p>Sample data:<\/p>\n<p><code><\/p>\n<pre lang=\"yaml\">\r\n--- \r\ncountry: uk\r\nntpservers: \r\n- 1.uk.pool.ntp.org\r\n- 2.uk.pool.ntp.org\r\nfoo.com: \r\n  contact: webmaster@foo.com\r\n  docroot: \/var\/www\/foo.com\r\n<\/pre>\n<p><\/code><\/p>\n<p>All of this data is accessible using the exact same extlookup function.  Configuration of the YAML backend:<\/p>\n<p><code><\/p>\n<pre lang=\"yaml\">\r\n---\r\n:parser: YAML\r\n:precedence:\r\n- environment_%{environment}\r\n- common\r\n:yaml:\r\n  :datadir: \/etc\/puppet\/extdata\r\n<\/pre>\n<p><\/code><\/p>\n<h2>Puppet Backend<\/h2>\n<p>Nigel Kersten has been working on the proposal of a <a href=\"http:\/\/projects.puppetlabs.com\/issues\/6079\">new data format called the PDL<\/a>.   I had pretty high hopes for the initial targeted feature list but now it seems to have been watered down to a <a href=\"https:\/\/github.com\/nigelkersten\/puppet-get\">minimal feature set extlookup with a different name and backend<\/a>.<\/p>\n<p>I implemented the proposed data lookup in classes and modules as a extlookup backend and made it full featured to what you&#8217;d expect from extlookup &#8211; full configurable lookup orders and custom overrides.  Just like we&#8217;ve had for years in the CSV version.<\/p>\n<p>Personally I think if you&#8217;re going to spend hours creating data that describes your infrastructure you should:<\/p>\n<ul>\n<li>Not stick it in a language that&#8217;s already proven to be bad at dealing with data<\/li>\n<li>Not stick it in a place where nothing else can query the data<\/li>\n<li>Not stick it in code that requires code audits for simple data changes &#8211; as most change control boards really just won&#8217;t see the difference.<\/li>\n<li>Not artificially restrict what kind of data can go into the data store by prescribing a unmovable convention with no configuration.<\/li>\n<\/ul>\n<p>When I show the extlookup based ENC I am making I will really show why putting data in the Puppet Language is like a graveyard for useful information and not actually making anything better.  <\/p>\n<p>You can configure this backend to behave exactly the way Nigel designed it using this config file:<\/p>\n<p><code><\/p>\n<pre lang=\"yaml\">\r\n---\r\n:parser: Puppet\r\n:precedence:\r\n- %{calling_class}\r\n- %{calling_module}\r\n:puppet:\r\n   :datasource: data\r\n<\/pre>\n<p><\/code><\/p>\n<p>Which will lookup data in these classes:<\/p>\n<ul>\n<li>data::$calling_class<\/li>\n<li>data::$calling_module<\/li>\n<li>$calling_class::data<\/li>\n<li>$calling_module::data<\/li>\n<\/ul>\n<p>Or you can do better and configure proper precedence which would replace the 1st 2 above with ones for datacenter, country, whatever.  The last 2 will always be in the list.  An alternative might be:<\/p>\n<ul>\n<li>data::$customer<\/li>\n<li>data::$environment<\/li>\n<li>$calling_class::data<\/li>\n<li>$calling_module::data<\/li>\n<\/ul>\n<p>You could just configure this behavior with the extlookup precedence setting.  Pretty nice for those of you feeling nostalgic for <em>config.php<\/em> files as hated by Sysadmins everywhere.<\/p>\n<p>And as you can see you can also configure the initial namespace &#8211; <em>data<\/em> &#8211; in the config file.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>NOTE: This ended up being a proof of concept for a more complete system called Hiera please consider that instead. Back in 2009 I wrote the first implementation of extlookup for Puppet later on it got merged &#8211; after a much needed rewrite &#8211; into Puppet mainstream. If you don&#8217;t know what extlookup does please [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[1],"tags":[121,85,21],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2095"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=2095"}],"version-history":[{"count":17,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2095\/revisions"}],"predecessor-version":[{"id":2224,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2095\/revisions\/2224"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=2095"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=2095"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=2095"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}