Puppet backend for Hiera part 2

by R.I. Pienaar | Jun 11, 2011 | Uncategorized

Note: This project is now being managed by Puppetlabs, its new home is http://projects.puppetlabs.com/projects/hiera

Last week I posted the first details about my new data source for Hiera that enables its use in Puppet.

In that post I mentioned I want to do merge or array searches in the future. I took a first stab at that for array data and wanted to show how that works. I also mentioned I wanted to write a Hiera External Node Classifier (ENC) but this work completely makes that redundant now in my mind.

A common pattern you see in ENCs are that they layer data – very similar in how extlookup / hiera has done it – but that instead of just doing a first-match search they combine the results into a merged list. This merged list is then used to include the classes on the nodes.

For a node in the production environment located in dc1 you’ll want:

node default {
   include users::common
   include users::production
   include users::dc1
}

I’ve made this trivial in Hiera now, given the 3 files below:

common.json

{"classes":"users::common"}

production.json

{"classes":"users::production"}

dc1.json

{"classes":"users::dc1"}

And appropriate Hiera hierarchy configuration you can achieve this using the node block below:

node default {
   hiera_include("classes")
}

Any parametrized classes that use Hiera as in my previous post will simply do the right thing. Individual classes variables can be arrays so you can include many classes at each tier. Now just add a role fact on your machines, add a role tier in Hiera and you’re all set.

The huge win here is that you do not need to do any stupid hacks like load the facts from the Puppet Masters vardir in your ENC to access the node facts or any of the other hacky things people do in ENCs. This is simply a manifest doing what manifests do – just better.

The hiera CLI tool has been updated with array support, here is it running on the data above:

$ hiera -a classes
["users::common"]
$ hiera -a classes environment=production location=dc1
["users::common", "users::production", "users::dc1"]

I’ve also added a hiera_array() function that takes the same parameters as the hiera() function but that returns an array of the found data. The array capability will be in Hiera version 0.2.0 which should be out later today.

I should also mention that Luke Kanies took a quick stab at integrating Hiera into Puppet and the result is pretty awesome. Given the example below Puppet will magically use Hiera if it’s available, else fall back to old behavior.

class ntp::config($ntpservers="1.pool.ntp.org") {
    .
    .
}
 
node default {
   include ntp::config
}

With Lukes proposed changes this would be equivalent to:

class ntp::config($ntpservers=hiera("ntpservers", "1.pool.ntp.org")) {
    .
    .
}

This is pretty awesome. I wouldn’t hold my breath to see this kind of flexibility soon in Puppet core but it shows whats possible.

Puppet backend for Hiera

by R.I. Pienaar | Jun 6, 2011 | Uncategorized

Note: This project is now being managed by Puppetlabs, its new home is http://projects.puppetlabs.com/projects/hiera

Yesterday I released a Hierarchical datastore called Hiera, today as promised I’ll show how it integrates with Puppet.

Extlookup has solved the basic problem of loading data into Puppet. This was done 3 years ago at a time before Puppet supported complex data in Hashes or things like Parametrized classes. Now as Puppet has improved a new solution is needed. I believe the combination of Hiera and the Puppet plugin goes a very long way to solving this and making parametrized classes much more bearable.

I will highlight a sample use case where a module author places a module on the Puppet Forge and a module user downloads and use the module. Both actors need to create data – the author needs default data to create a just-works experience and the module user wants to configure the module behavior either in YAML, JSON, Puppet files or anything else he can code.

Module Author

The most basic NTP module can be seen below. It has a ntp::config class that uses Hiera to read default data from ntp::data:

modules/ntp/manifests/config.pp

class ntp::config($ntpservers = hiera("ntpservers")) {
    file{"/tmp/ntp.conf":
        content => template("ntp/ntp.conf.erb")
    }
}

modules/ntp/manifests/data.pp

class ntp::data {
    $ntpservers = ["1.pool.ntp.org", "2.pool.ntp.org"]
}

This is your most basic NTP module. By using hiera(“ntpserver”) you load $ntpserver from these variables, the first one that exists gets used. In this case the last one.

• $ntp::config::data::ntpservers
• $ntp::data:ntpservers

This would be an abstract from a forge module, anyone who use it will be configured to use the ntp.org base NTP servers.

Module User

As a user I really want to use this NTP module from the Forge and not write my own. But what I also need is flexibility over what NTP servers I use. Generally that means forking the module and making local edits. Parametrized Classes are supposed to make this better but sadly the design decisions means you need an ENC or a flexible data store. The data store was missing thus far and I really would not recommend their use without it.

Given that the NTP module above is using Hiera as a user I now have various options to override its use. I configure Hiera to use the (default) YAML backend for data but to also load in the Puppet backend should the YAML one not provide an answer. I also configure it to allow me to create per-location data that gives me the flexibility I need to pick NTP servers I need.

:backends: - yaml
           - puppet
:hierarchy: - %{location}
            - common

I now need to decide how best to override the data from the NTP module:

I want:

• Per datacenter values when needed. The NOC at the data center can change this data without change control.
• Company wide policy the should apply over the module defaults. This is company policy and should be subject to change control like my manifests.

Given these constraints I think the per-datacenter policy can go into data files that is controlled outside of my VCS like with a web application or simple editor. The common data that should apply company wide need to be controlled under my VCS and managed by the change control board.

Hiera makes this easy. By configuring it as above the new Puppet data search path – for a machine in dc1 – would be:

• $data::dc1::ntpservers – based on the Hiera configuration, user specific
• $data::common::ntpservers – based on the Hiera configuration, user specific
• $ntp::config::data::ntpservers – users do not touch this, it’s for module authors
• $ntp::data:ntpservers – users do not touch this, it’s for module authors

You can see this extends the list seen above, the module author data space remain in use but we now have a layer on top we can use.

First we create the company wide policy file in Puppet manifests:

modules/data/manifests/common.pp

class data::common {
   $ntpservers = ["ntp1.example.com", "ntp2.example.com"]
}

As Hiera will query this prior to querying any in-module data this will effectively prevent any downloaded module from supplying NTP servers other than ours. This is a company wide policy that applies to all machines unless specifically configured otherwise. This lives with your code in your SCM.

Next we create the data file for machines with fact $location=dc1. Note this data is created in a YAML file outside of the manifests. You can use JSON or any other Hiera backend so if you had this data in a CMDB in MySQL you could easily query the data from there:

hieradb/dc1.yaml

---
ntpservers: - ntp1.dc1.example.com
            - ntp2.dc1.example.com

And this is the really huge win. You can create Hiera plugins to get this data from anywhere you like that magically override your in-manifest data.

Finally here are a few Puppet node blocks:

node "web1.prod.example.com" {
   $location = "dc1"
   include ntp::config
}
 
node "web1.dev.example.com" {
   $location = "office"
   include ntp::config
}
 
node "oneoff.example.com" {
   class{"ntp::config": 
      ntpservers => ["ntp1.isp.net"]
   }
}

These 3 nodes will have different NTP configurations based on their location – you should really make $location a fact:

• web1.prod will use ntp1.dc1.example.com and ntp2.dc1.example.com
• web1.dev will use the data from class data::common
• oneoff.example.com is a complete weird case and you can still use the normal parametrized syntax – in this case Hiera wont be involved at all.

And so we have a very easy to use a natural blend between using param classes from an ENC for full programmatic control without sacrificing the usability for beginner users who can not or do not want to invest the time to create an ENC.

The plugin is available on GitHub as hiera-puppet and you can just install the hiera-puppet gem. You will still need to install the Parser Function into your master until the work I did to make Puppet extendable using RubyGems is merged.

The example above is on GitHub and you can just use that to test the concept and see how it works without the need to plug into your Puppet infrastructure first. See the README.

The Gem includes extlookup2hiera that can convert extlookup CSV files into JSON or YAML.

Hiera: a pluggable hierarchical data store

by R.I. Pienaar | Jun 5, 2011 | Uncategorized

Note: This project is now being managed by Puppetlabs, its new home is http://projects.puppetlabs.com/projects/hiera

In my previous post I presented a new version of extlookup that is pluggable. This is fine but it’s kind of tightly integrated with Puppet and hastily coded. That code works – and people are using it – but I wanted a more mature and properly standalone model.

So I wrote a new standalone non-puppet related data store that takes the main ideas of using Hierarchical data present in extlookup and made it generally available.

I think the best model for representing key data items about infrastructure is using a Hierarchical structure.

The image above shows the data model visually, in this case we need to know the Systems Administrator contact as well as the NTP servers for all machines.

If we had production machines in dc1, dc2 and our dev/testing in our office this model will give the Production machines specific NTP servers while the rest would use the public NTP infrastructure. DC1 would additional have a specific Systems Admin contact, perhaps it’s outsourced to your DR provider.

This is the model that extlookup exposed to Puppet and that a lot of people are using extensively.

Hiera extracts this into a standalone project and ships with a YAML backend by default, there are also JSON and Puppet ones available.

It extends the old extlookup model in a few key ways. It has configuration files of it’s own rather than rely on Puppet. You can chain multiple data sources together and the data directories are now subject to scope variable substitution.

The chaining of data sources is a fantastic ability that I will detail in a follow up blog post showing how you would use this to create reusable modules and make Puppet parametrized classes usable – even without an ENC.

It’s available as a gem using the simple gem install hiera and the code is on GitHub where there is an extensive README. There is also a companion project that let you use JSON as data store – gem install hiera-json. These are the first Gems I have made in years so no doubt they need some love, feedback appreciated in GitHub issues.

Given the diagram above and data setup to match you can query this data from the CLI, examples of the data is @ GitHub:

$ hiera ntpserver location=dc1
ntp1.dc1.example.com

If you were on your Puppet Master or had your node Fact YAML files handy you can use those to provide the scope, here the yaml file has a location=dc2 fact:

$ hiera ntpserver --yaml /var/lib/puppet/yaml/facts/example.com
ntp1.dc2.example.com

I have a number of future plans for this:

• Currently you can only do priority based searches. It will also support merge searches where each tier will contribute to the answer. The answer will be a merged hash
• A Puppet ENC should be written based on this data. This will require the merge searches mentioned above.
• More backends
• A webservice that can expose the data to your infrastructure
• Tools to create the data – this should really be Foreman and/or Puppet Dashboard like tools but certainly CLI ones should exist too.

I have written a new Puppet backend and Puppet function that can query this data. This has turned out really great and I will make a blog post dedicated to that later, for now you can see the README for that project for details. This backend lets you override in-module data supplied inside your manifests using external data of your choice. Definitely check it out.