The Puppet 4 Lookup Function

Puppet 4 has a new lookup subsystem exposed to the user in a few places:

• The lookup() function
• Automatic parameter lookups
• Configuring the automatic parameter lookups via Data in Modules

I’ve not been able to figure out everything the docs have been trying to say about this function but it turns out they were copied from the deep_merge gem and it actually has better examples in some cases. So I thought a post exploring it and it’s various forms is in order

It’s pivotal to the use of data in Puppet so while you probably don’t need to fully grasp all of it’s intricacies as in this post a passing knowledge is valuable as is knowing how to find good help for it. I do think there’s some opportunity for improving the UX of this function though.

As usual the challenge when faced with all these options isn’t in how to use them all but in which options to use when that won’t result in a giant unmaintainable mess down the line. I think this function is definitely on the wrong side of the line in this regard. It’s massive and unwieldy in that it is exposing internals of Puppet in a 1:1 manner to the user.

So I would not recommend writing code that calls this function directly unless in extraordinary circumstances. With the Data in Modules and Automatic Parameter Lookup features you can achieve this, see the last section of the post for that.

First though you need to know the behaviours and terminology of the lookup() function in order to get to a point where you can use the other methods, so lets dive in.

Lookup Patterns

Basic usage

lookup("some::thing", String, "first", "default value")

Here we’re looking up the key some::thing and it has to be a String from the data store. It will do a first style lookup which is your basic traditional Hiera first-match-wins and there’s a default. Apparently there is no simple case lookup(“some::thing”, “default”) which seems like it would be the most common use. You can come kind of close though with (more on this below):

lookup({"name" => "some::thing", "default_value" => "default"})

Anyway, you’re not really going to be using the lookup function directly much so this is probably fine

The thing to note here are the lookup strategies, there are a few and you will always have to know them:

So this is your basic replacement for the old hiera(), hiera_hash() and hiera_array() and as you can see from the last 2 the merge strategy isn’t set globally like in old Hiera, this is a big improvement.

I will not go into a full exploration of what Tiers mean, the old Hiera docs are pretty good for that. Effectively a merge strategy describe what Hiera does when it finds interesting data in many different levels of data or in different data sources.

Complex Strategies for Setting Defaults

First I’ll look at the task of setting defaults. Hiera had quite basic features in this space which was enough to get going but lookup has some nice additions.

First the above lookup can also be written like this:

lookup({"name" => "some::thing", "value_type" => String, "default_value" => "default", "merge" => "first"})
lookup({"name" => "some::thing", "default_value" => "default"}) # though accepts any data type

So this is quite nice because now you can decide the order of arguments and which to include.

There’s a more powerful way to set defaults though:

function some_module::params() {
  $result = {
    "some_module::thing" => "default",
    "some_module::other_thing" => false
  }
}
 
lookup({"name" => "some_module::thing", "default_values_hash" => some_module::params()})

Which at first does not seem a huge improvement, but if you’re thinking about strategies to replace something like params.pp you could come up with some interesting patterns using this method. For example you can have a module function like here and an environment one (it supports environment level native functions) and combine them like environment_params() + some_module::params() to come up with layered sets of defaults, in effect this would be a micro hiera on it’s own programmed in pure Puppet DSL.

And finally you can use a lambda to set the default:

lookup("some::thing") |$key| { "Could not find a value for key '${key}', please configure it in your hiera data" }

Here we return a custom string instead that tells the user what is going on rather than blow up badly and we can of course include any helpful information like fact values and such to help them find the right place in your possibly complex data store.

Sticking to the Lambda I saw Henrik mention this on IRC yesterday:

$result = with(lookup("some::thing")) |$value| { if $value =~ Array { $value } else { [$value] } }

This does a lookup and ensures that the result is always an array, like the Ruby code Array(thing). These 2 Lambda approaches can’t really be done without calling lookup() specifically, so probably a bit niche.

I won’t go into all the details just now about Data in Modules and Merge Strategies but to see how these things tie together you should know you can set these option hashes via your data layer, see the linked to blog post for some details about this. The last section of this post shows a end to end working setup with Data in Modules and Merge Strategies in data.

Merge Strategies

We’ve seen the basic merge strategies above:

lookup("some::thing", String, "first", "default value")
lookup({"name" => "some::thing", "value_type" => String, "default_value" => "default", "merge" => "first"})

Here the strategy is first. But when the strategy is deep this can also be a hash with more merging options.

The most interesting for me is the knockout_prefix one. A common question when using Hiera for node classification is how to exclude a class from a certain node. This was kind of doable at least in Puppet 4 by using Arrays like:

include(hiera_array("classes", []) - hiera_array("exclude_classes", []))

Which will lookup classes and exclude_classes and subtract them from each other. This is a hack, lets look at a better option:

Given data like this:

# common.yaml
classification:
  classes:
    - sensu
    - sysadmin

# node1.example.net.yaml
classification:
  classes:
    - --sensu
    - nagios
    - webserver

What we’re trying to say is that the node1.example.net is not monitored by Sensu but by Nagios instead, the following lookup achieves this and includes the resulting classes:

$classification = lookup({"name" => "classifiation", 
        "merge" => {
          "strategy" => "deep", 
          "knockout_prefix" => "--",
          "sort_merge_arrays" => true
        }
})
 
$classification["classes"].include

Additionally I sorted the merged arrays. The tells it to remove data that matches the prefix. You can remove just some array member like here or entire keys from a resulting hash.

There’s another option where if some array member was a hash and you wanted to merge these hashes in the result sets you can set merge_hash_arrays. At that point you should probably rather rethink your data though tbh.

And the last one which I cannot figure out any use for and was quite baffled at is about turning Strings into Arrays. Henrik says they did not add this one for a reason other than it’s available on the deep_merge gem.

Lets change the data for our node to look like this:

# node1.example.net.yaml
classification:
  classes:
    - --sensu,nagios
    - webserver

While leaving the common data as is. If you set “unpack_arrays” => “,” in the merge options it will take every string found, split it by “,” which would turn this into a array of [“–sensu”, “nagios”] and then merge it up and then perform any knockouts so you get the same outcome ie. [“nagios”, “sysadmin”, “webserver”].

You should probably rethink your data instead if you find this useful 🙂 That said though this –sensu,nagios does look like a search and replace, so perhaps in the context of a classifier utility it’s not all bad.

CLI tool

To recreate the above lookup on the cli you’d do (though only once PUP-6050 is fixed):

% puppet lookup --hiera_config hiera.yaml --merge deep --knock-out-prefix "--" --unpack-arrays "," --sort-merge-arrays classification
---
classes:
- sysadmin
- nagios
- webserver

This is fine, but it’s a lot nicer than that. If you add the option –explain you get this:

Merge strategy deep
  Options: {
    "sort_merge_arrays" => true,
    "merge_hash_arrays" => false,
    "knockout_prefix" => "--",
    "unpack_arrays" => ","
  }
  Data Binding "hiera"
    Found key: "classification" value: {
      "classes" => [
        "sysadmin",
        "nagios",
        "webserver"
      ]
    }
  Data Provider "EnvironmentDataProvider"
    No such key: "classification"
  Merged result: {
    "classes" => [
      "sysadmin",
      "nagios",
      "webserver"
    ]
  }

A bit lacking in the case of old school hiera data since old Hiera does not emit the right kinds of detail for it to show where it gets your data from. It’s handy though since you can see the merge options hash and what data providers are queries. See below for the full potential.

Bringing it all together

When I started this fairly epic post I said I do not recommend people use lookup() directly, so lets take a look at pulling this all together.

I’ll make a simple classifier class like above in a module. Note the classes variable would above be done with the huge lookup() but not here. We do not want to use the lookup() function instead use Automatic Parameter Lookup:

class classifier($classes) {
  $classes.include
}

I’ll set it up for data in modules and add to it the lookup options:

# production/modules/classifier/data/common.yaml
lookup_options:
  classifier::classes:
    merge:
      strategy: deep
      knockout_prefix: "--"
      unpack_arrays: ","
      sort_merge_arrays: true

Note this is basically a lookup() call but attached to a specific key – classifier::classes. This way as we add more classification data we can have different strategies and such, doing it here means it works across all types of Hiera data old and new.

Now the data, I am using the environment data provider here – so no classic hiera at all:

First we configure our production environment to have it’s own instance of Hiera and it’s own hiera.yaml – take note, this is huge. Per environment hiera and hierarchies now works!

# production/environment.conf
environment_data_provider = hiera

# production/hiera.yaml
---
version: 4
datadir: "hieradata"
hierarchy:
  - name: "%{trusted.certname}"
    backend: "yaml"
  - name: "common"
    backend: "yaml"

Here’s our production environment data:

# production/hieradata/common.yaml
classifier::classes:
  - sensu
  - sysadmins

# production/hieradata/dev1.devco.net.yaml
classifier::classes:
  - nagios
  - --sensu
  - webserver

At this point it all works a charm, our node knocks out Sensu and brings in Nagios. This is a major wishlist item that old hiera_include() did not have!

Note this is just Array data that’s being knocked out and not Hash data here, while the deep strategy is supposed to work with Hashes only, so I am a bit surprised it works but I’ll take it as it makes this classifier better.

% puppet lookup --environmentpath environments classifier::classes
---
- sysadmins
- nagios
- webserver

And if we added –explain you can finally get the massive benefit of finally learning how Hiera finds your data:

% puppet lookup --environmentpath environments --explain classifier::classes
Merge strategy deep
  Options: {
    "knockout_prefix" => "--",
    "sort_merge_arrays" => true,
    "unpack_arrays" => ","
  }
  Data Binding "hiera"
    No such key: "classifier::classes"
  Data Provider "Hiera Data Provider, version 4"
    ConfigurationPath "/home/rip/temp/lookup/environments/production/hiera.yaml"
    Merge strategy deep
      Options: {
        "knockout_prefix" => "--",
        "sort_merge_arrays" => true,
        "unpack_arrays" => ","
      }
      Data Provider "%{trusted.certname}"
        Path "/home/rip/temp/lookup/environments/production/hieradata/dev1.devco.net.yaml"
          Original path: "%{trusted.certname}"
          Found key: "classifier::classes" value: [
            "nagios",
            "--sensu",
            "webserver"
          ]
      Data Provider "common"
        Path "/home/rip/temp/lookup/environments/production/hieradata/common.yaml"
          Original path: "common"
          Found key: "classifier::classes" value: [
            "sensu",
            "sysadmins"
          ]
      Merged result: [
        "sysadmins",
        "nagios",
        "webserver"
      ]
  Module "classifier" using Data Provider "Hiera Data Provider, version 4"
    ConfigurationPath "/home/rip/temp/lookup/environments/production/modules/classifier/hiera.yaml"
    Merge strategy deep
      Options: {
        "knockout_prefix" => "--",
        "sort_merge_arrays" => true,
        "unpack_arrays" => ","
      }
      Data Provider "%{trusted.certname}"
        Path "/home/rip/temp/lookup/environments/production/modules/classifier/data/dev1.devco.net.yaml"
          Original path: "%{trusted.certname}"
          Path not found
      Data Provider "common"
        Path "/home/rip/temp/lookup/environments/production/modules/classifier/data/common.yaml"
          Original path: "common"
          No such key: "classifier::classes"
  Merged result: [
    "sysadmins",
    "nagios",
    "webserver"
  ]

Every data file and every config file is shown and the full merge logic in all it’s glory is included. Huge win over previous hiera.

The result is a bit dense but if you follow along you can see it all works quite nicely and it’s super helpful for debugging cases where hiera just don’t work.

It’s a bit awkward – here I am doing it on the node the data is for, but for other nodes you would need their facts. As I understand it, it basically compiles the catalog and profiles the lookups during that process, so it needs facts as usual.

Conclusion

So that’s a rather epic exploration of the lookup() function which eventually ended us up with – do not use the lookup() function 🙂

You can see how this is a big step forward and in the end by using environment and module data – and no site data – I am not using old Hiera at all anymore as far as I know. This is purely the new lookup subsystem and it’s really powerful.

• Environments and Modules can have data and independent hierarchies
• The lookup subsystem is fully exposed in lookup() but the bulk of the features are accessible via lookup_options and so the Automatic Parameter Lookups
• It has a really good CLI command which once a few bugs are sorted can bring amazing visibility to where your data comes from and what data is assigned to a node. Even without those bugs fixed though if you use lookup_options as in the last option it’s totally usable today