Updating Puppet modules for deferred functions

Puppet 6 brought the ability to defer functions to runtime on the agent, and now we've released improvements that make this easier to do. Read on to find out more and to make sure your modules are ready to be deferred.

What are Puppet functions again?

First, let's do a very brief recap. Puppet functions are bits of code that are executed during catalog compilation. They can do various things, such as including other classes in the catalog or marking resources in a given scope as no-op, but the kinds of functions we are interested in today are those that return a value. For example, fqdnrand() will deterministically generate a random number and shellquote() will transform its input and return a string safe to execute by shells like Bash.

The values returned by these functions are compiled directly into the catalog and become as static as the rest of the document. In other words, if you inspect a catalog generated for a specific node, you can see exactly what random number was used to specify the schedule for a cron job, or you can see exactly how a shell command was quoted or escaped before being invoked. And because they're immutable, that catalog will always result in the same desired state being enforced.

But in some circumstances, it would be more useful to defer that to the Puppet agent to execute as part of the catalog enforcement. For example, many infrastructures mandate that secrets be provided by something like a highly protected secrets server rather than funneling them through a configuration management server. In cases like this, we care less about the specific value being enforced than we do about what it represents. We don't need the Puppet server to actually know the database account password a node is configured with. We just need to instruct the agent to put the password, whatever it resolves to, in the appropriate configuration file.

In cases like this, you can instead compile a reference to the function itself into the catalog and instruct the agent to invoke it at runtime like so:

class { 'profile::myappstack':
  db_adapter  => 'postgresql',
  db_address  => 'pgsql.example.com',
  db_password => Deferred(
                   'vault_lookup::lookup',
                   ['appstack/dbpass', 'https://vault.example.com']
                 ),
}

Now, instead of the catalog containing the password and the Puppet server having access to the password, the agent itself will retrieve the password directly from Vault using its own registered credentials. This allows the infrastructure admins to have much more fine-grained control over the access to sensitive information and to quickly rotate or revoke them as needed.

Any function that returns a value can be deferred in this way, and in many cases it's just this easy. There are some challenges though, some which have been simplified by recent Puppet updates. We'll talk about those first.

Puppet language updates

In Puppet's first implementation, the catalog was effectively pre-processed to resolve deferred functions into values. This means that before the catalog was enforced, the agent would scan through it and invoke each deferred function. The value returned would be inserted into the catalog in place of the function. Then the catalog would be enforced as usual. The problem with this approach is that if the function depended on any tooling installed as part of the Puppet run, then it would fail on the first run because it was invoked prior to installation. If the function didn't gracefully handle missing tooling, it could even prevent the catalog from being enforced at all.

As of Puppet 7.17, functions can now be lazily evaluated with the new preprocess_deferred setting. This instructs the agent to resolve deferred functions during enforcement instead of before. In other words, if you use standard Puppet relationships to ensure that tooling is managed prior to classes or resources that use the deferred functions using that tooling then it will operate as expected and the function will execute properly.

Puppet 7.17 also improves the way typed class parameters are checked. The data type of a deferred function is Deferred, and older versions of Puppet would actually use that type when checking class signatures. For example, if the profile::myappstack class we referenced in the example above specified that the db_password parameter should be a String, then the example would have failed because the Deferred type would not match the expected String type.

As of Puppet 7.17, this is no longer a problem. Deferred functions are introspected and the return type they declare will be used for type matching. If the function doesn't explicitly declare a return type, Puppet will print a warning, but the compilation will succeed. No code changes are required to take advantage of this improvement, but if you're writing classes that might be used with older Puppet versions, you might consider using a variant datatype such as Variant[String, Deferred] for parameters that are expected to be deferred.

class profile::myappstack(
  String db_adapter,
  String db_address,
  Variant[String, Deferred] db_password,
) { ...

The third, and probably most challenging, concern is that depending on how authors write their modules, you may or may not be able to pass deferred functions in as parameters to many popular Forge modules. Let's look at some examples and learn how to anticipate them and future proof our own modules for deferred functions.

There are four major causes of incompatibility, and probably other variations that follow similar patterns. We'll start with the simplest and work towards the most complex.

Problem #1: Puppet language functions cannot be deferred

As of Puppet 4.2, many functions can be written directly in the Puppet language rather than in Ruby. Functions like this are often used to transform data, such as this example from docs that turns an ACL list into a resource hash to be used with the create_resources() function. Because these functions are not pluginsynced to the agent, they cannot be deferred. In general, this isn't much of a concern because operations like connecting to a Vault server cannot be done easily in the Puppet language anyways. But if you do have such a need, then this function needs to be rewritten in the Ruby language.

Problem #2: strings and resource titles cannot be deferred

A value that comes from a deferred function cannot be used in a resource title or interpolated into a string. For example, let's say that you added debugging code to the myappstack profile to see what password the Vault server was returning.

class profile::myappstack(
  String db_adapter,
  String db_address,
  String db_password,
) {
  notify { "Password: ${db_password}": }
  #...
}

Instead of the password you expect to see, it's the text form of the Deferred function!

$ puppet agent -t
…
Notice: Password: Deferred({'name' =>'vault_lookup::lookup', 'arguments' => ['appstack/dbpass', 'https://vault.example.com']})

And if you wrote it without string interpolation, like notify { $db_password: }, then it would fail compilation completely and give you a seemingly nonsensical error that might give seasoned C++ programmers template flashbacks.

Error: Evaluation Error: Illegal title type at index 0. Expected 
String, got Object[{name => 'Deferred', attributes => {'name' => 
Pattern[/\A[$]?[a-z][a-z0-9_]*(?:::[a-z][a-z0-9_]*)*\z/], 'argum
ents' => {type => Optional[Array], value => undef}}}] (file: /Us
ers/ben.ford/tmp/deferred.pp, line: 6, column: 12) on node arach
ne.local

The solution to this problem is that variables you expect to be deferred should not be used as resource titles or in interpolated strings. The notify in this example should be refactored like so:

notify { 'vault server debugging':
    # We cannot interpolate a string with a deferred value
    # because that interpolation happens during compilation.
    message => $db_password,
}

If you need to interpolate a deferred value into a string, you can do that by deferring the sprintf() function. For example, you could write that notify like so:

notify { 'vault server debugging':
    # Defer interpolation to runtime after the value is resolved.
    message => Deferred(
                 'sprintf',
                 ['Password: %s', $db_password]
               ),
}

Don't forget to remove this message once the Vault problem has been resolved so it doesn't leak your secrets!

Problem #3: function arguments can (usually) not be deferred

Closely related to the first problem, function arguments cannot be deferred, unless the function is designed for it. Functions are evaluated during compilation, so if you defer an argument they'll operate on a Ruby object instead of the resolved value. This is usually noticed when trying to render templated files. For example, if that myappstack profile managed a configuration file with a template, it would include the same text form of the deferred vault lookup function as above:

$ cat /etc/myappstack/db.conf
dbpassword = Deferred({'name' =>'vault_lookup::lookup', 'arguments' => ['appstack/dbpass', 'appstack/dbpass']})

In order to properly handle deferred functions, the function using them must also be deferred. For example, you could defer the rendering of the database configuration file using the new deferrable_epp() function that defers template rendering when needed. Using that function to generate templated files allows you to transparently handle deferred parameters. This function is available starting in puppetlabs-stdlib version 8.4.0. Note that it requires you to explicitly pass in the variables you'll be using in the template.

If you need to support earlier versions of stdlib, then you'll need to write the boilerplate logic yourself, which might look something like this:

$variables = { 'password' => $db_password }

if $db_password.is_a(Deferred) {
  $content = Deferred(
               'inline_epp',
               [find_template('profiles/myappstack.db.epp').file, $variables],
             )
}
else {
  $content = epp('profiles/myappstack.db.epp', $variables)
}

file { '/path/to/configfile':
    ensure  => file,
    content => $content,
}

Problem #4: deferred values cannot be used for logic

A value that's not known until runtime cannot be used to make conditional decisions, since all logic is resolved during compilation. An example of this is the puppetlabs-postgresql module, which handled provided password hashes differently based on the algorithm used to create them. If the user expects that password hash to be provided at runtime by a secret server, then it's not known at runtime and the compiler can't choose the appropriate codepath.

The resolution for this kind of problem is to refactor so these decisions don't need to be made during compilation, or so that different data is used to make decisions. In the case of our PostgreSQL module, we refactored that code so that the complete codepaths affected by that conditional were all deferred. The same code will run, but it will all be evaluated during runtime on the agent.

Depending on the type of decision to be made, you could also refactor into using facts, which are evaluated on the agent prior to catalog compilation and then making conditional decisions based on the resolved values of the facts.

Summary

I'm sure you see a common thread in each of these problem scenarios. Values calculated by deferred functions are simply not known at compile time. This means that nothing processed during compilation can use them. You cannot use a deferred value to interpolate into a string, or use it as a key for a selector, or make a logical decision with. You cannot render it directly into a templated file, instead you need to include the template source in the catalog and compile it at runtime.

Effectively, a deferred function is only useful when passing the value it generates directly to a parameter of a declared resource, and modules need to be written to take this into account. Module authors should anticipate that people might want to defer certain parameters, such as passwords or tokens or other secret values and handle those cases by refactoring any use of these values out of compile time and into runtime.

Ecosystem updates that simplify Deferred use cases:

• Set preprocess_deferred when your functions depend on tooling installed by the Puppet run.
• Deferred functions are interpolated so that their return types can be used to match data types required by class signatures.
• The new deferrable_epp() function will automatically defer epp template rendering when appropriate.

Good luck! We're always excited to see the cool things you build.

Ben is the Community and DevRel lead at Puppet.

Learn more

• Read the deferred function docs.
• See a presentation I gave at CfgMgmtCamp 2019