Skip to main content

Recursive deployment of puppet environments with r10k and librarian-puppet

By treating roles and profiles as puppet modules, we can use r10k and librarian-puppet to manage the deployment of our puppet code into our puppet environements.

I shall assume that puppet is configured to use to use directory environments and that the environment path is $confdir/environments (ie. the default location). I also assume that both r10k and librarian-puppet are installed and in the path.

You should also understand and embrace the role-profile-module pattern, first described by Craig Dunn and subsequently by Adrian Thebo and Gary Larizza. Quoting Gary:

  • Roles abstract profiles
  • Profiles abstract component modules
  • Hiera abstracts configuration data
  • Component modules abstract resources
  • Resources abstract the underlying OS implementation 

I find the following points useful to clarify the purpose of each of the layers in this model:

  • Roles, profiles, and component modules can all be implemented as puppet modules
  • Each node is assigned exactly one role (either in site.pp or, preferably, using some external node classifier)
  • Each role includes one or more profiles
  • Each profile loads configuration data and feeds it into the component modules – this is where your business logic should go
  • Each component module should be generic and contain no site-specific data. You should be able to publish all your component modules on PuppetForge without leaking any secrets.

We can further extend this model to include environments. An environment can be thought of as a group of roles and can also be implemented as a puppet module.

So, how do we set this up?

At the top-level, we put a Puppetfile in the puppet config dir containing a list of our environments. This will look something like this:

#!/usr/bin/env ruby#^syntax detectionforge 'https://forgeapi.puppetlabs.com'mod 'yo61-env_production',  :git => 'git@github.com:yo61/puppet-demo_env_production.git'mod 'yo61-env_staging',  :git => 'git@github.com:yo61/puppet-demo_env_staging.git'

Each environment is defined as a puppet module. Any valid Puppetfile syntax may be used to specifiy the module location, including alternate branches or specific version tags. 

Each of the environment modules should contain all the usual things you would put in a puppet environment, eg. a manifests/site.pp, etc. as well as a Puppetfile containing a list of all the roles to be deployed to this environment. The Puppetfile for a simple environment would look something like this:

#!/usr/bin/env ruby#^syntax detectionforge "https://forgeapi.puppetlabs.com"# list all the roles that are included in this environmentmod 'yo61-role_default',  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',  :path => 'modules/role_default'mod 'yo61-role_foo',  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',  :path => 'modules/role_foo'mod 'yo61-role_bar',  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',  :path => 'modules/role_bar'

Like the top-level Puppetfile used to defined environments, each role is defined as a puppet module.

Each of the role modules will contain a simple class that loads the profiles used by the role, and a Puppetfile containing a list of all profiles used by the role. The Puppetfile for a simple role would look something like this:

#!/usr/bin/env ruby#^syntax detectionforge "https://forgeapi.puppetlabs.com"# list all the profiles that are included in this rolemod 'yo61-profile_common',  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',  :path => 'modules/profile_common'

Each of the profile modules will contain all the puppet code required to define the business logic, load configuration data, etc. and a Puppetfile containing a list of all the component modules used by the profile. The Puppetfile for a simple profile would look something like this:

#!/usr/bin/env ruby#^syntax detectionforge "https://forgeapi.puppetlabs.com"# include all the modules used by this profilemod 'puppetlabs-stdlib',mod 'stahnma-epel'mod 'puppetlabs-ntp'

Again, any valid Puppetfile syntax may be used.

We've now defined all our environments, roles, and profiles and we're ready to deploy each environment.

First, we run r10k to deploy each of the environment modules into the environment dir (/etc/puppet/environments):

# switch to the location of the top-level Puppetfilecd /etc/puppetPUPPETFILE_DIR=/etc/puppet/environments r10k puppetfile install

This will create a directory in /etc/puppet/environments for each of the environments defined in the top-level Puppetfile.

Next, we change into each of the newly-created environment directories and run librarian-puppet to install all the roles required by that environment.

cd /etc/puppet/environments/productionLIBRARIAN_PUPPET_PATH=modules librarian-puppet install --no-use-v1-api --strip-dot-git

The best bit is that librarian-puppet supports recursive module dependency resolution so this one command installs not only the roles, but also all the profiles, and component modules required by each of the roles.

My next article will present a script that runs r10k and librarian-puppet as described in this article, and also updates puppet environments atomically

 
Labels:

Comments

Post a Comment

Popular posts from this blog

Python logging with rich - writing to stderr - plain output when writing to file

Rich is a Python library for writing rich text (with color and style) to the terminal, and for displaying advanced content such as tables, markdown, and syntax highlighted code. Rich provides RichHandler , a logging handler for python's logging module which will format and colorize text written by the module. However, RichHandler writes to stdout by default. More specifically, it writes to a rich Console object which, by default, writes to stdout. To make RichHandler write to stderr by default, you must pass in a Console object which has been configured to write to stderr: import logging from rich.console import Console from rich.logging import RichHandler DATEFMT = "%Y-%m- %d T%H:%M:%SZ" FORMAT = " %(message)s " logging . basicConfig( level = "NOTSET" , format = FORMAT, datefmt = DATEFMT, handlers = [RichHandler(console = Console(stderr = True ))], ) logger = logging . getLogger(__name__) logger . i...
Post a Comment
Read more

Fix python import order on save in vim with ruff and ale

My IDE of choice is vim. I use various tools to perform linting and code formatting, and configure them all with ALE  (the Asynchronous Lint Engine). After using several discrete tools ( black , isort , flake8 , etc) I have settled on using Ruff to do my python code formatting and linting. Here's the relevant fragment of my ALE config in my .vimrc: " ALE config let g :ale_fixers = { \ 'python' : [ 'ruff' , 'ruff_format' ], \} let g :ale_linters = { \ 'python' : [ 'ruff' ], \} let g :ale_python_ruff_use_global = 1 One of the last remaining wrinkles I had was getting Ruff to automatically sort import statements. Sorting imports is performed by the Ruff linter, not the formatter, which is documented here . The fix on the command line is to add an option, like this: ruff check --select I --fix The difficulty I had was getting this to happen in the editor when the file was saved. It turns out, all I needed to do was ...
Post a Comment
Read more

Escaping special characters in wget username or password

I recently offered to help out with the hosting of a WordPress  site. It’s currently hosted somewhere with no shell access – just ftp – and there are a lot of images to transfer. I quickly figured out I could use wget to mirror the site, using something like: wget -m ftp://username:password@example.com However, this broke in this case because the username for the site contained an @ character (the username was user@example.com ). Turns out the solution was to encode the special chars using HTML notation. This is the command that did the trick: wget -m ftp://user%40example.com:password@example.com
Post a Comment
Read more