Friday, July 29, 2016

Chef cookbook Promotion Across Environments

Creating Chef Environments to match your Infrastructure Environments allows
you to speak the same language with other groups within your organization and
lessens the overhead of managing constraints by keeping the number of
Chef Environments to a bare minimum.

A typical organization might have Infrastructure Environments like:

  • Dev
  • Staging
  • Prod

A Chef Environment can be represented via json like this:

    "name": "Staging",
    "description": "This is the Staging pre-production environment.",
    "cookbook_versions": {
        "acme-finance-web": "= 1.0.2",
        "acme-marketing-web": "= 0.1.2",
        "acme-sales-pos": "= 0.1.1"
    "json_class": "Chef::Environment",
    "chef_type": "environment",
    "default_attributes": {
    "override_attributes": {

As you can see above Chef allows you to constrain cookbook versions in an
Environment. Any nodes in that Environment will only be able to run these
cookbook versions.

For more on Versioning see:

Opt-in Will Scale Easier

It may be tempting to use the Chef Environment to constrain every cookbook
that exists. However, this will lead to version dependency nightmares.

For example, if Team A has Cookbook “team-a”, and Team B has Cookbook “team-b”
and both of them depend upon a shared Library Cookbook: “library-z” which happens
to be constrained in an Environment with an equality constraint; any new Release
of “library-z” will need to be coordinated across both Team A and Team B. This
strategy will simply not scale.

A much easier approach to managing this is to allow for a “Opt-in” style where
Team A and Team B are able to move to newer versions of “library-z” only when
they are ready and after thorough testing. The latest version of “library-z”
cookbook exists in all environments because there are no Chef Environment version
constraints for it. The “library-z” version constraints are encapsulated in consuming
cookbook’s metadata.rb.

This requires an approach of only managing version constraints for individual
Team cookbooks in Chef Environments by carving out a uniquely named “Role” for
them, version Pinning that “Role Cookbook” in the Environment and allowing the
teams to manage all dependencies within their Role Cookbook. This allows Teams
more autonomy to move at varying speeds promoting artifacts across Chef
Environments without fear of a suprise unresolvable dependency constraint
imposed by another Team or Group when they promote into a new Environment.
It significantly lessons the burdens on the Chef Administrators / DevOps
Champions who then do not have to “police” cookbook promotion, allowing the whole
Organization to begin autonomously developing with Chef.

An organization’s pipeline should be able to “promote” these Role Cookbooks from
one environment to the next after each dependent component passes comprehensive
automated unit, integration, regression, smoke and acceptance/functional tests
of the whole stack. Promotion entails the Pipeline adding or updating the Chef
Environment constraint for the Role Cookbooks in an automated fashion. Perhaps
using a tool like Knife Spork:

Individual users should not have the permissions to manipulate Chef Environments
directly. Rather, changes should be initiated through version control and pushed
to the Chef Server via automation in the Pipeline.

To implement this, it is important to understand the Cookbook types.

Cookbook Archetypes

Let’s look at the Cookbook types that fundamentally enable this to happen safely
and with ease.

The Library Cookbook

Most basic build block. These are meant to be reusable and depended upon by other
Cookbooks to provide functionality.

Library Cookbooks provide things like:
- Adding LWRPs that abstract common functionality
- Including Libraries that add Ruby modules/classes for any depending cookbooks

Generally these do not have attributes since there is nothing to configure
and often do not include any recipes.

Library cookbooks may depend on other library cookbooks or application cookbooks.
They never depend on a Role Cookbook and they never depend on a Wrapper cookbook.

An example:

The Application Cookbook

These cookbooks are a level above Library cookbooks. They always contain at least
one recipe (the default recipe) to install a particular piece of software sharing
the same name as the cookbook. If the application the cookbook manages contains
multiple components then each one is broken up into it’s own recipe and the recipe
is named after the component it will install. Things are broken up in this way so
you could install various components spread across a number of nodes within an

These cookbooks almost always contain a set of attributes which act as the runtime
configuration for the cookbook. These attributes can do something like setting a
port number or even describing the desired state of a service.

These cookbooks are always named after the application they manage.

Application cookbooks may depend on Library Cookbooks and other Application
Cookbooks. They never depend on Role Cookbooks. They never depend on a Wrapper
or Base Cookbook unless they are intended to be internal to your organization
and will never be distributed to the Chef Community Site.

Every Application cookbook should live in it’s own version control repository.

The Wrapper Cookbook

This is the lightest Cookbook out of all the known Cookbook patterns. It does a
very simple job of depending on an Application Cookbook and then exposing a
recipe for each recipe found in the Application Cookbook that it is wrapping. In
these recipes a single call to include_recipe “{wrapped-cookbook}::{wrapped-recipe}”
will be found along with a number of node.set[] functions which override the
default values of the wrapped Cookbook.

Wrapper cookbooks depend on Application Cookbooks only. They do not depend on
other Wrapper Cookbooks, Library Cookbooks, or Role Cookbooks.

These cookbooks follow the naming convention {organization}-{wrapped_cookbook}
or even sometimes {application}-{wrapped_cookbook}.

Every Wrapper cookbook should live in it’s own version control repository.

The Base Cookbook

Each organization should have one of these. This cookbook does the job of setting
the MOTD on your machines or creating users and setting them up with zsh instead
of bash. This cookbook can become a sort of “junk drawer” so you should be
careful when adding to it. Things should only be added here when it doesn’t make
sense to place them in another spot.

This is another cookbook pattern that you don’t see out in the wild because it’s
specific to your organization and shouldn’t be shared with anyone else.

Base Cookbooks may depend on Library Cookbooks, Application Cookbooks, or Wrapper
Cookbooks. They never depend on an Role Cookbook.

These cookbooks follow the naming convention {organization}-base.
Every Base cookbook should live in it’s own version control repository.

Example metatdata.rb of a Base Cookbook:

name             'acme-base-linux'
maintainer       'Acme Co., Inc'
maintainer_email ''
license          '# Copyright (c) 2016 Acme Co., Inc, All Rights Reserved.'
description      'Installs/Configures acme-base-linux'
long_description, ''))

version          '2.0.4'

source_url ''
issues_url ''

depends 'chef-client', '= 4.3.3'
depends 'chef-handler-profiler', '= 1.0.1'
depends 'chef-splunk', '= 1.8.0'
depends 'datadog', '= 2.1.0'
depends 'datadog_support', '= 2.6.0'
depends 'firewall', '= 2.3.0'
depends 'ntp', '= 1.7.0'
depends 'openssh', '= 1.5.2'
depends 'opsmatic', '= 0.1.26'
depends 'opsmatic_support', '= 0.1.7'
depends 'selinux', '= 0.8.1'
depends 'splunk_support', '= 0.6.0'
depends 'sysctl', '= 0.6.4'
depends 'sysctl_support', '= 1.2.0'
depends 'tmpwatch', '= 2.0.0'
depends 'yum-gd', '= 0.7.1'
depends 'os-hardening', '= 1.2.1'

Example Default Recipe of a Base Cookbook:

raise if node['platform'] == 'windows'

include_recipe 'yum-gd' if node['platform_family'] == 'rhel'
include_recipe 'chef-handler-profiler'
include_recipe 'chef-client::config'
include_recipe 'chef-client::service' if node['chef_client']['service']['enabled'] == true
include_recipe 'chef-client::delete_validation'

if node['acme-selinux']['enabled'] == false
  include_recipe 'selinux::permissive'
  include_recipe 'selinux::enforcing'

The Role Cookbook

This is the piece that ties the release process of your development cycle
together and allows you to release software that is easy to install and to configure.

A Role Cookbook should not contain any logic of it’s own - it should simply
include_recipe on the Application Cookbooks necessary. One exception is that it
can include logic about which nodes should get which services, and any
orchestration that has to happen. The version of the Role Cookbook will be
pinned in the Chef Environment.

Again, the Role Cookbook has metadata.rb depends entries (with version pins) for
each cookbook that you will want to converge on the node and will have
include_recipes entries in the default recipe for each dependent recipe component

Be sure to include the “base” cookbook if you have one.

Example metatdata.rb of a Role Cookbook:

name             'acme-finance-web'
maintainer       'Acme Finance'
maintainer_email ''
license          'All rights reserved'
description      'Role cookbook for finance-web application'
long_description, ''))
version          '1.0.2'

depends 'acme-base-linux', '~> 1.1.0'
depends 'finance-cookbook', '= 2.3.1'

Example Default Recipe of the Role Cookbook:

include_recipe 'acme-base-linux'
include_recipe 'finance-application::keys'
include_recipe 'finance-application::web'

Sometimes a Chef Role may be created to house the Role Cookbook. However, it
should NOT contain any attribute overrides and anything other than one
run_list entry: the Role Cookbook. This is important because Chef Roles are not
versioned. Changing a Chef Role immediately affects ALL nodes across all
environments that have the Role applied. Instead, allow the Role Cookbook to version
the run_list and dependency pins and perhaps override attributes.

Example Chef Role:

  "name": "acme-finance-web",
  "description": "Installs and configures all components for a Finance Web server.",
  "json_class": "Chef::Role",
  "default_attributes": {
  "override_attributes": {
  "chef_type": "role",
  "run_list": [
  "env_run_lists": {

Finally, the Chef Environment Files with Role Cookbook version pins:

This Development Environment has no constraints - latest versions get used.

    "name": "Dev",
    "description": "This is the Dev.  The latest and greatest versions.",
    "cookbook_versions": {
    "json_class": "Chef::Environment",
    "chef_type": "environment",
    "default_attributes": {
    "override_attributes": {

Staging Environment with equality version contraints.

    "name": "Staging",
    "description": "This is the Staging pre-production environment.",
    "cookbook_versions": {
        "acme-finance-web": "= 1.4.1",
        "acme-marketing-web": "= 1.1.8",
        "acme-sales-pos": "= 0.10.9"
    "json_class": "Chef::Environment",
    "chef_type": "environment",
    "default_attributes": {
    "override_attributes": {

Production Environment with equality version contraints.

    "name": "Prod",
    "description": "This is the Production environment.",
    "cookbook_versions": {
        "acme-finance-web": "= 1.0.1",
        "acme-marketing-web": "= 0.1.0",
        "acme-sales-pos": "= 0.1.0"
    "json_class": "Chef::Environment",
    "chef_type": "environment",
    "default_attributes": {
    "override_attributes": {

Notes: In other articles and discussion circles you may sometimes hear the words “Role Cookbooks”
and “Environment Cookbooks” used interchangeably.

Much of this was cribbed from:

Chef Ldap users

How to handle “can’t find user for xxx” error after trying to reference newly added ldap user from earlier in converge


Have you ever run into this issue:
Scenario: node is bootstrapped and during the 1st converge gets added to AD or some ldap OU
Later in the recipe a resource (such as file) is defined wth an owner attribute using an ldap user.
can't find user for xxx
This is because when chef-client began executing it didn’t know about the new ldap users. An Ohai reload doesn’t solve the problem either.
The following cookbook suggests one way to work around this problem and still be able to use the new ldap users
during the first converge.

Chef Intranet Scaffolding

What kind of scaffolding does it take to run Chef on a network with no Internet access?

Busser / Serverspec

Historically, the trickiest part of operating Chef in an Air Gapped environment has been local development when using the test-kitchen verifier.
The default for a long time has been busser / serverspec (this changed to inspec in version 0.16.28 of chef-dk)

The trouble is that Busser is not Proxy friendly and does not allow use of an alternate GemSource to

The recommended approach is simply migrating over to inspec rather then try to tackle setting proxy environment variables, creating custom box images with the gems pre-installed, etc.

inspec does not reach out to the Internet for any additional gems to install. Additionally, migrating from serverspec is pretty straight forward. Keep in mind that inspec currently does not allow for nested describe blocks, other than that, it’s resources are mostly the same with extra functionality and flexibility of custom extensions.

# .kitchen.yml
   name: inspec
   format: doc

The developers did an awesome job with the help sub-command to inspec - so there’s no reason not to migrate!

Check it out. I use it all the time myself :) With kitchen-inspec (included in ChefDK) you can easily run your tests against a kitchen instance.

~/Devel/ChefProject/tmp$ inspec help
  inspec archive PATH                # archive a profile to tar.gz (default) or zip
  inspec check PATH                  # verify all tests at the specified PATH
  inspec compliance SUBCOMMAND ...   # Chef Compliance commands
  inspec detect                      # detect the target OS
  inspec exec PATHS                  # run all test files at the specified PATH.
  inspec help [COMMAND]              # Describe available commands or one specific command
  inspec init TEMPLATE ...           # Scaffolds a new project
  inspec json PATH                   # read all tests in PATH and generate a JSON summary
  inspec shell                       # open an interactive debugging shell
  inspec supermarket SUBCOMMAND ...  # Supermarket commands
  inspec version                     # prints the version of this tool

  [--diagnose], [--no-diagnose]  # Show diagnostics (versions, configurations)

~/Devel/ChefProject/tmp$ inspec exec test/integration/default --format json
{"(generated from test.rb:1 d5357d303b3769b9e8e88dc69924750c)":{"title":null,"desc":null,"impact":0.5,"refs":[],"tags":{},"code":"          
rule =, profile_id, {}) do\n            res = describe(*args, &block)\n          end\n","source_location":
["/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/inspec-0.26.0/lib/inspec/profile_context.rb",181],"results":[{"status":"failed","code_desc":"File /etc/
postfix should be file","run_time":0.014447,"start_time":"2016-07-21 11:29:01 -0400","message":"expected `File /etc/postfix.file?` to return true, got false"},{"status":"failed","code_desc":"File /etc/postfix should be mode 644","run_time":0.021464,"start_time":"2016-07-21 11:29:01 -0400","message":"expected `File /etc/postfix.mode?(644)` to return true, got false"},{"status":"passed","code_desc":"File /etc/postfix should be owned by \"root\"","run_time":0.000424,"start_time":"2016-07-21 11:29:01 -0400"}]}},"groups":{"test.rb":{"title":null,"controls":["(generated from test.rb:1 d5357d303b3769b9e8e88dc69924750c)"]}},"attributes":[]}},"other_checks":[]}~/Devel/ChefProject/tmp$

~/Devel/ChefProject/tmp (master *%)$ kitchen verify
-----> Starting Kitchen (v1.10.2)
-----> Setting up <default-ubuntu-1404>...
       Finished setting up <default-ubuntu-1404> (0m0.00s).
-----> Verifying <default-ubuntu-1404>...
       Use `/Users/jmiller/Devel/ChefProject/test123/test/integration/default` for testing

Finished in 0.00027 seconds (files took 0.67817 seconds to load)

File /tmp/file
  should exist
  should be file
  should not be directory
  should not be block device
  should not be character device
  should not be pipe
  should not be socket
  should not be symlink
  should not be mounted

Finished in 0.02156 seconds (files took 0.7893 seconds to load)
9 examples, 0 failures

       Finished verifying <default-ubuntu-1404> (0m0.47s).
-----> Kitchen is finished. (0m1.88s)
~/Devel/ChefProject/test123 (master *%)$

Ruby Gems, RPMs, DEBs, etc.

If you’re going to operate Chef without Internet access, you will need to utilize an artifact server to host packages, binaries, gems, rpms, etc.

One very good option for this is Artifactory Pro. It supports almost any artifact / repo known to man, and it allows mirroring or acting like a caching proxy to it.


Host the chefdk package internally so that users can easily install it.

Chef Client

Host the chef-client package internally; to be used for bootstrapping nodes and in local development with test-kitchen

You can reference this location in your .kitchen.yml file:

  name: chef_zero
    chef_omnibus_url: http://my.web.server/chef-pkgs/

The does not require much, it can be as simple as:


cd /tmp/
wget http://my.web.server/chef-pkgs/chef-12.8.1-1.el7.x86_64.rpm
sudo rpm -Uvh /tmp/chef-12.8.1-1.el7.x86_64.rpm

Bootstrap considerations

Use --bootstrap-install-sh http://my.web.server/chef-pkgs/ knife option to point to the location of the installer script.

Installing additional gems via chef gem_package resource

gem_package 'train' do
  options('--no-rdoc --no-ri --no-user-install --source https://your.gem.server')

Another approach is to simply update what amounts to root’s .gemrc
adding your repo and removing

gembin = Chef::Util::PathHelper.join(Chef::Config.embedded_dir,'bin','gem')
execute 'set internal gem repo' do
 command "#{gembin} source -r -a https://your.gem.server"
 action :run

Alternatively, you can bundle up the gems you need into a tarball, host that as an artifact internally, then download and extract:

$ gem install --no-rdoc --no-ri --install-dir tmp/ --no-user-install mixlib-install
Fetching: artifactory-2.3.3.gem (100%)
Successfully installed artifactory-2.3.3
Fetching: mixlib-versioning-1.1.0.gem (100%)
Successfully installed mixlib-versioning-1.1.0
Fetching: mixlib-shellout-2.2.6.gem (100%)
Successfully installed mixlib-shellout-2.2.6
Fetching: mixlib-install-1.1.0.gem (100%)
Successfully installed mixlib-install-1.1.0
4 gems installed
$ ls -l tmp/
drwxr-xr-x  2 jmiller  staff   68 Jul 19 19:01 build_info
drwxr-xr-x  6 jmiller  staff  204 Jul 19 19:01 cache
drwxr-xr-x  2 jmiller  staff   68 Jul 19 19:01 doc
drwxr-xr-x  2 jmiller  staff   68 Jul 19 19:01 extensions
drwxr-xr-x  6 jmiller  staff  204 Jul 19 19:02 gems
drwxr-xr-x  6 jmiller  staff  204 Jul 19 19:02 specifications
$ cd tmp
$ tar cvf gems.tar *

Use it later in a recipe:

download_location = ::File.join(Chef::Config[:file_cache_path], 'gems.tar')

remote_file download_location do
  source 'https://your.artifact.server/gems.tar'
  action :create

execute 'extract gems' do
  command "tar -xvf #{download_location} -C #{node[:languages][:ruby][:gems_dir]}"
  not_if do

Other Chef Infrastructure

One of the best methods of installing Chef components is with the chef-ingredient cookbook.

Alas, it requires installation of mixlib-install ruby gem and all of its dependencies. Refer to the gem installation methods above to accomplish this in an Air Gapped environment.

Cookbook Repositories

Cookbooks inevitably have dependencies; managing those in the development phase is accomplished via Berkshelf.

Chef Server

As of version 12.4.0, Chef Server has a Universe endpoint, which provides the same output as Supermarket or berkshelf-api universe endpoints.

# Berksfile
source :chef_server


If a source is configured with the location :chef_server, then Berkshelf will use the configured Chef Server as an API source. This requires Chef Server 12.4.0 or newer, or Hosted Chef.

Internal Supermarket

A private supermarket provides an easily searchable cookbook repository (with friendly GUI and command line interfaces) that can run on a company’s internal network.

You can publish to a Supermarket in several ways, some notable methods:
* (feature has been moved into core Chef in versions greater than 12.11.18)

# Berksfile
source ''



MiniMart allows you to specify a list of cookbooks to mirror in YAML. MiniMart will then download any of the listed cookbooks and their dependencies to your machine. Once you are satisfied with the mirrored cookbooks, MiniMart can generate a Berkshelf compatible index of the available inventory.

# Berksfile
source ''


Tuesday, September 7, 2010

Sendmail failing quietly

Ran into this issue today. I noticed that we were not receiving some of our emails from our notification/monitoring server (a RHEL box). A quick check through maillog on the host didn't indicate any problems. However, I did observe that it seemed like mail hadn't been sent in a few days - which was not normal.

I tried a test with mailx which resulted in this:

[root@somehost]# mailx -s "testing123" me
testing 1 2 3...
fprintf.c:50: SM_REQUIRE((fp) != NULL && (fp)->sm_magic == (SmFileMagic)) failed


The issue turned out to be that /var had run out of available inodes due to an application writing thousands of tiny files. This was verified via dumpe2fs output on the Logical Volume. Once the inode issue was corrected mail began flowing normally again.

Monday, October 29, 2007

bash and tab completion of symlinks to directories

When using bash, ever wonder why you have to hit tab twice with completion when the object is a symlink to a directory?
For example:

bash-3.1$ mkdir -p a/b/c/d
bash-3.1$ ln -s a/b/c carrot
bash-3.1$ cd c<tab>arrot<tab>/

It's due to the GNU readline library. Readline has several variables that can be set to influence its behavior, including tab completion of symlinks to directory names. If you find double tapping tab annoying, the fix is trivial. Simply add these two lines to your inputrc:

set mark-directories On
set mark-symlinked-directories On

inputrc will be /etc/inputrc or your ~/.inputrc, either will work.