Digg

Allow me to introduce myself. I'm Paul, Sr. Systems Engineer and resident Puppetmaster for Digg. "Puppetmaster?" I hear you say, "What do puppets have to do with Digg?" Well, that's what I thought I'd write about today.

To answer the question, it isn't "puppets", it's Puppet, the open-source configuration management tool, and a relatively new addition to the Operations Secret Sauce here at Digg. Dr. Timeless and Joe have already given you a good overview of the complex architecture that makes Digg possible; in my posts I'm going to go into detail on how we manage some of the components of that architecture using Puppet (and touching on other tools along the way).

First off, let me answer a couple of questions that I've had to answer in the past. Why configuration management? Why Puppet, specifically? Configuration management buys you a number of benefits, and other people have written about those benefits more extensively, and more eloquently, than I can. Suffice it to say the Ops team at Digg believes configuration management is an incredibly powerful tool for managing complexity in a large-scale architecture (which, by complete coincidence, is exactly what we have!)

We chose Puppet after evaluating a number of other options (cfengine, and bcfg2 among others). Like many people, we thought that Puppet, cfengine, and bcfg2 were the top contenders in this space. Cfengine is, in a way, the grand-daddy of open-source configuration management tools. Unfortunately, certain design decisions and philosophical stances it assumes leave it somewhat behind the curve. Bcfg2 showed a lot of promise, but in the end it lost out due to a few key factors. First, there was a lack of "Bundles" supporting the things we needed to manage; Puppet had "types" which more closely matched our needs. Second, the philosophy of Bcfg2 is a "total management" philosophy; you can't deploy a Bcfg2 configuration that only manages a small portion of a machine's configuration. For our needs, the ability to implement configuration management incrementally was very important, and Puppet gave us that flexibility. Third, the overall design philosophy of Puppet makes a lot of sense to us. Last, and least importantly, I had more experience with Puppet that I could draw on as we rolled it out.

For those of you that are unfamiliar, Puppet is a multi-tentacled beast project with several components: a declarative language for describing system configuration, a standalone parser for that language, a resource abstraction layer for providing platform-agnostic manipulation of the resources described by the language, and a client-server daemon for distributing and applying configurations described by the language. (Phew, what a mouthful!) The foundation of Puppet's power is the resource abstraction layer, which I will talk about below. The rest of the Puppet components (and how we use them) will have to wait for another post.

Puppet's resource abstraction layer (the RAL, to save me some typing) allows us to think of various aspects of system configuration as "resources". Puppet comes with a tool called ralsh which allows you to interact directly with the underlying RAL; you can use it to list resources or manipulate them in the same way Puppet does. "But what exactly is a resource!?" I hear you cry. A resource is a discrete component of system configuration. Some examples are: cron jobs, users, host file entries, mount points... Puppet's RAL knows how to manage a wide variety of resources natively; it is also fairly easy to extend it to manage resources it doesn't already understand.

ralsh is an incredibly powerful tool, and decidedly underused in the Puppet community. Let’s pretend you’ve inherited a decidedly non-homogeneous architecture, filled with various flavors of Linux and *BSD. Want to add a user? Just remember that on linux you can use adduser, or useradd, but on FreeBSD you probably want pw, and who knows how many different command-line switches you need to remember, or maybe just consult the man pages every time… Or, use ralsh. It provides an interface that works on every platform Puppet works on. The interface is the same on every platform. Want to know what users are defined? ralsh user lists them. Just want to know about this guy named baduser? ralsh user baduser will give you the dirt. Want to add a user? Watch this:

~$ ralsh user newuser uid=9999 gid=9999 home=/home/newuser shell=/bin/bash ensure=present
notice: /User[newuser]/ensure: created
user { ‘newuser’:
uid => ‘9999′,
gid => ‘9999′,
home => ‘/home/newuser’,
shell => ‘/bin/bash’,
password => ‘!’,
ensure => ‘present’
}


This doesn't get really cool until you realize you can use the exact same tool and syntax to manage any type of resource that Puppet understands. Check it out:

~$ ralsh package xmlstarlet ensure=present
package { 'xmlstarlet':
ensure => '1.0.1-2'
}

~$ ralsh cron check_my_mail command="/usr/bin/fetchmail mail.my.mail.server" user=plathrop hour='*' minute='*/30' ensure=present
notice: /Cron[check_my_mail]/ensure: created
cron { 'check_my_mail':
command => '/usr/bin/fetchmail mail.my.mail.server',
monthday => 'absent',
hour => ['*'],
environment => '',
target => 'plathrop',
special => '',
minute => ['*/30'],
user => 'plathrop',
ensure => 'present',
weekday => 'absent',
month => 'absent'
}

This is totally cool! Suddenly I don't have to care what platform I'm on, and I can think of these things in an abstract, encapsulated manner. I can choose platforms based off of what they are good at instead of requiring homogeneity in order to minimize the costs of management. Like OpenBSD's firewall, but you have a Debian network? By all means, throw an OpenBSD box in there; you can use the same commands to manage both. Want the performance of a FreeBSD network stack for a certain application? Go for it, you already have the tools you need to administer it.

Not only that, but if you are building a Puppet infrastructure, you can use ralsh to explore your existing configuration; the output you see above is valid Puppet code, and can be used in a Puppet configuration as-is!

Next time I'll talk more about the higher-level components of Puppet; the RAL, though awesome, is just the foundation. The Puppet language allows us to treat configurations as code; we can use the same techniques software engineers use to manage their codebases to manage our systems configurations. The Puppet client/server allows us to apply consistent configurations across a cluster of machines, and manage the entire life-cycle of each system, from deployment to retirement. Combined with a flexible and robust automated deployment system like Debian FAI, Puppet can help you drastically reduce the intervention required to bring a machine up from bare-metal to production; saving both time and money as well as giving you a chance to focus on more important issues.

See you next time.

--Paul