Puppet is amazing. It changed my career (thanks to Luke , and before him Mark). However, I have some itches. I have attempted to write about these before, but haven’t felt like pushing the ‘publish’ button.

I’ve been running Puppet in a atypical way for some time now.

No Puppet Master
No distribution packaging
No commit until I know something works
Only test from the outside
The only thing I feel I need to

This week we’re going to begin our coverage of the Puppet Module Tool (PMT). The PMT is the command-line interface to the Puppet Forge—an online repository for re-usable Puppet modules. As of Puppet 2.7.14 and Puppet Enterprise 2.5 PMT is no longer a separate product; it’s now part of Puppet Core. What this means for you is that PMT is ready-to-go out of the box.

Talk given at the 6th PuppetCamp in Edinburgh.

Covers some of the useful information puppet stores and how to get at it.

Rump helps you run Puppet locally against a Git checkout.

Rump encourages a Puppet workflow where you quickly + iteratively develop your Puppet manifests on a single machine, then push your changes up to a repository to deploy to the rest of your infrastructure.

This workflow also complements a Capistrano or MCollective-style deployment, where you remotely instruct Rump to check out a copy of the latest manifests and run them.

Rump also has the ability to freeze Puppet in to the manifests repository, letting you quickly test different versions of Puppet without waiting for packages to appear, and reducing the dependencies on a system to run Puppet down to just Ruby and git.

Skewer is a tool that helps you run Puppet code on arbitrary machines. Skewer can:

Spawn a new virtual machine via a cloud system
Provision it with puppet code
Update the puppet code on a machine that you've provisioned already
Delete it

Skewer exists because sometimes, a Puppetmaster server isn't needed. If you run a small number of nodes, or simply want to bootstrap cloud nodes, you might like this tool.

In a recent blog post we discussed verifying and testing your Puppet code, which generated a large amount of interest. One of the readers asked why we’d bother with rspec-puppet if it simply verifies attributes of a resource. It’s true that, if you have a manifest that simply contained a collection of static resources, rspec-puppet doesn’t provide much additional value. It shines when you start developing define resources, parametrized classes, or any module that contains complex logic. In this blog post we’ll use a staging module to demonstrate Test Driven Development (TDD) with Puppet by writing rspec-puppet. For people who aren’t familiar with TDD, it refers to writing the rspec-puppet test cases before writing the Puppet manifests. This method allows you to be more confident in your code (by validating the unit test), to isolate problems (by testing code as it’s being developed), and to discover bugs earlier, when they are easier to troubleshoot and fix. In the diagram below, the complexity, time, and cost of testing increases as we move up the testing stack from unit (rspec), module acceptance (virtual machine), separate

I know we’ve been covering Puppet Labs modules for the last few weeks, but I’d like to specifically call out all the great community contributions we’ve had on this module. In fact, this module started out as a community members project, was forked by someone at Puppet Labs, modified and later enjoyed significant contributions from other community members.

We’re taking a brief break from your regularly scheduled module of the week goodness with a mini series to discuss some of the elements that go into writing great Puppet modules. I’ll try to sum it up in a few sentences: Your module should be immediately useful to me after I install it. Documentation should be complete and the code itself should adhere to the community style guide. If I need to modify the behavior of your module, you should provide me an interface to do it so I don’t have to modify your modules manifests. If I contribute features or bug fixes, I expect you to explain the process and collaborate with me on improving the module. I’ll be honest, writing a module isn’t always an easy task, and all the things discussed here will add time and complexity to that module. This multi-part series on writing great modules is all about giving you the knowledge and tools to make your module more reusable, immediately useful to others and easier for your team and the community at large to maintain. My theory is that time spent initially building a more robust module will return you ten-fold the value over the life of that module in your environment. With that in mind, I hope you find these concepts useful and return to the series next time as I dig deeper, leaving you to build your next great Puppet Forge module!

This week’s module helps implement and manage Jenkins CI with Puppet. We all need to test our code when we make changes. Saying so is easy, doing so is a lot more difficult. Jenkins makes testing easier. Every time you make a change, Jenkins will test everything still works. It’s easy to get started with Jenkins by using a Puppet module written by R. Tyler Croy. This module automatically installs the Jenkins software, configures the service, and provides an simple way to add functionality by automatically installing any of the 400+ Jenkins plugins. You should give this module a spin if you’re looking for an easy way to try out Jenkins or puppetize an existing Jenkins server. Install it, classify a node with the jenkins class in the Puppet Enterprise Console and Jenkins is running and ready to do your bidding.

Kanban and DevOps Roundtable (Video)
DAMON EDWARDS | WEDNESDAY, MARCH 28, 2012 AT 6:30PM
Ok so it's more of a semi-circle than a roundtable... I was at the first ever Kanban for DevOps class this past week in Sunnyvale, CA and after looking around the room I couldn't let these folks go without getting them on video:
- Luke Kanies (Puppet Labs)
- John Willis (Enstratus)
- Gene Kim (Author)
- Dominica DeGrandis (David J. Anderson & Associates)

Lucky for our readers, they didn't disappoint. We talk about why we think Kanban is an excellent tool for solving DevOps flow problems and our Kanban experiences thus far. 

Here is the video:

It has been quite a while since I’ve had anything sufficiently detailed or interesting to post on a technical theme so it was quite a welcome divergence from the usual sprint work recently when we started having some Puppet performance issues. First up, like most people working at big corporations I can’t give you any figures that might give away numbers of servers or anything else sensitive but obviously we run enough machines to cause performance issues on a reasonably set up Puppet environment. I’ll share some statistics with you as well which give context and show the kind of before and after effects of the work.

If you are used to working with Puppet you might be accustomed to runs taking perhaps a minute, or a few minutes for complicated catalogs. When your run time starts to exceed ten minutes it starts getting uncomfortable, and when it gets much more than that people start complaining. We had noticed some of our core systems were taking an extremely long time to complete runs (which were meant to be occurring every 15 minutes) and had also had some reports of extremely long runs for some of our customers.

First off, we ruled out this bug although we suffered from it briefly. If you are still running 2.6 and use run stages, please upgrade to 2.7 or you will soon (or already) be suffering from fairly bad performance during client graph processing. We were seeing several orders of magnitude worse performance as you can see from the bug report.

How can you start to logically proceed towards a performance fix?

Whoever has used Puppet on production servers has probably lived, at least once, the uncomfortable situation when a change on Puppet manifests has introduced "unwanted effects" on target nodes if not real failures.
There are many things that can go wrong when you make a change in whatever part of your Puppet code: you may introduce a wrong configuration file that triggers a service that doesn't restart correctly or have a modification you expected to be limited to some servers and, for some reason, due to wrong logic or poor understanding of your Puppet code, be applied to the "wrong" servers.
Then there are other more mundane errors, that might occur with syntax errors that prevent Puppet from running on a node, but do not affect its functionality.

Let's see a brief summary of the common errors and the testing methods available for Puppet.
Common errors and testing methods

Syntax errors are quite common for the unexperienced Puppet Master still battling with Puppet's somehow picky syntax, but they are also the less dangerous and easier to test. It's a very good idea to force syntax checks, both on Puppet manifests and Erb templates directly at the commit stage, with an hook command of your favorite SCM (search for something like "puppet syntax check hook" for implementation details). Another somehow unusual but effective method to test the manifests syntax is to use Puppet doc. A simple command like:

puppet doc --outputdir /var/www/doc/ --mode rdoc

not only generates a nice Rdoc output like the one here but does a syntax check for all your manifests.
Moreover, if you like to code with an IDE, Geppetto is definitively your choice, having, among the others, also syntax checking and autocompletion features.

Since puppet and later chef took off, there was discussion in the community about code reusability (chef cookbooks or puppet modules). As far as I can see, there is hardly any progress to see and it is still a big PITA to use other people’s cookbooks or modules.

As I am mostly working with chef these days, I will present my ideas with respect to chef cookbooks. From my experience with puppet, I expect them to apply similarly.

If you haven't noticed already, we are getting pretty excited about cloud computing at Evolving Web. Our latest deployment included 5 Rackspace Cloud servers being controlled by Puppet and provided the client with the ability to scale up and down easily depending on traffic. Scaling up involves spinning up a new server instance and configuring it for its role. But what is the best way to configure it? By hand? By scripts? By images? Well technically any of those methods will work, but I want to tell you about one you may not have heard of.

Configuration management tools are becoming more and more important when working in the cloud. Configuration is done programatically, and in one place, so for example to change your PHP <?phpmemory_limit?>, you only edit one file and that change will propagate to all machines that require the config file. This is just a simple example of what configuration management tools can do, and how they can help simplify your sys-admin tasks.

Puppet really is great.  Don't ever get me wrong there.  It's saved me masses of time and money over the last few years, and allowed me to do my job quickly and efficiently.  
That said, it really does have issues with scalability.  After about 20-30 clients using WEBBrick, everything kinda falls over a bit.
We had this problem at Baseblack.  We've now got ~60 workstations and rendernodes all using Puppet for configuration management and software deployment.  It's great.  It far simplifies the process of rolling out updates and upgrades to new build machines.  
The problem we had was most clearly shown by the frequency with which the PSON error comes up.

When I started writing Puppet manifests, I soon figured out, that there are many possibilities for mistakes, which I usually found when test-applying (--noop) changes to a machine. This manual process was quite cumbersome, only to find a missing comma. So I wondered how I could test manifests more easily and on my development machine.

After playing with existing tools like Ohad Levy’s Manitest, I stumbled upon Cucumber and wrote some glue code, so that I could write cucumber features for my manifests.

In my first round of testing, I wrote features like this one:

Cft (pronounced sift) watches a system administrator as she makes changes to a system. Its basic principle is heavily influenced by Gnome's Sabayon. Instead of the desktop though, cft is focused on traditional system administrators and how they maintain machines, mostly with command line tools.

Cft uses puppet as its backbone for expressing the configuration of a system, and for understanding in greater detail what changes the admin has made to the system. Sites that already use puppet to manage their configuration can easily integrate cft's output into the overall site configuration; for sites that use some other means of configuration management, cft's output provides a complete record of changes made and serves as a good starting point of integrating the changes back into the site's configuration.
Example

In lieu of colorful screenshots, a simple example will explain in more detail how cft works: cft organizes changes in sessions to let the user better indicate what aspect of the system they are fixing (the webserver, the mailserver etc.) A simple session to fix the configuration of postfix and to start and enable the service might entail root running the following commands:

tux:1# cft begin postfix
tux:2# yum -y install postfix
tux:3# vi /etc/postfix/main.cf
tux:4# chkconfig postfix on
tux:5# service postfix start
tux:6# cft finish postfix

If you’re using Puppet, and using Puppet to automatically generate configurations for other parts of your infrastructure by using stored configurations, then one thing you have to do is clean up the stored configurations database once you have destroyed a node, otherwise whatever stored config you created will continue to persist despite you removing that node.

Luckily for us, the folks at PuppetLabs ship a script that we can use to clean up after a node. It’s called puppetstoredconfigclean.rb and was located at /usr/share/puppet/ext/puppetstoredconfigclean.rb on my system. You can pass this script the fqdn of a host, or a list of fqdn’s delimited by a space and the script will remove all references from the stored configs DB.

So that’s great, except now I’m leveraging this thing called the cloud(in my case, the cloud is a CloudStack based private cloud) and I have the ability to spawn and destroy nodes on a whim….I don’t want to have to manually run this script each time I destroy a node.

Martin Englund posted an open question to the Puppet mailing list a few days ago asking how people are verifying their systems are built as expected: When you write code, you always use unit testing & integration testing to verify that the application is working as expected, but why don't we use that when we install a system? What are you using to verify that your system is correctly configured and behaves the way you want? He linked to a blog post demonstrating how he was verifying his machines using Cucumber. Coincidentally, about a week earlier at Devopsdays in Gent, I was talking to Felix Kronlage and Bernd Ahlers from bytemine about doing similar things through testing SSH and mail delivery with cucumber-nagios. It's pretty cool people are thinking about doing BDD/TDD with infrastructure, and it's even cooler that the tools are at the point where doing this is actually possible. When doing software testing, your testing tool is normally separate from the language and libraries you're building the software with (but almost always written in the same language). When testing your infrastructure, I think it makes perfect sense to apply this practice.

n a previous blog post, we introduced use cases for separating configuration data from Puppet code. This post (part one of a two part series) will go in-depth with installing, configuring, and using Hiera, but let’s first look at WHY we would need Hiera.
Introduction to the SSH module

One of the benefits of Hiera is its ability to take an existing module and adapt it to a hierarchical-based lookup system. Typically, one of the first modules that people adapt to Puppet code is the SSH module. Let’s look at a simple ssh class definition:

Justin Kulesza and I have been using Puppet to manage the configuration of some virtualized servers. Recently, we added some modules that introduced unexpected dependency cycles. As we worked to untangle the complex series of dependencies, it became difficult to keep track of just how everything was connected.

In order to make sense of the complex web of chaining relationships, I decided to take advantage of Puppet’s ability to generate DOT graph files of resource relationships. Although it’s still sparsely documented [1], this option can be specified either on the command line or in puppet.conf.

Unfortunately, when we first took a look at the resulting relationships.dot for our configuration in GraphViz, it was hardly any help at all…

With the release of facter 1.5.8, it is a great opportunity to revisit this utility and review how it works in conjunction with Puppet. This is a multi-part blog post that will progressively dive deeper into facter and writing custom facts for puppet. Facter is a standalone tool based on Ruby that provides system information in “key => value” pairs:

In facter part 1 we showed how to use facter, as well as how to create custom facts using ruby and environment variables. In this post, we will go into further details how to test, distribute custom facts in your environment whether you are using puppet standalone or in master/agent mode. As a refresher, here’s the ruby code for system_role.rb:

Puppet editing like a pro

I've spent some time recently on setting up my environment to work more productively on writing puppet manifests. This blogpost highlights some of the findings to get me more productive on editing puppet files and modules. Some older information can be found at Editor Tips on the puppetlabs website.
Tip 1: Syntax highlighting,snippet completion

Puppet syntax is very specific, it's important to get clues about missing curly braces, semi-colums, etc .. as fast as possible. There is support for this in the most common editors:

Puppet Faces is one of the features I am proudest to deliver in the new Puppet 2.7.0 release. (Well, the RC series heading toward release, anyway.) They represent a new API for creating subcommands, as well as extending existing subcommands to add extra capabilities‚ without having to modify the supplied source code.

In this post, and the rest of the series, I am going to dive right on down deep into the technical details of Faces and how they can be used to program (and reprogram) a whole bunch of Puppet.

The key capabilities of Faces that make them important are:

Faces provide a defined, documented, robust API to the internals of Puppet.
They let you have that same robust, documented power for your extensions.
Faces have solid backward compatibility baked in to their core.
Faces give you access to a whole mess of capabilities for free.
You get a command line interface to your Face for free…
…and we have a whole lot more coming down the pipe, too.

Collection of Test Driven Infrastructure Links

This pages shows a list of links i've collected related to the topic(s) of

behavior driven infrastructure
test driven infrastructure
test driven administration
test driven deployment
monitoring driven infrastructure

Sometimes you want to have specific versions of a file for a given host, location or operating system while having a default fallback for the standard cases. By using an array of sources in a file resource, with the value of a fact as part of the on-disk file name, you can easily make your configs more granular where needed while keeping sensible defaults.

Facter Part 3: Caching and TTL

Sep 13
By
Nan Liu
in
Blog •
6 Comments »

In facter part 2, we explored how to write custom facts in ruby along with testing and distribution. Since facter is executed every single time puppet runs, there are benefits to cache the information on the client and refresh based on some TTL. Looking ahead, both features are on our roadmap for facter 2.0, but it’s not complicated to implement them for custom facts. To get started, here’s a practical custom fact that retrieves geo information using api.hostip.info:

A tiny ruby rack application which exposes the data from puppet as JSON over HTTP. gem install web-puppet Provides a simple command line tool which runs a built in web server. On accessing the specified port you should get a JSON response containing the current node information.

Syntastic is a syntax checking plugin that runs files through external syntax checkers and displays any resulting errors to the user. This can be done on demand, or automatically as files are saved. If syntax errors are detected, the user is notified and is happy because they didn't have to compile their code or execute their script to find them.

At the time of this writing, syntax checking plugins exist for applescript, c, coffee, cpp, css, cucumber, cuda, docbk, erlang, eruby, fortran, gentoo_metadata, go, haml, haskell, html, javascript, json, less, lua, matlab, perl, php, puppet, python, rst, ruby, sass/scss, sh, tcl, tex, vala, xhtml, xml, xslt, yaml, zpt

One of the issues that crops up when working with Puppet is ensuring that your manifests do what you expect. Errors are bound to happen. A missed brace can make a manifest not compile, or forgetting to include a module or set a variable may mean that running Puppet on the host fails to enforce the expected state. All in all, it would help to have some tools to make sure we’re writing valid code, that it does what it expects, and that if it doesn’t we catch it as soon as possible.

One problem I'm wrestling in my day job at Web Engineering is: how do you know when a system you are building is ready?

When we build a new system, it goes through the following steps:

Jumpstart
Installs the OS and sets up basic configuration, like hostname, domainname, network.
Puppet
System specific configuration
Manual steps
This includes things which are too system dependent to automate, like creating a separate zpool for application data on external storage

Slides Available: http://www.slideshare.net/PuppetLabs/martin-puppetconf
This presentation is for those who are about to start deploying puppet, in the early stages of deployment or those who have a large number of existing systems to bring under puppet control. The presenter will talk about how Sun Microsystems went from old fashioned system administration with less than 30 systems per admin, to fully automated puppet-bliss with over 150 systems per admin. The migration from slowly rotting systems managed by ssh in a loop, to one button push builds in perpetual compliance, contained both pitfalls and valuable lessons, which will be covered during the presentation.

Editors that support Puppet

This section lists all the editors that supports the Puppet language.

An example puppet project, using cucumber-puppet.

Have a look at the commit log to see how the project evolves.

There are several points to take away: * use an overarching catalog policy to specify generic behaviour of all catalogs * use module specific features to describe those parts of your manifests or templates, that contain logic * use module specific features to describe custom extensions like functions * write high level feature files, don't get yourself hung up on details * as a corollary, steps are hardly reusable, they should be tailored to your specific situation * keep your step implementations short, just write the code you wish you had * the bulk of the work should be done by custom helper methods

Are you wondering how to accomplish some task with cucumber-puppet? Do you have any feedback on the work presented here? Just open an issue and let me know!

Patrick Debois, the godfather of DevOps, recently posted an article called Puppet unit testing like a pro. This article spawned some discussion about cucumber-puppet usage and how to get started. As I wanted to develop some ideas about this anyways, I setup an example project on github. You can follow the commit history to see how the project evolved and how I currently think cucumber-puppet should be used:

Although Puppet Enterprise gives you the wonderful tools to automate your Linux administration, you still cannot deploy a server with a single command. With the following script, you can.

Why is this so cool?

When, for instance, a new Tomcat server is needed, I create a DNS entry for that server and give a single command and get some coffee. Before it has cooled down enough to drink, the developer has his Tomcat-app deployed and the server is operational. (It takes about 8 minutes to get from scratch to a fully operational server.)

You’ve bought Pro Puppet, downloaded a couple of modules from the Puppet Forge (and have written some of your own too), and you’re on your way to implementing your Puppet environment when it hits you: something feels bulky with the way you’ve designed your Puppet code. Your modules may not be portable between environments (development, testing, production) without significant tweaks, each of your node declarations may require a number of variables in order for the code to work, or you’re constantly needing to open up your modules to account for changes in your environment.

Puppet is used for managing the infrastructure for many IaaS software packages, including Eucalyptus, OpenStack, and OpenNebula. OpenNebula is an IaaS manager which can not only manage a large amount of different virtualization and public cloud platforms, it can also emulate the API’s provided by EC2 and OCCI. It’s great for creating private and public clouds, as well as hybrids of the two.

Puppet Labs (or, more specifically, Ken Barber) has developed a powerful integration between OpenNebula and Puppet. The installation and configuration of OpenNebula be managed with this solution, and a virtualized infrastructure can be provisioned starting from bare metal using only Puppet code. The module for Puppet that integrates with OpenNebula can be downloaded from the Forge here: http://forge.puppetlabs.com/puppetlabs/opennebula

Sometimes it is pretty obvious when Puppet changes something in your infrastructure and bad things happen in a big dramatic way. Other times it’s not so obvious. It can be invaluable to be able to correlate changes made by Puppet to other events happening in your infrastructure.

For example, in this diagram we have plotted the load average from a group of servers. Blue vertical lines mark points in time when puppet modified a resource on a host in the group. We can see that immediately following a puppet change the load spiked on one of the servers.

Puppet labs's use of the term puppetmaster is rather clever (in contrast to other un-necessarily offensive uses of "master" in the software world).

While I appreciate the clever name, I'm less impressed with the concept.

At May First/People Link we've spent the last several years (including the last couple months in earnest) working to transition management of our 90-some servers from a collection of hand-written bash scripts to puppet.

Over the years, we've worked hard to keep our servers as secure as possible. We have a team of about a half dozen people who all have root access on all servers. It's all key-based access. To help mitigate a disaster if one person's keys were compromised, we've implemented monkeysphere on all servers, allowing us to easily revoke access.

After spending so much time thinking through our root-access strategy and fully implementing the monkeysphere to reduce our exposure to a single point of vulnerability, I was disappointed by puppet's use of a puppet master. For those less familiar with puppet, it goes something like this:

One server (or god forbid multiple servers), run an externally accessible daemon. Each and every server on your network runs a daemon as root that periodically communicates with the puppet master, receives new instructions, and then (again, as root) executes these instructions.

In other words, if your puppet master is compromised, I'm not sure exactly what you would need to do, short of rebuilding every server in your network.

To make matters worse, it seems as though some users generate and store all server ssh keys (private and public) on the puppet master and then push the private keys to their respective nodes. That means an intruder doesn't need to write to the puppet master, just reading these keys would be enough to compromise all servers in your network.

If you’ve worked with configuration management systems for a while, a situation like this has probably cropped up and ruined your day. If you’re lucky enough to have a homogeneous environment, then you might have a staging environment that you can test changes out on, but what if you don’t? A slight mistake in that harmless change you’re working on could stop a service on hundreds of machines or worse (purge the mysql-server package and all it’s data *cough*).

Why you should be writing unit tests for your Puppet modules

Prevent situations like the one above.
Catch any problems moving between Puppet releases before it hits production.
Now we can do this too

One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration.

In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.

Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood's talk on cucumber-nagios.

On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.

It’s been a long time since my last puppet blog post about file content offloading. Two puppetcamps even passed (more on the last one in a next blog article). A new major puppet release (2.6) was even released, addressing lots of performance issues (including the file streaming patch I contributed).

In this new major version, I contributed a new 3rd party executable (available in the ext/ directory in the source tree) that allows to simulate concurrent nodes hammering a puppetmaster. This tool is called puppet-load.

I decided this week-end to try the more popular puppet master stacks and benchmark them with puppet-load (which is a tool I wrote to simulate concurrent clients).

My idea was to check the common stacks and see which one would deliver the best concurrency. This article is a follow-up of my previous post about puppet-load and puppet master benchmarking

The Example42 Puppet modules collection has tried to establish a standard, integrated, reusable and coherent way to manage Puppet modules based on the principles exposed in a pair of PuppetCamp presentations: Puppet Modules Standards and Interoperability and ReUse Your Modules!

It has been initially developed and conceived for Puppet versions 0.2x and has introduced and followed some concepts that I've found quite useful in various situations, such as monitoring and firewall abstraction, "include and play" approach without the enforcing of custom logic, easily extendable multiple operating system support, attempt (only partially successful) to separate "core" module elements from custom ones, standard structure that allows scaffolding and relatively quick creation and upgrade of modules.

Still, experience and the evolution of the Puppet language have suggested new and better ways of "doing things with Puppet" and, more important, most of the modules are going to have some issues (as probably a relevant part of the existing Puppet code in the world) when Puppet 2.8 will be released and variables dynamic scoping will be discontinued.

Puppet is used for managing the infrastructure for many IaaS software packages, including Eucalyptus, OpenStack, and OpenNebula. OpenNebula is an IaaS manager which can not only manage a large amount of different virtualization and public cloud platforms, it can also emulate the API’s provided by EC2 and OCCI. It’s great for creating private and public clouds, as well as hybrids of the two.

Puppet Labs (or, more specifically, Ken Barber) has developed a powerful integration between OpenNebula and Puppet. The installation and configuration of OpenNebula be managed with this solution, and a virtualized infrastructure can be provisioned starting from bare metal using only Puppet code. The module for Puppet that integrates with OpenNebula can be downloaded from the Forge here: http://forge.puppetlabs.com/puppetlabs/opennebula

27
Dec 2011

Whenever I need to setup a new service on one of my hosts, the first thing I do is head to the forge and GitHub to try and find a decent Puppet module that already exists for it.

I almost always leave in disappointment.

Puppet modules are libraries

Much like string.h provides everything you need to manipulate strings in C, your Puppet modules should provide everything needed to manage a service out of the box. By that I mean, I want to pull down your module to enable the functionality I need in Puppet without modifying your module at all.

Find out what's making your Puppet runs so bloody slow!

Taking Puppet Enterprise deployment automation one step further
Although Puppet Enterprise gives you the wonderful tools to automate your Linux administration, you still cannot deploy a server with a single command. With the following script, you can.

Why is this so cool?

When, for instance, a new Tomcat server is needed, I create a DNS entry for that server and give a single command and get some coffee. Before it has cooled down enough to drink, the developer has his Tomcat-app deployed and the server is operational. (It takes about 8 minutes to get from scratch to a fully operational server.)

Justin Kulesza and I have been using Puppet to manage the configuration of some virtualized servers. Recently, we added some modules that introduced unexpected dependency cycles. As we worked to untangle the complex series of dependencies, it became difficult to keep track of just how everything was connected.

In order to make sense of the complex web of chaining relationships, I decided to take advantage of Puppet’s ability to generate DOT graph files of resource relationships. Although it’s still sparsely documented [1], this option can be specified either on the command line or in puppet.conf.


Unfortunately, when we first took a look at the resulting relationships.dot for our configuration in GraphViz, it was hardly any help at all…

Blueprint is a simple configuration management tool that reverse-engineers servers. It figures out what you’ve done manually, stores it locally in a Git repository, generates code that’s able to recreate your efforts, and helps you deploy those changes to production.

As more or less promised in my series of post about Puppet Extension Points, here is the first post about Puppet Internals.

The idea is to produce a series of blog post about each one about a Puppet sub-system.

Before starting, I first want to present what are the various sub-blocks that forms Puppet, or Puppet: the Big Picture:

Listing Puppet Managed Files
Sometimes it's the little niggles that annoy people the most. As my team progress in to puppet they have an annoying habit of asking very good questions; which can sometimes be a struggle to answer. Todays best question was - "How do I tell if this file is under puppets control?"

While there are a couple of different ways to check (grepping through your git checkout or modifying the file and running puppet were the immediate winners) the best way is probably to look inside the catalog and check against the title of the File resources it contains. While this gets you most of the way the problem is a little harder than it looks because of an edge case. If puppet is managing an entire directory then the files in that directory are not explicitly listed in the catalog.

So we need to look in two places, the catalog and state.yaml. Remembering the greps (and the line transformations needed) requires more mental space than I'm willing to invest so I've written puppet-ls to do all the work for me.


$ puppet-ls /etc/mcollective
/etc/mcollective/facts.yaml
/etc/mcollective/server.cfg

Run the command, specify the directory to check and any shown files are puppet managed. It's not a ground breaking script but it can help people migrating to puppet as they bring more of their systems under its control.

How we use cucumber-puppet
by Oliver on Monday, May 16th, 2011.

I got a question recently by email in followup to my presentation at Puppet Camp in Amsterdam about how we use cucumber-puppet. I touched on the subject only briefly in my talk but what I did say is that it revolutionised my approach to Puppet in general. Don’t get too high an opinion of the tool from that statement! Behaviour-driven development in general was a new thing to me and did change my ways of thinking, but my opinions of it in conjunction with Puppet have changed over the months slightly.

There is NO reason, not to use a version control system while developing puppet manifest/modules. Stating that should be an open door. It allows you to go back in time, share things more easily and track your changes. There is a lot of information out there on how to work with git or any other system. But here a few tips that might help you developing modules:
Tip 1 : give each module it's own repo and use a superproject to join

In a lot of blogposts and even in the excellent Pro Puppet book I see people checking in their entire environment directory into version control.

I'm all for version control but if you manage your modules dir as one flat repository, you loose the way to easily update and share modules from the forge. In essence you are doing a copy that starts living it's own life.

Hierarchical data is a good fit for the representation of infrastructure information. Consider the example of a typical company with 2 datacenters and on-site development, staging etc.

All machines need:

ntp servers
sysadmin contacts

By thinking about the data in a hierarchical manner you can resolve these to the most correct answer easily:

Whenever I need to setup a new service on one of my hosts, the first thing I do is head to the forge and GitHub to try and find a decent Puppet module that already exists for it.

I almost always leave in disappointment.
Puppet modules are libraries

Much like string.h provides everything you need to manipulate strings in C, your Puppet modules should provide everything needed to manage a service out of the box. By that I mean, I want to pull down your module to enable the functionality I need in Puppet without modifying your module at all.

It’s been a long time since my last blog post, almost a year. Not that I stopped hacking on Puppet or other things (even though I’m not as productive as I had been in the past), it’s just that so many things happened last year (Memoir’44 release, architecture work at Days of Wonder) that I lost the motivation of maintaining this blog.

But that’s over, I plan to start a series of Puppet internals articles. The first one (yes this one) is devoted to Puppet Extension Points.

Since a long time, Puppet contains a system to dynamically load ruby fragments to provide new functionalities both for the client and the master. Among the available extension points you’ll find:

Puppet unit testing like a pro

A big thanks to Atlassian for allowing me to post this series!!

In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.

Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood's talk on cucumber-nagios.

On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.

Because we are using Puppet here at Atlassian, I was out to understand the current state of puppet testing. A lot can already be found at http://puppetlabs.com/blog/testing-modules-in-the-puppet-forge/

Note that I've purposely named this blog 'Puppet unit testing', as the tests I'm describing now, don't run against an actual system. Therefore it's hard to test the actual behavior.

After a delightfully overdone comedy routine by Tom Limoncelli and Doug Hughes, Ben Rockwood took the stage to deliver his keynote speech titled “The DevOps Transformation.” DevOps is a growing movement in IT designed to blur the line between developers and operators. DevOps is a cultural and professional movement, not a tool or product. It is not a title, nor is it a person. Most importantly, it is not just dev & ops, it’s *dev*ops*.

Ben opened with an very interesting discussion of the philosophical foundations of DevOps. Appropriately done, DevOps first seeks to answer “why?”, then “how?”, and finally “what?”. Done backwards, which is often the case, systems often fail to meet the goals. Ben also brought up Ackoff’s Five Contents of the Mind: data, information (the “reporters’ questions”), knowledge (how to), understanding (why), wisdom (insight). Junior Admins fall between information and knowledge. Senior Admins have reached understanding, and Architects live in wisdom.

The talk progressed into a history of quality management, from the early 20th century to modern frameworks like ITIL. Ben argues that “quality is the degree to which a good or service is what the customer expects it to be.” For example, a Big Mac is not necessarily a great hamburger, but a quality Big Mac is one that meets the customer’s expectations. Similarly, the only measure of the quality of a steak is it’s “steakiness”.

Although it’s possible to work toward DevOps nirvana group-by-group, the best way to work with both Dev and Ops at the same time. This could be as easy as arranging a get-together outside the office. “Beer is the best tool to implement DevOps,” Ben claims.

Ben left the audience with some protips to set the tone for the rest of the conference:

DevOps is a journey of discovery not a destination.
DevOps starts with “why?”, and a holistic vision.
Quality is a result of the effectiveness of the interactions across the entire value stream.
The cloud is here to stay and it’s awesome.
If it ain’t fun, it ain’t working.

Firstly, a little background on this topic. During PuppetConf this year I attended a very interesting talk by Digant C. Kasundra about the Puppet implementation at Stanford University. At one point he asked, “Who uses definitions?”, and I raised my hand. The next question was, “Who uses parameterized classes?”, and I also raised my hand, this was followed by “Who uses both?”, and I was one of a small minority of people who raised a hand again. The next question was, “Who knows what the difference between a parameterized class and a definition is?”. I took a second or two to think about this and the talk moved on after no-one in the audience raised their hand, and I didn’t get a chance to answer this in the Q&A due to running out of time, but I’ve been thinking about it since. Digant’s view was that the two are very similar and he advocates the use of definitions, which is certainly not a bad point of view, but I don’t think you should be using one or the other, but rather, use either one appropriately, and given the fact that no-one could answer Digant’s question in his talk, I felt it worth expanding on the issue in a blog post and would really welcome any feedback.

Lots has been written about Vagrant. It simply is a great tool: people use it as a sandbox environment to develop their Chef recipes or Puppet manifests in a safe environment.

The workflow usually looks like this:

you create a vagrant vm
share some puppet/chef files via a shared directory
edit some files locally
run a vagrant provision to see if this works
and if you are happy with it, commit it to your favorite version control repository

Specifically for puppet, thanks to the great work by Nikolay Sturm and Tim Sharpe, we can now also complement this with tests written in rspec-puppet and cucumber-puppet. You can find more info at Puppet unit testing like a pro.

One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration.

There is NO reason, not to use a version control system while developing puppet manifest/modules. Stating that should be an open door. It allows you to go back in time, share things more easily and track your changes. There is a lot of information out there on how to work with git or any other system. But here a few tips that might help you developing modules:

Increasingly organizations are moving away from a homogeneous desktop environment. In addition to systems with Microsoft Windows, businesses are using various flavors of Unix including OS X across their environment. These systems can bring lots of advantages in meeting unique user and development needs, but they still need to adhere to corporate standards for security, configuration management and overall IT control. Puppet can help meet your corporate needs for these systems quickly and efficiently.

Whether your challenge is simply gaining visibility into the Mac OS X desktops deployed in your environment, meeting security standards, enforcing least privilege requirements, or simply accelerating administration and updates on these desktops, Puppet helps bring your Mac desktops up to your corporate standards. Puppet allows you to be more efficient in your management of these resources and to take the administration of these systems out of the hands of your end users.

One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration.

Puppet is a wonderful system automation tool, however the learning curve can be a little steep. We've collected some of the errors messages and "strange" behaviour you may come across together with explanations to help overcome these hurdles and boost adoption of this fabulous tool.

If you have any useful errors and explanations, please do send them in and we'll update this article.

OpenNebula Demo Environment

This repository contains a self-contained Puppet environment which demonstrates the capabilities of Puppet and OpenNebula integration.

Just last week, Amazon Web Services rolled out the first production release of the Amazon Linux AMI and with it some powerful new features for CloudFormation. There’s a lot there (like updating stacks and IAM integration) but we at DevStructure are most excited about Application Bootstrapping.

The new Amazon Linux AMI comes with the aws-cfn-bootstrap package, which can unpack tarballs, place configuration files, install packages, and restart services at provision time. Sound familiar? That’s right, this new section of the CloudFormation template language uses the blueprint(5) file format!

Blueprint
Blueprint reverse-engineers servers

Easy configuration management.
Detect relevant packages, files, and source installs.
Generate reusable server configs.
Convert blueprints to Puppet or Chef.
No DSLs, no extra servers, no workflow changes.

Blueprint looks inside popular package managers, finds changes you made to configuration files, and archives software you built from source. It runs on Debian- and RPM-based Linux distros with Python >= 2.6 and Git >= 1.7.
Blueprint I/O moves blueprints around

Centralized configuration management.
Export and backup server configurations.
Push and pull blueprints anywhere.
Bootstrap servers painlessly.

Blueprint I/O pushes and pulls blueprints to and from a Blueprint I/O Server, making it easy to use blueprints anywhere. DevStructure provides a free Blueprint I/O Server at https://devstructure.com, which stores blueprints in Amazon S3. Alternatively, you can build your own backend server implementing the Blueprint I/O API.

This project is based on the Amazon Cloud Formation framework. It adds a Puppet Face to enable simple creation of a fully operational Puppet Enterprise stack utilizing module content from the Puppet Forge.

This project contains an aws cloud formation template that is capable of deploying a cluster of nodes with puppet enterprise installed. http://aws.amazon.com/cloudformation/

The general goal is that you can specify your desired stack in a simple yaml document and use a single command to create an entirely functional puppet environment from that configuration document.

One config file, one command and you're done!

Puppet, Private Clouds and Building Solutions - Ken Barber
Ken Barber's talk from the Puppet Users Meet Up at Forward on Thursday 20th October 2011.

The presentation used in this video, with links to all the modules demoed can be found here: http://links.puppetlabs.com/kensdemo

If you’ve worked with configuration management systems for a while, a situation like this has probably cropped up and ruined your day. If you’re lucky enough to have a homogeneous environment, then you might have a staging environment that you can test changes out on, but what if you don’t? A slight mistake in that harmless change you’re working on could stop a service on hundreds of machines or worse (purge the mysql-server package and all it’s data *cough*).
Why you should be writing unit tests for your Puppet modules

Prevent situations like the one above.
Catch any problems moving between Puppet releases before it hits production.
Now we can do this too

At Sonian, we monitor an ever changing number of Amazon EC2 instances. As I write this post, that number is 476, expected to rise and fall before the day is done. The "elastic" nature of our infrastructure makes monitoring it a not so trivial task.

We have found the standard tools from the community toolbox to be inadequate when operating in the "cloud". Up until recently, Sonian utilized several tools in conjunction to monitor systems and collect metrics; Nagios, Collectd, Graphite, and Ganglia.

Sensu is an open source cloud monitoring framework, purpose built for the cloud, by Sonian.

Sensu comes from the Japanese name for a folding fan made from bamboo and paper, which represent the Fanout exchange concept utilized by the AMQP (protocol).

The Sensu framework provides foundation for monitoring cloud infrastructure and applications.

Fully extensible via source code, with active community.

Framework includes monitoring server, platform-independent agent, and web-based dashboard.