In a recent blog post we discussed verifying and testing your Puppet code, which generated a large amount of interest. One of the readers asked why we’d bother with rspec-puppet if it simply verifies attributes of a resource. It’s true that, if you have a manifest that simply contained a collection of static resources, rspec-puppet doesn’t provide much additional value. It shines when you start developing define resources, parametrized classes, or any module that contains complex logic. In this blog post we’ll use a staging module to demonstrate Test Driven Development (TDD) with Puppet by writing rspec-puppet. For people who aren’t familiar with TDD, it refers to writing the rspec-puppet test cases before writing the Puppet manifests. This method allows you to be more confident in your code (by validating the unit test), to isolate problems (by testing code as it’s being developed), and to discover bugs earlier, when they are easier to troubleshoot and fix. In the diagram below, the complexity, time, and cost of testing increases as we move up the testing stack from unit (rspec), module acceptance (virtual machine), separate

Integration Testing Resque with Cucumber
Square takes integration testing seriously. We use Cucumber and RSpec to test our code during every step of development: on developer pairing machines, continuous integration servers, staging servers and production servers.

The Problem
Traditional Cucumber tests exercise the web stack from the web server through the database, but they don’t typically cover asynchronous tasks like background processing and scheduled jobs. Integration tests involving these asynchronous jobs are hard to write due to race conditions between the test process and the background workers. We faced this problem when we began using Resque for processing background jobs. We love Cucumber and we love Resque; we wanted to find a way to use them together.

After doing TDD full time for years, I have a hard time writing code without a test. One example that I find particularly difficult is writing data migrations.

Some schema changes require more than just setting a default value for all existing rows. For example, let’s say you have this schema:

Lots has been written about the merits of continuous integration and I am a firm believer in it. However, while our team at work uses test-driven development, we still don’t have a CI server. This week I went on a quest to change that.

In order to minimize maintenance, we decided to give tddium a try. It is a hosted CI service for Ruby (on Rails) projects.

Tddium is hosted test and continuous delivery service for Ruby that speeds up your development process, parallelizes your tests automatically, and frees you from managing your test environment.

Also check out our Heroku Add-On for fast, easy setup CI for your Heroku apps!

Work faster and smarter.

Works with any Ruby test suite
Integrates with any git server
Detects your app's dependencies and sets them up automatically
Runs your tests in parallel automatically
Automatically deploys your code when tests pass
Sends build notifications via email and Campfire
Use your favorite technologies.

RSpec, Cucumber, Spinach, Turnip, Test::Unit, and Jasmine (via Webkit)
MySQL, Postgres, Mongo, Redis, Memcached, Riak
Solr, ThinkingSphinx, and ElasticSearch
Capybara, Selenium, WebKit, Culerity
Ruby 1.9.3, 1.9.2, 1.8.7, and REE
CouchDB and RabbitMQ

Martin Englund posted an open question to the Puppet mailing list a few days ago asking how people are verifying their systems are built as expected: When you write code, you always use unit testing & integration testing to verify that the application is working as expected, but why don't we use that when we install a system? What are you using to verify that your system is correctly configured and behaves the way you want? He linked to a blog post demonstrating how he was verifying his machines using Cucumber. Coincidentally, about a week earlier at Devopsdays in Gent, I was talking to Felix Kronlage and Bernd Ahlers from bytemine about doing similar things through testing SSH and mail delivery with cucumber-nagios. It's pretty cool people are thinking about doing BDD/TDD with infrastructure, and it's even cooler that the tools are at the point where doing this is actually possible. When doing software testing, your testing tool is normally separate from the language and libraries you're building the software with (but almost always written in the same language). When testing your infrastructure, I think it makes perfect sense to apply this practice.

In facter part 1 we showed how to use facter, as well as how to create custom facts using ruby and environment variables. In this post, we will go into further details how to test, distribute custom facts in your environment whether you are using puppet standalone or in master/agent mode. As a refresher, here’s the ruby code for system_role.rb:

Collection of Test Driven Infrastructure Links

This pages shows a list of links i've collected related to the topic(s) of

behavior driven infrastructure
test driven infrastructure
test driven administration
test driven deployment
monitoring driven infrastructure

One problem I'm wrestling in my day job at Web Engineering is: how do you know when a system you are building is ready?

When we build a new system, it goes through the following steps:

Jumpstart
Installs the OS and sets up basic configuration, like hostname, domainname, network.
Puppet
System specific configuration
Manual steps
This includes things which are too system dependent to automate, like creating a separate zpool for application data on external storage

How we use cucumber-puppet
by Oliver on Monday, May 16th, 2011.

I got a question recently by email in followup to my presentation at Puppet Camp in Amsterdam about how we use cucumber-puppet. I touched on the subject only briefly in my talk but what I did say is that it revolutionised my approach to Puppet in general. Don’t get too high an opinion of the tool from that statement! Behaviour-driven development in general was a new thing to me and did change my ways of thinking, but my opinions of it in conjunction with Puppet have changed over the months slightly.

This document is a step-by-step guide for Cucumber, a tool that is quickly becoming the weapon of choice for many agile teams when it comes to functional test automation, creating executable specifications and building a living documentation.

Puppet unit testing like a pro

A big thanks to Atlassian for allowing me to post this series!!

In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.

Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood's talk on cucumber-nagios.

On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.

Because we are using Puppet here at Atlassian, I was out to understand the current state of puppet testing. A lot can already be found at http://puppetlabs.com/blog/testing-modules-in-the-puppet-forge/

Note that I've purposely named this blog 'Puppet unit testing', as the tests I'm describing now, don't run against an actual system. Therefore it's hard to test the actual behavior.

In the Ruby world, we have a few different testing libraries. One of the ones that has gained a lot of popularity and use is rspec. RSpec takes a slightly different approach to the idea of testing applications, by testing behavior rather than only specific methods. I thought I'd take some time to show everyone why rspec is so useful in testing your applications.

Welcome to the CI for Vagrant, a tool for building and distributing virtualized environments. In order to maintain the level of stability that users of Vagrant have come to expect, this CI system will run a wide range of tests against all commits to the Vagrant project.

minitest provides a complete suite of testing facilities supporting TDD, BDD, mocking, and benchmarking.

Simple depth first ruby testing, watch and learn.

What’s it for?

When you spend time researching something new, it’s pretty easy to forget what you found. That means next time, you have to re-research it again.

A lot of the tech jobs we do manually aren’t challenging or fun, but they’re finicky and have to be done just right — they’re chores. Things that are important to do, but that are better automated than done manually.

That’s what babushka is for. Once you describe a job using its DSL, babushka knows not only how to accomplish each part of the job, but also how to check if each part is already satisfied. For each component of the job, a test, along with the code to make that test pass — test-driven sysadmin.