Integration Testing Resque with Cucumber
Square takes integration testing seriously. We use Cucumber and RSpec to test our code during every step of development: on developer pairing machines, continuous integration servers, staging servers and production servers.

The Problem
Traditional Cucumber tests exercise the web stack from the web server through the database, but they don’t typically cover asynchronous tasks like background processing and scheduled jobs. Integration tests involving these asynchronous jobs are hard to write due to race conditions between the test process and the background workers. We faced this problem when we began using Resque for processing background jobs. We love Cucumber and we love Resque; we wanted to find a way to use them together.

When I started writing Puppet manifests, I soon figured out, that there are many possibilities for mistakes, which I usually found when test-applying (--noop) changes to a machine. This manual process was quite cumbersome, only to find a missing comma. So I wondered how I could test manifests more easily and on my development machine.

After playing with existing tools like Ohad Levy’s Manitest, I stumbled upon Cucumber and wrote some glue code, so that I could write cucumber features for my manifests.

In my first round of testing, I wrote features like this one:

Ten tips for writing better Cucumber steps
January 1st, 2012

Martin Englund posted an open question to the Puppet mailing list a few days ago asking how people are verifying their systems are built as expected: When you write code, you always use unit testing & integration testing to verify that the application is working as expected, but why don't we use that when we install a system? What are you using to verify that your system is correctly configured and behaves the way you want? He linked to a blog post demonstrating how he was verifying his machines using Cucumber. Coincidentally, about a week earlier at Devopsdays in Gent, I was talking to Felix Kronlage and Bernd Ahlers from bytemine about doing similar things through testing SSH and mail delivery with cucumber-nagios. It's pretty cool people are thinking about doing BDD/TDD with infrastructure, and it's even cooler that the tools are at the point where doing this is actually possible. When doing software testing, your testing tool is normally separate from the language and libraries you're building the software with (but almost always written in the same language). When testing your infrastructure, I think it makes perfect sense to apply this practice.

One problem I'm wrestling in my day job at Web Engineering is: how do you know when a system you are building is ready?

When we build a new system, it goes through the following steps:

Jumpstart
Installs the OS and sets up basic configuration, like hostname, domainname, network.
Puppet
System specific configuration
Manual steps
This includes things which are too system dependent to automate, like creating a separate zpool for application data on external storage

Patrick Debois, the godfather of DevOps, recently posted an article called Puppet unit testing like a pro. This article spawned some discussion about cucumber-puppet usage and how to get started. As I wanted to develop some ideas about this anyways, I setup an example project on github. You can follow the commit history to see how the project evolved and how I currently think cucumber-puppet should be used:

Mit den üblichen Check-Plugins welche Nagios von Haus aus mitbringt, lässt sich die Verfügbarkeit von Diensten wunderbar prüfen. Hier beginnt aber auch schon das Dilemma: Die Verfügbarkeit eines Dienstes sagt noch nicht wirklich viel über die praktische Nutzbarkeit des Dienstes aus. Ein Webserver kann weiter munter Webseiten mit dem Status-Code 200 ausliefern, auch wenn die Webseite für den Nutzer praktisch nicht mehr nutzbar ist. Stellen Sie sich die eBay-Webseite ohne funktionierende Suchmöglichkeit vor – die Seite ist zwar grundsätzlich verfügbar, ein für den Nutzer wichtiger Bestandteil der Funktionalität ist jedoch nicht.

This document is a step-by-step guide for Cucumber, a tool that is quickly becoming the weapon of choice for many agile teams when it comes to functional test automation, creating executable specifications and building a living documentation.

Puppet unit testing like a pro

A big thanks to Atlassian for allowing me to post this series!!

In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.

Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood's talk on cucumber-nagios.

On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.

Because we are using Puppet here at Atlassian, I was out to understand the current state of puppet testing. A lot can already be found at http://puppetlabs.com/blog/testing-modules-in-the-puppet-forge/

Note that I've purposely named this blog 'Puppet unit testing', as the tests I'm describing now, don't run against an actual system. Therefore it's hard to test the actual behavior.

Turnip is a Gherkin extension for RSpec. It allows you to write tests in Gherkin and run them through your RSpec environment. Basically you can write cucumber features in RSpec.

Gherkin is the language that Cucumber understands. It is a Business Readable, Domain Specific Language that lets you describe software’s behaviour without detailing how that behaviour is implemented.

Gherkin serves two purposes – documentation and automated tests. The third is a bonus feature – when it yells in red it’s talking to you, telling you what code you should write.

Gherkin’s grammar is defined in the Treetop grammar that is part of the Cucumber codebase. The grammar exists in different flavours for many spoken languages (37 at the time of writing), so that your team can use the keywords in your own language.

There are a few conventions.

Single Gherkin source file contains a description of a single feature.
Source files have .feature extension.

We are evaluating two tools for testing modules. One is rspec-puppet, the other is cucumber-puppet. The hope is to get standards for testing into the module forge and hook it into a CI framework such as hudson or jenkins. This will enable module forge users to immediately evaluate a module in terms of its test failure rate, and enable developers to see the intended behavior of the module. The higher level goal is to open a discussion among module developers on the Puppet Forge about the intended behavior of modules, what they need to do, what they should and should not be doing.

Tom Sulston explains how to manage systems with Cucumber and Puppet based on BDD principles, including practical tricks and pitfalls. The session demoes using those tools.