Cucumber is NOT a testing framework!

I have been repeating this statement over and over on the Cucumber forum, but apparently with no good result. To me this is a simple statement and I fail to understand what’s difficult to understand in its simplicity: Cucumber (JVM, Ruby, C, the flavour doesn’t matter) is not a testing framework.

I’ll try to further explain the statement, not because I believe it needs to be explained, but hoping an explanation will solve doubts. To do that I’ll try to state the facts in way as much practical as I can.

JUnit, TestNG and DBUnit are examples of a testing framework: they provide facilities to put a software in a certain condition, leverage some parts of the software and assert the outcome responds to the expected parameters.

The goal of those frameworks is to provide the tools to achieve the result (testing the software) with the least possible effort and the maximum outcome. As such, they have programming structures for asserting conditions are met, for setting up the test rig and tear down the test rig as well, ensuring the system is always in a clean condition before a new test is run.

Cucumber doesn’t have any of those facilities, simply because it is not a testing framework.

Cucumber is a specifications tool, something completely different. It uses Gherkin (the language) to describe in a natural language the software specifications.

Does anyone of you remember the Use Case Template? 

Well a Cucumber feature is much more like one of those than a bunch of tests, would you agree?

There was once a time when we were describing software systems by collecting a few of those use case templates in a Word document, right?

Now, it wasn’t much of a trouble to describe a system like that, at least at first instance, but after a couple of maintenance cycles the above documents were not describing the system any more: I haven’t found one single situation where the above documents were maintained and kept in sync with the implementation. So the system documentation was just obsolete after a few months of operation.

Why was that? Because the cost of maintaining such documentation was not justified by the short term benefit of doing so: why should I spend 30 minutes to find the correct place where to apply the patch in the system documentation if the fix itself costs me 15 minutes?

So, Cucumber steps in and tries to transform the above specifications into something valuable and alive. How?

The specification format (or structure, if you prefer) is almost free, so to leave the writer the freedom to express himself freely. This is even less structured than the template above, but there are good practices telling us how to maximize the result, like providing the list of actors and the general goal at the very beginning.

The file is plain text, so to avoid any requirement on the tool used to open and modify the document and doesn’t add any formatting to avoid reader to be distracted or writer being dragged into endless make it prettier sessions.

A file represents a use case (feature in the Gherkin language), so you end up having multiple files, each one representing a piece of the software. This greatly simplifies version management, collaboration and merging, enabling multiple writers and revisors working on a single system. It’s not uncommon to use the same version control system in use for the system to store the documentation.

Files can be structured in a hierarchical fashion via folder structure, so big systems with lots of features can organize their specifications.

Files can be annotated via tags so to create parallel organizational structures, with the folder structure still being the prominent one: this enables additional categorizations, useful to track other associations between use cases (features), as much as what the introductory Use Case Diagram was doing in the Word document.

What makes the biggest confusion, though, is Gherkin files can be executed.

That is what Cucumber provides: the support software structures and an execution environment for Gherkin files.

Why? To create a connection between the system documentation and the documented system, so to ensure the documentation is aligned to the system it describes.

How? By mapping each statement in each scenario with some lines of code in the language you like (Ruby? C++? Java? Scala?). If the code ensures somehow the system does what it is expected to do, than the documentation is in sync with the implementation.

Only this last part resembles a test. Only at this stage there is some code involved. And usually testing frameworks and automation libraries are used to do such verification, like JUnit and Selenium.

So, if you need to test your system, please use a testing framework! If, instead, you want to document your system, you are welcome to use Gherkin and Cucumber!

Generalization pitfalls

Experienced developers, including me, tend to prefer generalized code over highly specialized one, but they usually love very simple and highly readable code much more and the two don’t always pair nicely.

Disclaimer

I’ll use Java and Cucumber to make my point clear (I hope) but what I’m going to assert is not strictly related to neither of those, actually is not even strictly related to programming!

I’ve smashed my face into this problem when I started using Cucumber and my team decided a single generalized method to map to a UI button click should be sufficient: I believe many of you would agree the code below isn’t a bad idea.

@When("I click on \"(.*)\" button")
public void buttonClick(String buttonLabel) {
  // ui specific code to click the button
}

I now tend to advice against such generalizations because it’s very rare a single method can suffice your needs because they are too generic to be implementable. Let me explain.

The very first step to implement such method is to resolve the label into some sort of reference to the UI button to click and then click such component. Now, unless you establish some sort of very strict convention you’ll end up with something like the following:

@When("I click on \"(.*)\" button")
public void buttonClick(String buttonLabel) {
  UIButton button = null;
  if ("OK".equals(buttonLabel))
    button = this.getOkButton();
  else if ("Cancel".equals(buttonLabel))
    button = this.getCancelButton();
  else if ("Save".equals(buttonLabel))
    button = this.getSaveButton();
  else if ("Edit".equals(buttonLabel))
    button = this.getEditButton();
  else if ("Delete".equals(buttonLabel))
    button = this.getDeleteButton();
  else
    // button unknown!
  button.click();
}

I’m obviously simplifying the code above for sake of readability: the very first implementation of that method was 200 lines when we got to 20 buttons!

Please note the issue is not related to the Cucumber capturing group nor to the string parameter, but to the use we are doing of it: we are de-generalizing a generalized method!

Now, wouldn’t it been simpler and more readable to have something like the following?

@When("I click on \"OK\" button")
public void buttonOkClick() {
  UIButton button = this.getOkButton();
  button.click();
}

@When("I click on \"Cancel\" button")
public void buttonCancelClick() {
  UIButton button = this.getCancelButton();
  else if ("Save".equals(buttonLabel))
  button.click();
}

@When("I click on \"Save\" button")
public void buttonSaveClick() {
  UIButton button = this.getSaveButton();
  button.click();
}

@When("I click on \"Edit\" button")
public void buttonEditClick() {
  UIButton button = this.getEditButton();
  button.click();
}

@When("I click on \"Delete\" button")
public void buttonDeleteClick() {
  UIButton button = this.getDeleteButton();
  button.click();
}

The code is longer, I agree, but it’s way much clear, a lot easier to debug and, above all, if there’s a missing button mapping a clear error is returned!

You might argue this case is not applicable to you because a strict button naming convention is well established on your project and a button is always identifiable from it’s label (let’s suppose you are using an HTML based UI and each button has an id in the form of label-btn so that the above can be resolved into the following:

@When("I click on \"(.*)\" button")
public void buttonClick(String buttonLabel) {
  // enforce naming convention here and prepare the buttonLabel
  UIButton button = this.getButton(buttonLabel + "-btn");
  button.click();
}

Lovely, isn’t it? Now a paginated data set comes in, something like the one in the picture below, and you suddenly need to assign an id of «PreviousPage-btn (note the initial angle quote) to an HTML tag!

If you modify the above code into the following one you are falling into the same pitfall, no excuse granted!

@When("I click on \"(.*)\" button")
public void buttonClick(String buttonLabel) {
  if (buttonLabel.startsWith("«") || buttonLabel.startsWith("»"))
    buttonLabel = buttonLabel.substring(6);
  UIButton button = this.getButton(buttonLabel + "-btn");
  button.click();
}

The code above might still look clean and neat, but it is indeed a source of problems: it’s your first shovel of dirt while digging your own grave. You should have gone with the following instead:

@When("I click on previous page button")
public void buttonPrevPageClick() {
  UIButton button = this.getButton("prevPage-btn");
  button.click();
}

@When("I click on next page button")
public void buttonNextPageClick() {
  UIButton button = this.getButton("nextpage-btn");
  button.click();
}

@When("I click on \"(.*)\" button")
public void buttonClick(String buttonLabel) {
  UIButton button = this.getButton(buttonLabel + "-btn");
  button.click();
}

Again, a little more code but a lot cleaner and readable and, above all, it doesn’t imply you have to type EXACTLY the same button label characters into your feature files or do some killer loops to describe a test for a stupid button!

Now, if you read this post up to this point you have probably fallen into this pitfall multiple times and you probably still have doubts about what I’m saying, probably due to the fact you really love the generalization concept/method, so here is the generalized reason why method generalization is not always a good practice:

While performing step to stepdef matching Cucumber is performing a de-generalization, going from a generic string to a specific method.
By using stepdef parameters you are performing a generalization, allowing multiple strings to match the same method.

But when within the stepdef you add control logic on the parameters you are performing an additional de-generalization, trying to identify a specific match for a specific parameter value: something Cucumber already tried to do during stepdef matching!

It’s like trying to change gold for gold by transforming gold into silver and then back into gold, all with little or no gain and a lot of unnecessary confusion.

Let Cucumber do it’s work and don’t try to generalize everywhere, instead let the natural language unleash it’s expressive power and use it at your own benefit.

Now, while I used Cucumber steps and stepdefs as a demonstration, this concept is applicable in many other contexts, and here it comes my statement in a more generalized version:

Question yourself twice whenever you are trying to generalize something right after a de-generalization has just happened: you’ll discover you are doing the wrong thing 99% of the time.

Whenever you are certain your case falls within that remaining 1% then you are surely doing the wrong thing: doubting about yourself is your only salvation!

If you allow me the hyperbole, wouldn’t be messy to develop by always using Java Reflection? Well, Java Reflection is the generalization above the whole Java framework: one API to rule them all!

Living documentation can be readable and fast

I’ve just found out an interesting article about ATDD and I want to share it.

http://claysnow.co.uk/living-documentation-can-be-readable-and-fast/

Embedding screenshots in Cucumber JVM

I was trying to improve our automated test suite and I thought that it could be useful to capture a screenshot of the browser whenever a test fails.
The current Cucumber JVM implementation highly simplifies this task, but the task is not achieved the way I thought, so this is the reason for this post.

Normally you would use a JUnit TestRule to augment all your test cases with a feature to take screenshots, but for Cucumber JVM it’s much easier thankfully to the Execution Hooks:

public class ScreenshotHook {
  @Autowired
  private WebDriver driver;

  @After
  public void embedScreenshot(Scenario scenario) {
    if (scenario.isFailed()) {
      try {
        byte[] screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
        scenario.embed(screenshot, "image/png");
      } catch (WebDriverException wde) {
        System.err.println(wde.getMessage());
      } catch (ClassCastException cce) {
        cce.printStackTrace();
      }
    }
  }
}

As you can see I’m injecting the WebDriver instance (I’m using the Cucumber JVM Spring integration) and defining an embedScreenshot method which is annotated with cucumber.annotation.After: this is the important bit because methods annotated as such will be executed after each cucumber scenario.

For sake of completeness I want to tell you there is another hook cucumber.annotation.Before that can be used for things like logging into your application eliminating those repetitive log in steps.

Please note that in pre 1.1.0 versions of Cucumber JVM Scenario was ScenarioResult.

A few Cucumber tricks

I’ve been a fan of TDD since I discovered it a few years ago. Thankfully to my friend and colleague Augusto a few months ago I’ve discovered the beauty of BDD and ATDD: I got so fascinated by this practice I started an Open Source project for an Eclipse plugin featuring a rich editor for Cucumber feature files. It’s called Natural and I think it’s pretty cool if you want to check it out.

After having used Cucumber for a while and many, many mistakes I now have got some experience I wish to share with you.

One of the key features of feature files is that they represent a live project documentation, something developers write and maintain and everyone can understand. We believe so much in this concept that we wanted to give our product owner and business partners a nice view on those files, that’s why we used Relish to publish them in a pretty colored format. At the same time we had to face some problems while practically use these files for automation testing.

After a few development cycles we were so excited by Cucumber and ATDD that our automation test suite counted hundreds of tests: it was consuming too much time to be completely executed on each commit by our CI system.
My suggestion is to track in your feature files the development cycle each test has been developed: this can be easily achieved by using a tag, something like @sprint-7.
This way you can instruct your CI system to execute the current sprint automation on each commit and schedule a full test nighttime (and launch time possibly).

Another issue was we were developing automation test before the functionality was implemented, so following TDD principles. By doing this our CI was reporting a build failure until the functionality was completely implemented. This is not a problem as it is expected to have failing tests during development, but the continuous failure messages received by the team members brought us to the point we were starting to ignore those messages, vanquishing the purpose of the automation suite and the CI system itself. The solution I suggest is to temporarily annotate the tests that are expected to fail with @future, instructing your CI to skip any test annotated as such: the developers can still execute those tests, but the CI will just ignore them. Once a feature is completed it will be developer’s responsibility to remove such annotation.

With hundreds of tests we had problems in having the tests properly organized so here comes the folder structure and naming convention. Please consider those files not as test, but as your application documentation: as such you want to organize then in a manner they are easily accessible by a business person. My suggestion is to organize the tests in files describing different features, grouping the features into folders representing functional areas. Considering company search and report search capabilities I can imagine a search.feature and an advanced-search.feature, both inside a company folder and the same file names in a report folder.

Defect and user stories tracking, if desirable, can be once again achieved by using tags like @US1234 and @DE9876, but I would rather avoid cluttering the feature files with such information as they will tend to distract a business person from the real value of those files: application documentation.

Automation test coverage is a report we found business users find quite valuable, but it was tricky to achieve considering our application was a web one. Nevertheless we managed to have it running on our CI thankfully to Maven and its amazing set of plugins.

I’ve prepared a template Maven project for those reading this post to use as a starting point and guideline if you like the solutions we adopted: it will definitely help me in the future to avoid redoing all the steps from scratch! Please, read the README file before asking for clarifications 🙂