Cucumber is NOT a testing framework!

I have been repeating this statement over and over on the Cucumber forum, but apparently with no good result. To me this is a simple statement and I fail to understand what’s difficult to understand in its simplicity: Cucumber (JVM, Ruby, C, the flavour doesn’t matter) is not a testing framework.

I’ll try to further explain the statement, not because I believe it needs to be explained, but hoping an explanation will solve doubts. To do that I’ll try to state the facts in way as much practical as I can.

JUnit, TestNG and DBUnit are examples of a testing framework: they provide facilities to put a software in a certain condition, leverage some parts of the software and assert the outcome responds to the expected parameters.

The goal of those frameworks is to provide the tools to achieve the result (testing the software) with the least possible effort and the maximum outcome. As such, they have programming structures for asserting conditions are met, for setting up the test rig and tear down the test rig as well, ensuring the system is always in a clean condition before a new test is run.

Cucumber doesn’t have any of those facilities, simply because it is not a testing framework.

Cucumber is a specifications tool, something completely different. It uses Gherkin (the language) to describe in a natural language the software specifications.

Does anyone of you remember the Use Case Template? use-case-template-visio

Well a Cucumber feature is much more like one of those than a bunch of tests, would you agree?

There was once a time when we were describing software systems by collecting a few of those use case templates in a Word document, right?

Now, it wasn’t much of a trouble to describe a system like that, at least at first instance, but after a couple of maintenance cycles the above documents were not describing the system any more: I haven’t found one single situation where the above documents were maintained and kept in sync with the implementation. So the system documentation was just obsolete after a few months of operation.

Why was that? Because the cost of maintaining such documentation was not justified by the short term benefit of doing so: why should I spend 30 minutes to find the correct place where to apply the patch in the system documentation if the fix itself costs me 15 minutes?

So, Cucumber steps in and tries to transform the above specifications into something valuable and alive. How?

The specification format (or structure, if you prefer) is almost free, so to leave the writer the freedom to express himself freely. This is even less structured than the template above, but there are good practices telling us how to maximize the result, like providing the list of actors and the general goal at the very beginning.

The file is plain text, so to avoid any requirement on the tool used to open and modify the document and doesn’t add any formatting to avoid reader to be distracted or writer being dragged into endless make it prettier sessions.

A file represents a use case (feature in the Gherkin language), so you end up having multiple files, each one representing a piece of the software. This greatly simplifies version management, collaboration and merging, enabling multiple writers and revisors working on a single system. It’s not uncommon to use the same version control system in use for the system to store the documentation.

Files can be structured in a hierarchical fashion via folder structure, so big systems with lots of features can organize their specifications.

Files can be annotated via tags so to create parallel organizational structures, with the folder structure still being the prominent one: this enables additional categorizations, useful to track other associations between use cases (features), as much as what the introductory Use Case Diagram was doing in the Word document.

What makes the biggest confusion, though, is Gherkin files can be executed.

That is what Cucumber provides: the support software structures and an execution environment for Gherkin files.

Why? To create a connection between the system documentation and the documented system, so to ensure the documentation is aligned to the system it describes.

How? By mapping each statement in each scenario with some lines of code in the language you like (Ruby? C++? Java? Scala?). If the code ensures somehow the system does what it is expected to do, than the documentation is in sync with the implementation.

Only this last part resembles a test. Only at this stage there is some code involved. And usually testing frameworks and automation libraries are used to do such verification, like JUnit and Selenium.

So, if you need to test your system, please use a testing framework! If, instead, you want to document your system, you are welcome to use Gherkin and Cucumber!

Advertisements

14 thoughts on “Cucumber is NOT a testing framework!

  1. In relation to your last statement regarding wanting to test vs document; how do you feel about using both a framework AND a BDD tool. For example Calabash is a framework much like Selenium is, however it uses Cucumber on top of it for ease of business language translation and documentation (as you mentioned). You can also use Cucumber on top of Selenium and with other frameworks as well. It seemed as you gave a hard black or white opinion so I was wondering how you felt about the gray area you left out of the article. Good read btw, thanks for posting.

    Like

    • Hi Piotr,
      Thanks for passing by and leaving your feedback. My point is not to avoid using Cucumber in combination with any framework, Selenium and JUnit are definitely welcome to be used in implementing the stepdefs as much as any other library (don’t know Calabash specifically) which might help in reducing the code.
      The problem would be if you write the Gherkin/Cucumber feature file in order to execute the tests: that would be definitely wrong.
      Even if well written, a test remains a test and it would be much easier to just write the test rather than blow your mind trying to describe the test in business language.
      Instead, if you write the feature file to describe your system, may be using some examples to better explain the behaviour, than it makes sense to implement the stepdefs by using the right frameworks/libraries.

      What is the difference? Where you start from, obviously. If you start with the idea to test your system, than you’ll end up with tons of feature files, most of which aren’t comprehensible by the business reader who just passes by.
      On the other end, if you start with the idea to document your system, everything you’ll write will have value for the occasional reader. On top of that, nobody prevents you to add a bunch of tests (integration ones more probably) to fill the gap, but they will not have the ‘driving’ Gherkin/Cucumber feature file, because they are not needed.

      My whole point is: if you need to write tests, using Cucumber is going to consume a lot of time with none or very little benefit. By just using the underlying libraries you’ll save a lot of time and effort. As a friend of mine tried to explain to me a few years ago: the real value of Cucumber is not in the tests, but in the communication it sparks when you write the files with the business. Remove the business representative from the equation and the overall result will be negative.

      I did learn this lesson on my own skin, you can trust me or go for your own experience, but don’t tell us we didn’t warn you 😉

      So, while there are grey areas out there, they usually are a little brighter or a little darker… Those that are completely neutral have still to smash into my face.

      Thanks for the comment, it was really welcome and I’m happy you raised your doubts!

      Like

  2. Pingback: Five Blogs – 21 January 2016 | 5blogs

    • Even if you are a troll…
      Yes, Cucumber is a framework/tool (call it as you prefer), it is not a testing framework/tool though. Please note I’ve stressed a word in the previous sentence.

      Like

  3. I listened to talk by Aslak in an LJC event, where Aslak explained that Cucumber is not a testing framework. Aslak explained how Cucumber is used for high quality communication. I used this technique in one of the interview test for a major media company in West London. The interviewer rejected my application stating my strange use of Cucumber.

    Like

    • I would say you have been lucky enough not to be selected for a job which would have led you in the wrong direction.
      Seriously, I’ve been recently involved in a conversation in the Cucumber mailing list where an entire company which advertise as Test and Test Automation expert is using Cucumber in the wrong way!
      So your experience sounds perfectly aligned to this, nonetheless I can’t stop saying they are wrong and be disappointed about such lack of understanding and professionalism: I don’t consider myself a Cucumber expert, but I believe I’ve understood its principles

      Like

  4. Hi there,

    grazie per il vostro articolo, è veramente interessante!

    One comment I do have: Selenium (I assume you are talking about Selenium WebDriver) is NOT a testing framework either. It is a browser automation framework. Selenium itself does not contain any methods for performing checks and assertions, which would classify it as a testing framework. Selenium is generally combined with JUnit or TestNG (for Java) or NUnit (for C#) to create a UI-level testing framework.

    For someone so keen on the proper definition of what is and what isn’t a testing framework, you might want to get this right too 🙂

    Like

    • You are 100% right Bas and you are not the first one pointing me to that. I’m going to update the article accordingly.
      Thanks for stumbling on this blog!

      Like

  5. Pingback: Behavior Driven Development ( BDD ) , Cucumber – VikramVI Knowledge Sharing

  6. Hi rlogiacco,

    Your post was just what I was feeling about cucumber…
    Could you please tell me whether we should use cucumber for API testing.

    I was thinking about creating a framework using Java+ restassured library. And not using cucumber..But some ppls in my team are arguing that cucumber will do better than anything for api testing… pls, shed some light..Thanks

    Thanks,
    Arun

    Like

    • Hi Arun,
      if you intend to only test your API my advice is to forget about Cucumber and go for direct testing, either via JUnit, TestNG or any other mean you believe fits the bill.
      On the other hand, if you also wish to create the API end user documentation in the format of usage examples and expected behavior than Cucumber might be worth the extra effort spent in describing the behavior in a natural language (plain English, I suppose).
      The best results with Cucumber are obtained writing the feature files before the implementation starts along with the business partner (or committer, if you prefer).

      BTW, if anybody asks you to use Cucumber for testing something than you should question the request: Cucumber is not a testing framework and apparently this is still misunderstood.

      Like

  7. Pingback: Data Analytics with Business Value - From TestNG to Cucumber

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s