Cucumber is NOT a testing framework!

I have been repeating this statement over and over on the Cucumber forum, but apparently with no good result. To me this is a simple statement and I fail to understand what’s difficult to understand in its simplicity: Cucumber (JVM, Ruby, C, the flavour doesn’t matter) is not a testing framework.

I’ll try to further explain the statement, not because I believe it needs to be explained, but hoping an explanation will solve doubts. To do that I’ll try to state the facts in way as much practical as I can.

JUnit, TestNG and DBUnit are examples of a testing framework: they provide facilities to put a software in a certain condition, leverage some parts of the software and assert the outcome responds to the expected parameters.

The goal of those frameworks is to provide the tools to achieve the result (testing the software) with the least possible effort and the maximum outcome. As such, they have programming structures for asserting conditions are met, for setting up the test rig and tear down the test rig as well, ensuring the system is always in a clean condition before a new test is run.

Cucumber doesn’t have any of those facilities, simply because it is not a testing framework.

Cucumber is a specifications tool, something completely different. It uses Gherkin (the language) to describe in a natural language the software specifications.

Does anyone of you remember the Use Case Template? use-case-template-visio

Well a Cucumber feature is much more like one of those than a bunch of tests, would you agree?

There was once a time when we were describing software systems by collecting a few of those use case templates in a Word document, right?

Now, it wasn’t much of a trouble to describe a system like that, at least at first instance, but after a couple of maintenance cycles the above documents were not describing the system any more: I haven’t found one single situation where the above documents were maintained and kept in sync with the implementation. So the system documentation was just obsolete after a few months of operation.

Why was that? Because the cost of maintaining such documentation was not justified by the short term benefit of doing so: why should I spend 30 minutes to find the correct place where to apply the patch in the system documentation if the fix itself costs me 15 minutes?

So, Cucumber steps in and tries to transform the above specifications into something valuable and alive. How?

The specification format (or structure, if you prefer) is almost free, so to leave the writer the freedom to express himself freely. This is even less structured than the template above, but there are good practices telling us how to maximize the result, like providing the list of actors and the general goal at the very beginning.

The file is plain text, so to avoid any requirement on the tool used to open and modify the document and doesn’t add any formatting to avoid reader to be distracted or writer being dragged into endless make it prettier sessions.

A file represents a use case (feature in the Gherkin language), so you end up having multiple files, each one representing a piece of the software. This greatly simplifies version management, collaboration and merging, enabling multiple writers and revisors working on a single system. It’s not uncommon to use the same version control system in use for the system to store the documentation.

Files can be structured in a hierarchical fashion via folder structure, so big systems with lots of features can organize their specifications.

Files can be annotated via tags so to create parallel organizational structures, with the folder structure still being the prominent one: this enables additional categorizations, useful to track other associations between use cases (features), as much as what the introductory Use Case Diagram was doing in the Word document.

What makes the biggest confusion, though, is Gherkin files can be executed.

That is what Cucumber provides: the support software structures and an execution environment for Gherkin files.

Why? To create a connection between the system documentation and the documented system, so to ensure the documentation is aligned to the system it describes.

How? By mapping each statement in each scenario with some lines of code in the language you like (Ruby? C++? Java? Scala?). If the code ensures somehow the system does what it is expected to do, than the documentation is in sync with the implementation.

Only this last part resembles a test. Only at this stage there is some code involved. And usually testing frameworks and automation libraries are used to do such verification, like JUnit and Selenium.

So, if you need to test your system, please use a testing framework! If, instead, you want to document your system, you are welcome to use Gherkin and Cucumber!

Advertisements

18 thoughts on “Cucumber is NOT a testing framework!

  1. In relation to your last statement regarding wanting to test vs document; how do you feel about using both a framework AND a BDD tool. For example Calabash is a framework much like Selenium is, however it uses Cucumber on top of it for ease of business language translation and documentation (as you mentioned). You can also use Cucumber on top of Selenium and with other frameworks as well. It seemed as you gave a hard black or white opinion so I was wondering how you felt about the gray area you left out of the article. Good read btw, thanks for posting.

    Like

    • Hi Piotr,
      Thanks for passing by and leaving your feedback. My point is not to avoid using Cucumber in combination with any framework, Selenium and JUnit are definitely welcome to be used in implementing the stepdefs as much as any other library (don’t know Calabash specifically) which might help in reducing the code.
      The problem would be if you write the Gherkin/Cucumber feature file in order to execute the tests: that would be definitely wrong.
      Even if well written, a test remains a test and it would be much easier to just write the test rather than blow your mind trying to describe the test in business language.
      Instead, if you write the feature file to describe your system, may be using some examples to better explain the behaviour, than it makes sense to implement the stepdefs by using the right frameworks/libraries.

      What is the difference? Where you start from, obviously. If you start with the idea to test your system, than you’ll end up with tons of feature files, most of which aren’t comprehensible by the business reader who just passes by.
      On the other end, if you start with the idea to document your system, everything you’ll write will have value for the occasional reader. On top of that, nobody prevents you to add a bunch of tests (integration ones more probably) to fill the gap, but they will not have the ‘driving’ Gherkin/Cucumber feature file, because they are not needed.

      My whole point is: if you need to write tests, using Cucumber is going to consume a lot of time with none or very little benefit. By just using the underlying libraries you’ll save a lot of time and effort. As a friend of mine tried to explain to me a few years ago: the real value of Cucumber is not in the tests, but in the communication it sparks when you write the files with the business. Remove the business representative from the equation and the overall result will be negative.

      I did learn this lesson on my own skin, you can trust me or go for your own experience, but don’t tell us we didn’t warn you 😉

      So, while there are grey areas out there, they usually are a little brighter or a little darker… Those that are completely neutral have still to smash into my face.

      Thanks for the comment, it was really welcome and I’m happy you raised your doubts!

      Like

  2. Pingback: Five Blogs – 21 January 2016 | 5blogs

    • Even if you are a troll…
      Yes, Cucumber is a framework/tool (call it as you prefer), it is not a testing framework/tool though. Please note I’ve stressed a word in the previous sentence.

      Like

  3. I listened to talk by Aslak in an LJC event, where Aslak explained that Cucumber is not a testing framework. Aslak explained how Cucumber is used for high quality communication. I used this technique in one of the interview test for a major media company in West London. The interviewer rejected my application stating my strange use of Cucumber.

    Like

    • I would say you have been lucky enough not to be selected for a job which would have led you in the wrong direction.
      Seriously, I’ve been recently involved in a conversation in the Cucumber mailing list where an entire company which advertise as Test and Test Automation expert is using Cucumber in the wrong way!
      So your experience sounds perfectly aligned to this, nonetheless I can’t stop saying they are wrong and be disappointed about such lack of understanding and professionalism: I don’t consider myself a Cucumber expert, but I believe I’ve understood its principles

      Like

  4. Hi there,

    grazie per il vostro articolo, è veramente interessante!

    One comment I do have: Selenium (I assume you are talking about Selenium WebDriver) is NOT a testing framework either. It is a browser automation framework. Selenium itself does not contain any methods for performing checks and assertions, which would classify it as a testing framework. Selenium is generally combined with JUnit or TestNG (for Java) or NUnit (for C#) to create a UI-level testing framework.

    For someone so keen on the proper definition of what is and what isn’t a testing framework, you might want to get this right too 🙂

    Like

    • You are 100% right Bas and you are not the first one pointing me to that. I’m going to update the article accordingly.
      Thanks for stumbling on this blog!

      Like

  5. Pingback: Behavior Driven Development ( BDD ) , Cucumber – VikramVI Knowledge Sharing

  6. Hi rlogiacco,

    Your post was just what I was feeling about cucumber…
    Could you please tell me whether we should use cucumber for API testing.

    I was thinking about creating a framework using Java+ restassured library. And not using cucumber..But some ppls in my team are arguing that cucumber will do better than anything for api testing… pls, shed some light..Thanks

    Thanks,
    Arun

    Like

    • Hi Arun,
      if you intend to only test your API my advice is to forget about Cucumber and go for direct testing, either via JUnit, TestNG or any other mean you believe fits the bill.
      On the other hand, if you also wish to create the API end user documentation in the format of usage examples and expected behavior than Cucumber might be worth the extra effort spent in describing the behavior in a natural language (plain English, I suppose).
      The best results with Cucumber are obtained writing the feature files before the implementation starts along with the business partner (or committer, if you prefer).

      BTW, if anybody asks you to use Cucumber for testing something than you should question the request: Cucumber is not a testing framework and apparently this is still misunderstood.

      Like

  7. Pingback: Data Analytics with Business Value - From TestNG to Cucumber

  8. Pingback: TestNG’den salatalıka geçmek – BigDatam

  9. This seems like an attempt to focus on the part of Cucumber that is actually just another bundled dependency (Gherkin) and to write off the actual code that the Cucumber library comprises as an afterthought. (In reality it is where most of the time is spent if you’re actually leveraging Cucumber at all.)

    If you want a specification and you want to collaborate with 3 amigos, just Gherkin is sufficient.

    If you want an executable specification conforming to the Gherkin grammar, then you use Cucumber. So what Cucumber’s value-add is, is emphatically not the Gherkin but indeed the test orchestration.

    The management of registered step definitions and hooks (test setup/teardown more loosely coupled to test cases), the execution of test cases, and the generation of test reports, is precisely what Cucumber provides.

    The fact that it needs to leverage an assertion library, whether (in Ruby) RSpec matchers, or Watir timeouts, etc., to actually provide meaningful runtime test errors, doesn’t make it something other than a test runner. As for setting up and tearing down the test rig, it provides a scaffolding for that with Hooks.
    So I completely disagree that Cucumber is not a testing (checking) framework. Yes, it is also a tool for managing specification by example. But if testing weren’t the point, you would have to explain away an awful lot of code that’s in the tool precisely for that very thing. Almost all of its code, in fact. That’s why you don’t need to use RSpec or JUnit or Spock or Jasmine if you’re using an implementation of Cucumber.

    I understand Matt’s, Steve’s, Aslan’s, et al. push to move the focus onto collaboration when it comes to using Cucumber. But that is largely because that’s how one would justify Cucumber over, say, another test runner for e2e tests. Because that’s a key, differentiating value-add of Cucumber. It provides a coupling between business-readable specification and automated checks while also serving as the runner for those checks. The jury is still out, in my opinion, on whether the coverage that such a coupled spec provides is sufficient given its cost (I don’t think Gherkin is rich enough to provide a full spec in many cases, is overly verbose in many others, and should not be used for edge cases and really anything that belongs in lower-order tests in the first place), but if in enough cases it is, then Cucumber has a valuable place in a business’s toolkit. But I think we have to be honest about what you’re signing up for when you bring Cucumber into the fold.

    Like

    • While I have to admit your entire reasoning about the spec verification framework is indeed spectacularly correct, I also have to disagree regarding something which might appear a lot obvious to you, but, trust me, it is not: That’s why you don’t need to use RSpec or JUnit or Spock or Jasmine if you’re using an implementation of Cucumber.

      That is totally wrong IMHO and you actually want to reduce the number of verifications executed by Cucumber to the bare minimum necessary to assert the specification adhesion: Cucumber verifications are slow (they compare to end-to-end tests) and require a lot of work and preparation when compared to unit tests.

      let’s put it in a different way: you should use Cucumber to verify the software you wrote conforms to the specifications described in the Gherkin files while you use tests to verify your software implementation conforms with your application design (unit tests) and architecture (integration tests).

      You can certainly say Cucumber is another e2e test runner, that is technically correct, but there are much more efficient solutions if what you are looking for is an e2e test runner: Cucumber value resides in its capability to bridge a pseudo natural language (Gherkin) with an e2e test: drop any of the two and you are left with plenty of more valuable alternatives.

      Would you recommend Cucumber to somebody willing to write down an e2e test suite for an already built application? I would, but only if he/she is going to update the Gherkin files whenever a change request comes in… If he/she cannot guarantee that condition I would NOT recommend Cucumber: he/she can save the time spent on writing the Gherkin files and the awful lot of code associated with the tool (btw, my Cucumber related code accounts for probably 5% of the step definitions, with all the remaining code being driving the browser or interacting with some server).

      And with regards to Gherkin, I have used it to describe REST APIs, crypto currency batch processes, workflows and, obviously, web applications. I also used Gherkin with Arduino and small IoT systems, but with no automation behind: just as it was a Microsoft Word document, so I should disagree with it being verbose or unable to provide full specs in many cases: I might agree about edge cases, but that applies to practically everything in this world.

      BTW, what do you mean with should not be used for … really anything that belongs in lower-order tests? What do you consider lower-order tests? Because this seems coming back to the title of my post: it’s not a testing framework.

      May be we can agree on a different statement: Cucumber is not practical as a testing framework per se, its value relies on bridging the Gherkin specifications to some code verification procedures.

      Being honest on what people are signing up with Cucumber was my most prominent goal with this post: my desire is to tell people to either use Cucumber for the value it can bring or stop trying to pull Cucumber toward JUnit/RSpec/Spock/Jasmine. Sadly, by experience, I can tell most of the time those requests come from people working for consultancy companies which do not understand Cucumber or Gherkin but selling their spectacular capability to ride the BDD hype with the only outcome of demonstrating Cucumber is a false promise. I’ve been using Cucumber and Gherkin for the past 6 years and everytime it failed at delivering its value has been when it has been used as a testing framework rather than a specification verification. The difference might sound thin, but IMHO it’s not.

      Thanks anyway for your thoughtful comment, I really appreciated the time and effort.

      Liked by 1 person

  10. I think I was misunderstood when I pointed out that you don’t need RSpec or whatever when you are using Cucumber. I should have worded it differently. I meant what I wrote only in the context of the Cucumber’s capabilities as a framework; in other words, that Cucumber doesn’t need a test runner to run tests because it IS the test runner.

    I definitely did not mean that Cucumber scenarios would be a substitute for unit tests or lower-order (i.e. not e2e) integration tests. Of course I agree with you about Cucumber being the wrong tool for the job if you’re looking for test coverage and you should keep the Cucumber “test suite,” as it were, as lean as possible.

    Regarding my statement about Gherkin being too verbose at times, or instead not rich enough, it’s definitely a case-by-case issue. For some apps it’s good enough, and in fact especially for stuff like APIs which have just about nothing that can’t be expressed in plain text. For others I have struggled, especially when there’s a heavy design component. So I usually ended up with links to comps or wireframes to complement the gherkin.

    Yes, we can agree on your new general statement about Cucumber 🙂

    I, too, have seen Cucumber lead to a lot of dissatisfaction at various clients. And sometimes I was even a part of that project and was unable to force them to course-correct. But I think I have come to a conclusion that Cucumber is not just being misused very frequently; it practically begs to be misused! So much about the framework is designed to build “power” into the step definitions, which puts pressure on the Gherkin to conform in ways that can compromise the spec. The direct coupling between steps and step definitions, something I also see in competitors like Thoughtworks’s Gauge tool, is where a lot of the pain resides. That’s my current thinking on it, anyway, though I’ve been known to change my mind! 🙂

    I appreciate the conversation.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s