Start with Cucumber—>A utility for automated testing

Cucumber

Introduction:

Cucumber is the Acceptance Test Driven Development called Acceptance Test Driven Planning. In this type of testing we use customer acceptance tests to drive the development of code. Thus it is collaborative effort between the customer and the delivery team. Sometimes they are written by the delivery team and then reviewed/approved by the customer. In either case, they are customer facing, and must be expressed in a language and format that customers can relate to. Cucumber gives us that language and format.

Once the requirement analysis has been done with the customer in an iteration/Release planning meeting then using Cucumber one can start writing all its high level Features and the Scenarios associated with each of them. Then they can be submitted to customer for verification purpose before starting any coding and if any changes are suggested by customer then they can be made into those features and scenarios. So we can consider these as an alternative to initial requirement document that we prepare for submission to client/customer. This will also help developer to know and implement the Acceptance criteria/tests for customer before-hand. Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language and serves as documentation, automated tests and development-aid – all rolled into one format. Cucumber works with Ruby, Java, .NET, Flex or web applications written in any language. It has been translated to over 30 spoken languages.

Cucumber reads plain text descriptions of application features with example scenarios, and uses the scenario steps to automate interaction with the code being developed. Cucumber features have three parts: a title, a brief narrative, and an arbitrary number of scenarios which serve as acceptance criteria.

For example:

Feature: pay bill on-line

– In order to reduce the time I spend paying bills

– As a bank customer with a checking account

– I want to pay my bills on-line

Scenario: pay a bill

Given checking account with $50

And a payee named Acme

And an Acme bill for $37

When I pay the Acme bill

Then I should have $13 remaining in my checking account

And the payment of $37 to Acme should be listed in Recent Payments

Explanation:

In the above example , line 1. is the feature title and line 2. to line 4. is brief narrative.

Line 1. (Feature) to line line 5. (Scenario) is called as documentation part as it does not involve any execution and will be printed on the console as it is. Whereas each statement from line 6. to line 11. is called as the step which is the actual executable part. Each of these steps will have some step-definition associated with it. So we can say that step is equivalent to method-call and step-definition that we will be writing for each of these steps can be termed as method. Details about step-definitions are mentioned in section 4. below along with its different types.

The Scenario: keyword is followed by a string and then a series of steps. Each step begins with any of five keywords: Given, When, Then, And and But. For cucumber feature steps the key words used here are Feature, Scenario, Given, When, Then, And and But. Feature is used to provide identification of the test group when results are reported. Scenario is used in the same fashion. Given, When, Then, And and But are all cucumber methods that take as their argument the string that follows. Given steps represent the state of the world before an event. When steps represent the event. Then steps represent the expected outcomes. And and But steps take on the quality of the previous step .These are the steps that cucumber will report as passing, failing or pending based on the results of the corresponding step matchers in the step_definitions.rb files.

RSpec and Cucumber:

Both RSpec and Cucumber help us specify the behaviour of code with examples that are programmatically tied to that code.

Rspec	Cucumber

Domain specific language used to describe the behaviour of objects	Domain specific language used to describe the behaviour of applications
To focus on more granular behavior	To focus on high level behaviour

Note: Rspec is a part of Cucumber itself. Meaning: Cucumber has 2 concentric circles associated with it among which the outer circle belongs to high level features from which we drop down to inner circle where Rspec comes into picture.

1.Support for Rails: Currently Cucumber supports rails version starting from 2.1.0 to 2.3.3 and is found to be working on it. All below versions are not supported.

2.Installation of tools Cucumber, Webrat and Rspec:

Here since we are mainly focusing on Cucumber we may not use Webrat and Rspec but it’s better to keep them installed in case we need to use them in future for some good reason.

Again there are 2 different ways of installation these tools Rubygems or Rails’ plugin mechanism. Always the preferred one is Rubygems as it is easiest is to install everything as a gem. Also Rails’ plugin mechanism requires some dependencies to be installed as well. Also you need to have installed GIT client on your machine using which these plugins are installed.

Commands using both these mechanisms are as given below:

Using Rubygems

[sudo] gem install rspec rspec-rails cucumber webrat

3. Use in sample Rails application:

3.1 Create your rails project using command: rails YourProjectName

3.2 Under the created project make the desired changes in config/database.yml file. This involves specifying desired adapter,database,password (if any used for connecting)

3.3 Bootstrap Cucumber:

This step involves configuration of Cucumber for its use with Ruby on Rails and Webrat. So we need a Rake task and a couple of files that will do this configuration. Use the command as given below:

ruby script/generate cucumber

Execution of this command will result into the creation of directory features and its sub-directories step_definitions and support and other necessary files created under corresponding sub-directories under your project.

Note:

The type of file in which we will be writing features (and scenarios and its steps) is a “.feature” file. It can be created using the command as given in point 3.4 below. This command will also generate corresponding step_definition file with extension as “.rb” under “/step_definitions” directory. Code inside this file can be removed and can be replaced with our own desired code.

3.4 Start or Create a feature to be tested:

ruby script/generate feature Yourfeaturename

This will generate a simple plain text feature with associated steps. The generated feature will fail unless you have set up a layout in your application. This is because webrat fails to parse HTML that isn’t well formed.(i.e has single <html> root node).

3.5 Run features:

If working on a fresh rails project, first set up the (empty) database. In that case we need to follow steps 3.5.1 and 3.5.2 below else skip these 2 steps.

3.5.1 Create the database using the configuration made in file config/database.yml using command:

rake db:create

3.5.2 Execute the command: rake db:migrate

3.5.3 Finally run the features using the command: rake feature

This should result in failing scenarios, because we haven’t wriiten any code yet.

3.5.4 Other ways of running features:

If you want to run only the features grouped under particular directory (placed inside the features directory) then the following command can be used:

cucumber –require features –require lib features/subdir/

Similarly if you want to run only a particular feature, then:

cucumber –require features –require lib features/yourfeaturefile.feature

Similary we can also run the desired features using the Gem’s ‘cucumber’ command as:

cucumber features/yourfeaturename.feature:7(line number) –format html > features.html

This command will run the scenario defined at line 7 of the specified feature, format it as HTML and pipe it to file features.html for viewing in a browser.

cucumber features –name “Failed login”

This command will run the scenario(s) named “Failed login”

Using TAGS to organize your features and schenarios:

Consider the example as given below:

@billing

Feature: Verify billing

@important

Scenario: Missing product description

Scenario: Several products

A scenario or feature can have as many tags as you like. Just separate them using spaces as shown below:

@billing @verification @xyz

Feature: Verify billing

You can use the –tags option to tell cucumber that you only want to run features or scenarios that have or don’t have certain tags. Examples:

cucumber –tags @billing #Runs both scenarios

cucumber –tags @important #Runs the first scenario

cucumber –tags ~@important #Runs the second scenario (scenarios without @important)

cucumber –tags ~@important,~@other #Won’t run tasks tagged @important or @other

4. Step definitions and its different types:

Step definitions are defined in ruby files under features/step_definitions/*_steps.rb. Here is a simple example:

Given /^I have (\d+) cucumbers in my belly$/ do |cukes|

#some ruby code

end

A step definition is analogous to a method definition / function definition in any kind of OO/procedural programming language. Step definitions can take 0 or more arguments, identified by groups in the Regexp (and an equal number of arguments to the Proc).

Then there are Steps which are declared in your features/*.feature files. Here is an example:

Given I have 93 cucumbers in my belly

A step is analogous to a method or function invocation. In this example, you’re “calling” the step definition above with one argument – the string “93”. Cucumber matches the Step against the Step Definition’s Regexp and takes all of the captures from that match and passes them to the Proc. All Step definitions are loaded (and defined) before Cucumber starts to execute the plain text.

When Cucumber executes the plain text, it will for each step look for a registered Step Definition with a matching Regexp. If it finds one it will execute its Proc, passing all groups from the Regexp match as arguments to the Proc.

Different types of steps and colours associated with them:

Successful steps

When Cucumber finds a matching Step Definition it will execute it. If the block in the step definition doesn’t raise an Exception, the step is marked as successful (green). What you return from a Step Definition has no significance what so ever.

Undefined steps

When Cucumber can’t find a matching Step Definition the step gets marked as yellow, and all subsequent steps in the scenario are skipped. If you use --strict this will cause Cucumber to exit with 1.

Pending steps

When a Step Definition’s Proc invokes the #pending method, the step is marked as yellow (as with undefined ones), reminding you that you have work to do. If you use --strict this will cause Cucumber to exit with 1.

Failed steps

When a Step Definition’s Proc is executed and raises an error, the step is marked as red. What you return from a Step Definition has no significance what so ever. Returning nil or false will not cause a step definition to fail.

Skipped steps

Steps that follow undefined, pending or failed steps are never executed (even if there is a matching Step Definition), and are marked cyan.

Ambiguous steps

Consider these step definitions:

Given /Three (.*) mice/ do |disability|

end

Given /Three blind (.*) / do |animal|

end

And a plain text step:

Given Three blind mice

Cucumber can’t make a decision about what step definition to execute, and will raise a Cucumber::Ambiguous error like “Ambiguous match of “Three blind mice”: telling you to fix the ambiguity.

Guess mode

Imagine you have these step definitions:

Given /we had (.*) / do |what|

end

#A really versatile step definition !

Given /we had (.*) on the (.*) / do |what,where|

end

And a plain text step:

Given we had drinks on the roof

The plain text step will match the Regexp of both step definitions and raise Cucumber::Ambiguous like “Ambiguous match of “we had drinks on the roof”:” . However, if you run Cucumber with --guess, it will guess that you were aiming for the step definition with 2 match groups.

There is ranking logic that gets invoked when the --guess option is turned on:

The longest Regexp with 0 capture groups always wins.
The Regexp with the most capture groups wins (when there are none with 0 groups)
If there are 2+ Regexpn with the same number of capture groups, the one with the shortest overall captured string length wins
If there are still 2+ options then an Ambiguous error is raised

So guess mode can be considered as workaround and it is recommend that we avoid such ambiguous regular expressions.

Redundant Step Definitions

In Cucumber you’re not allowed to use a regexp more than once in a Step Definition (even across files, even with different code inside the Proc), for example the following would cause a “Cucumber::Redundant error:Multiple step definitions have the same Regexp:”

Given /Three (.*) mice/ do |disability|

#some code

end

Given /Three (.*)mice/ do |disability|

#some code

end

5. Autotest Integration:

Cucumber comes with an autotest plugin that will hook into the regular autotest cycle to run the features after the tests/specs. The plugin is disabled by default but can be turned on by setting the AUTOFEATURE environment variable to ‘true’. For example:

$ AUTOFEATURE=true autospec

If you are not using RSpec:

$AUTOFEATURE=true autotest

By default the plugin will run all the features in the features dir. To change the way the features are run with autotest create two profiles in your cucumber.yml:

autotest : Used when Autotest wants to run only the failing scenarios.
autotest-all : Used when Autotest wants to run all the scenarios (after a red→green transition).

Advantage: There is no need to execute the commands again and again to test the correction made for any of the undefined or failed step or Schenario. Simply by saving the file in which changes were made all those changes will automatically tested by this autotest plugin.

Thus it constantly keeps us informed about which all schenarios and steps are pending /undefined/ failing in the feature without executing any command.

For details of Autotest Integration you can refer to link as given below:

http://wiki.github.com/aslakhellesoy/cucumber/autotest-integration

About “Cucumber.yml” file:

Cucumber lets you store and reuse commonly used cucumber command line arguments for a project in a cucumber.yml file. This file must be in your project’s root directory.

For example, by having this in your cucumber.yml at your project root:

default: –format profile features

html_report: –format progress –format html –out=features_report.html features.

You can reuse the arguments listed above as:

cucumber –profile html_report

By simply running ‘cucumber’ the default profile will be used (if a cucumber.yml file is found in the current directory).

5.Scenario Outlines

This is the tool offered by Cucumber for drying things up without sacrificing expressiveness and localization. Cucumber lets us define a single Scenario Outline and then provide tables of input data and expected output. For e.g Consider that we have a project for scrapping data from different sites say: timesofindia.com and ptinews.com . There are some generic testcases which apply to both sites. So

to execute these testcases against both these sites all the WHEN and THEN will be same but the GIVEN will differ (it will be either object for timesofindia site or ptinews site). So here GIVEN will be the dynamic parameter for which we can use tabular format provided by Scenario outlines.

We can make use of this tool when we have same events but with different arguments and different conditions.

Important note:

Here below “Scenario Outlines” we specify multiple scenarios and not single scenario. So please be sure that you specify that label as “Scenarios” else you may get some parse error.

Example:

Given /^an article$/ do |publication|

@article = publication.class.new

end

Feature: Article_agency value in article

Scenario Outline: Article_agency value should be empty when not present

Given an article for <publication>

When It has no Article_agency associated with it

Then It should return Article_agency as ”

|publication|

| Publications::PressTrustOfIndia | –>This is equivalent to–> Given a article “PressTrustofIndia”

| Publications::TimesOfIndia | –>This is equivalent to–> Given a article “TimesofIndia”

For more details on Scenario Outlines refer to page number 45. Of PDF.

6.Formatting output commands

6.1 Html formatter (Important)

Example: > cucumber features — format html > output.html

This will output all the result obtained after running the features into file called “output.html” under root directory. Here instead of output.html if you specify “output.txt” it will output all the html source under this file which may not be of any use.

6.2 TeamCity formatter

TeamCityFormatter: prints cucumber results in a format for interpretation by a TeamCity build agent. For more details and its use with Cucumber please refer to link given below:

http://github.com/darrell/cucumber_teamcity/

6.3 JUnit formatter

Create some output directory say “Junit” in the root directory or any other path you like. Then use the command given below:

Example: > cucumber features — format junit –out Junit

This command will output result of test into some xml file under specified output “Junit” directory. That is it will generate a report similar to Ant+Junit.

6.4 Pretty formatter

Example: > cucumber features — format pretty

This command will print the features as is but in colour’s on console. This is similar to our normal execution command “cucumber features” provided you have gem called win32console installed.

6.5 Profile formatter

Example: > cucumber features — format profile

This command will print the 10 slowest steps at the end on console.

6.6 Progress formatter (Important)

Example: > cucumber features — format progress

This command will print one character per schenario on console. For example if you have some passing scenarios then those will be printed as “green dots” for each passing scenario and if you have some failing scenarios then those will be printed as “red F” for each failing scenario.

6.7 Rerun formatter (Important)

Example: > cucumber features — format rerun

This command will print all the failing files with lines numbers if at all there are any failing schenarios. Here “failing files” means the “.feature” file whose scenario has failed.

6.7 Steps formatter

Example: > cucumber features — format steps

This command prints the location of step_definitions (that is path of step_definition file along with line number for each step definition).

6.8 Tag_cloud formatter (Important)

Example: > cucumber features — format tag_cloud

This command prints the tag_cloud of tag usage. That is number of occurrences of each of the tags used in all features and their scenarios are printed. For example if say “@TOI” is used 9 times through all features and scenarios then it will show value as 9 against tag TOI. This can be useful to cross-check the correctness of count of tests that are being executed for a particular tag.

6.7 Usage formatter

Example: > cucumber features — format usage

This command prints where the step_definitions are being used. That is all the steps and corresponding step_definitions associated with each of them.

7. Drying up the cucumber tests:

Concept behind this is the ability to call in the steps of one scenario from another to avoid duplication of code. Consider the canonical example of ‘log in’ scenario:

Scenario: User logs in

Given there is a User whose username is “matt”

And I follow “log in”

And I enter “matt” in “username”

And I enter the User’s password in “password”

And I press “Log In”

Then I should be logged in

And I should see the text “Hello matt”

Consider another scenario which involves usage of all steps mentioned in scenario above.

Scenario: User views their dashboard

Given I am logged in

And I follow the “dashboard” link

Then I should see “This is your dashboard”

This scenario has step “Given I am logged in” which has all the steps mentioned in first scenario as pre-requisite. So we make directly use those as shown below:

Given /I am logged in/ do

Given “there is a User”

Given “I follow \”log in\””

Given “I enter \”#{User.first.username}\” in \”username\””

Given “I enter \”#{User.first.password}\” in \”password\””

Given “I press \”Log In\””

end

With this we are directly calling steps from first scenario into second scenario without writing the code once again. Note that code will be written corresponding to these steps in first scenario only and it will be reused in second scenario.