We are moving toward a future where everything is going to be autonomous, fast, and highly efficient. To match the pace of this fast-moving ecosystem, application delivery times will have to be accelerated, but not at the cost of quality. Achieving quality at speed is imperative and therefore quality assurance gets a lot of attention. To fulfill the demands for exceptional quality and faster time to market, automation testing will assume priority. It is becoming necessary for micro, small, and medium-sized enterprises (SMEs) to automate their testing processes. But the most crucial aspect is to choose the right test automation framework. So let’s understand what a test automation framework is.

What Is a Test Automation Framework?

The term “test automation framework” refers to a structure that provides a readymade test environment to execute the automated test codes. The framework comes with tools and features that can help its user execute the test scripts and develop and analyze the process.

It is a complete system on its own that serves the specific purpose of test automation. It consists of individual guidelines, standard code practices, ideas, procedures, project ranking, reporting frameworks, test data injections, etc. all of which work to support a successful test automation system. Thus, the framework provides its user with an existing template to set the automation process to execute it successfully.

Each test automation framework comes with its specific advantages like easy coding, extensibility, adaptability, ease of understanding, reduced cost and easy maintenance, etc. Hence, it is advisable to employ more than one framework to access multiple benefits from different frameworks relevant to your project.

We decided to test the following frameworks: Nightwatch.js, Cucumber and Cypress. Whereas Nightwatch.js and Cucumber uses the selenium webdriver, Cypress is selenium independent.

Nightwatch.js

Nightwatch.js is an open-source, node.js powered, automated browser testing framework. It was developed in Amsterdam and derives its name from the famous painting ‘The Night Watch’ by Dutch 17th century artist Rembrandt van Rijn.

Nightwatch.js facilitates end-to-end testing of web applications and websites, by utilizing W3C Webdriver API (also known as Selenium Webdriver) as Selenium wrapper for interacting with different browsers. It helps in making the implementation of CI/CD much easier. At the same time, it can be used to perform Selenium test automation for both unit and integration tests. This popular automated browser testing framework provides several commands and assertions to perform interactions with web elements.

It is, at times referred to as a Selenium Binding library only, which is entirely true as it is mostly used together with Selenium Server. But while implementing, the tester must remember that Nightwatch.js in itself, is complete as it contains a built-in CLI test runner, an extendible command and assertion library, and support for page object model.

Pros:

• Clean syntax. Simple but powerful syntax enables you to write tests very quickly.
• Built-in test runner. Built-in command-line test runner can run the tests either sequentially or in parallel, together, by group, tags, or single.
• Nightwatch.js automatically manages the WebDriver services (Edge, Safari, GeckoDriver).
• Cloud services support. Works with cloud testing providers, such as SauceLabs and BrowserStack.
• Includes its own testing framework / assertions library.
• CSS & Xpath support. Either CSS or Xpath selectors can be used to locate and verify elements on the page or execute commands.
• Continuous Integration support. JUnit XML reporting is built-in so you can integrate your tests in your build process with systems such as Teamcity, Jenkins, Hudson etc.
• 3rd party integration with Cucumber. Though Cucumber is not officially supported, Nightwatch.js can be used with Cucumber.

Cons:

• It does not have many choices for unit test frameworks as it has an own testing framework and also support Mocha.
• No official BDD-style syntax support.
• Slightly lesser support.

Test example:

Average execution time: 7.695s

Cypress

Cypress is a next generation front end testing tool built for the modern web.

Cypress is most often compared to Selenium, however Cypress is both fundamentally and architecturally different. Cypress is not constrained by the same restrictions as Selenium. This enables you to write faster, easier and more reliable tests. Cypress users are typically developers or QA engineers building web applications using modern JavaScript frameworks.

Cypress enables you to write all types of tests:

• End-to-end tests
• Integration tests
• Unit tests

Thus Cypress is used to test a wide range of applications that are operational in a browser. Cypress is free and we need not buy a license for its use. It allows us to create our test cases while our application is being developed. So it can be used best in a Test Driven Development framework. Cypress also has the Dashboard service which helps to record our test execution.

Cypress can be used for all sorts of development activities as it gives a platform where debugging is fast and code maintainability is easy.

Pros:

• Cypress does not use Selenium. Most of the end to end tools that we have experimented with, are using Selenium, that’s why they have almost the same problems.
• Cypress supports any framework or website quite well. There are hundreds of projects using the latest React, Angular, Vue, Elm, etc. frameworks. Cypress also works equally well on older server rendered pages or applications.
• Cypress tests are only written in JavaScript. While you can compile down to JavaScript from any other language, ultimately the test code is executed inside the browser itself. There are no languages or driver bindings-there is and will only ever be just JavaScript.
• There are no dependencies, you put your test in package.json and that’s it.
• Cypress runs much, much faster in comparison with the end to end tools by Selenium that we have experimented.
• There is screenshot for every step of your script, which can be quite helpful if there is any false passing or failing test.
• Cypress has a clear syntax, it is easy to read it.
• Dashboard service which provides the visual representation of the test runs, their reports, and status on a single web window. Also, it proves handy in planning and visualizing the test runs of the Cypress Test Scripts.

Cons:

• The structure was different to the other Selenium end to end tools, so at first you may need to spend more time understanding the structure and finding the best way to create your scripts.
• Community. As Cypress is relatively new, the community is small. You will have trouble finding answers to problems etc.
• Page Object Model. It is something that has already been proven by time. Cypress supports a different approach which could be controversial.
• It’s only available for only one client (language) i.e. for JavaScript only. So to work with it you must know JavaScript.
• Cypress does support Xpath selectors. However, that does not come by default. In other words, we need ‘Cypress-Xpath’ external plugins to assist this selector.
• Only a few browsers are supported.

Test example:

Average execution time: 10.612s

Cucumber

Cucumber is an open-source testing framework that supports Behavior Driven Development for automation testing of web applications. The tests are first written in a simple scenario form that describes the expected behavior of the system from the user’s perspective.

Largely used for acceptance tests, Cucumber is written in Ruby, while the tests are written in Gherkin, a non-technical and human-readable language.

Behavior Driven Development gives us an opportunity to create test scripts from both the developer’s and the customer’s perspective as well. So in the beginning, developers, project managers, QAs, user acceptance testers and the product owner (stockholder), all get together and brainstorm about which test scenarios should be passed in order to call this software/application successful. This way they come up with a set of test scenarios. All these test scripts are in simple English language, so it serves the purpose of documentation also.

Pros:

• Reusability decreases necessary steps. There is no need to create a new step or even a new function in the code for the last step. As long as the code behind the step is sufficiently generic, the reusability will be a significant asset.
• It acts as a bridge between the business and technical language. We can accomplish this by creating a test case in plain English text.
• It allows the test script to be written without knowledge of any code, it allows the involvement of non-programmers as well.
• It allows the test script to be written without knowledge of any code, it allows the involvement of non-programmers as well.

Cons:

• Your feature file can easily become a very large and messy file.
• Running tests can take quite some time, especially when you want to run your test in different browsers.
• If you are going to test a JavaScript application by Cucumber and Java, it might be difficult for the development team to adjust itself which is less productive.
• Most people use the BDD „Given“, „When“ and „Then“ syntax in a structure which is not sensible.

Test example:

• Feature:

• Step-definitions:

Average execution time: 9.967s

References:

• ““Top 10 Test Automation Frameworks in 2020”, Suyash Dubey
• “The Top 10 Test Automation Frameworks in 2020”, Debjani Goswami
• “Introduction To Nightwatch.js For Selenium Testing”, Rahul Rana
• “Pros and cons of end to end testing tools”, Farzaneh Farhadi
• https://docs.cypress.io/guides/overview/why-cypress.html#In-a-nutshell
• “Test Automation using Selenium and Cucumber Framework: Tutorial”, Garima Tiwari

•