Due date: 23:59 p.m Oct 25th, 2021
The State of Quality Report 2021
Wouldn’t it be great to know how peers and experts are staying ahead of their quality assurance game? That’s why we're creating The State of Quality Report 2021 to collect the best QA practices from you, professionals of all levels.
We are happy to offer the first 100 respondents a $30 Amazon gift card, along with the final report from the survey results. Raise your voice!
The demand for delivering quality software faster — or “Quality at Speed” — requires organizations to search for solutions in Agile, continuous integration (CI) and DevOps methodologies. Test automation is an essential part of these aspects. The latest World Quality Report 2018–2019 suggests that test automation is the biggest bottleneck to deliver “Quality at Speed,” as it is an enabler of successful Agile and DevOps adoption.
Test automation cannot be realized without good tools; as they determine how automation is performed and whether the benefits of automation can be delivered. Test automation tools is a crucial component in the DevOps toolchain. The current test automation trends have increased in applying artificial intelligence and machine learning (AI/ML) to offer advanced capabilities for test optimization, intelligent test generation, execution, and reporting. It will be worthwhile to understand which tools are best poised to take advantage of these trends.
Here are the top test automation tools that are believed to best address the challenges in automation over the next few years. The tools included in this list are selected from these criteria:
- Supporting API and services testing
- Offering some AI/ML and analytics capabilities
- Popularity and maturity
1. Selenium
Selenium is the household name when it comes to test automation. It is considered the industry standard for user interface automation testing of Web applications. Almost nine out of ten testers are using or have ever used Selenium in their projects, according to Survey on Test Automation Challenges.
For developers and testers who have experience and skills in programming and scripting, Selenium offers flexibility that is unseen in many other test automation tools and frameworks. Users can write test scripts in many differents languages (such as Java, Groovy, Python, C#, PHP, Ruby, and Perl) that run on multiple system environments (Windows, Mac, Linux) and browsers (Chrome, Firefox, IE, and Headless browsers).
Selenium has recently released its first alpha version of Selenium in April 2019. The announcement of Selenium 4 official version is yet to be determined; but you can expect the release would come with many improved and enriched features.
To use Selenium effectively, users have to possess advanced programming skills and need to spend considerable time to build automation frameworks and libraries necessary for automation. This is a main disadvantage of Selenium, which is addressed in other tools built for codeless test automation like Katalon Studio.
Website: https://selenium.dev/
License: Open-source
2. Katalon Studio
Katalon Studio is a powerful and comprehensive automation solution for testing API, Web, mobile, and desktop application testing. It also has a rich feature set for these types of testing and supports multiple platforms including Windows, macOS, and Linux.
Leveraging Selenium and Appium engines, Katalon Studio provides a uniquely integrated environment for testers who find difficulties in integrating and deploying different frameworks and libraries to use Selenium and Appium, as well as those who are already familiar with these engines.
Katalon Studio was named a March 2019 Gartner Peer Insights Customers’ Choice for Software Test Automation with more than 530 positive reviews, which once again, proves the tool is now one of the biggest players in the market.
Highlights of the tool include:
- A complete feature set for test automation of API/Web services, Web, and mobile applications
- Supports both SOAP and RESTful for API and services testing
- Hundreds of built-in keywords for creating test cases
- Support BDD Cucumber to express test scenario in natural languages
- Can be used for both automated and exploratory testing
- The testing capability can be extended via plugins on Katalon Store In-depth views of reports on Katalon TestOps
In October 2019, Katalon has released its newest update — Katalon Studio 7 — with major enhancements that support scalable projects, solving the Wait issue in Selenium, sharing test artifacts across multiple projects, and more.
Website: https://www.katalon.com/
License: Free — $759 per license per year
3. UFT
UFT is a popular commercial tool to test desktop, Web, and mobile application. It has been extended to include a good set of capabilities for API testing. By supporting multiple platforms for the target application under test (AUT), UFT provides a convenient choice to test the AUT that operates on desktop, Web, and mobile.
UFT provides several advanced capabilities for smart object detection, image-based object detection, and correction. In January 2020, Microfocus has released the latest version of UFT (v15.0) with new features and enhancements that streamline the testing processes, improve testing efficiencies, and sustain quality while reducing testing time.
Several highlights of the tool that support API testing:
- Intuitive user interface for creating, executing, and reporting API tests
- Support generating API tests from WADL documents
- Tests’ actions, activities, and parameters can be visualized in diagrams
Website: https://software.microfocus.com/fr-ca/software/uft
License: From $3,200 per year.
4. TestComplete
TestComplete continues to be in the list this year for its powerful and comprehensive set of features for Web, mobile, and desktop application testing. Testers can use JavaScript, VBScript, Python, or C++Script to write test scripts.
Like UFT, TestComplete has an object recognition engine that can accurately detect dynamic user interface elements. This engine is especially useful in applications that have dynamic and frequently changing user interfaces.
The latest Testcomplete version 14.3 released in November 2019 includes native integration with Jenkins to accelerate the CI/CD pipeline, support for web testing components such as Shadow DOM and custom elements, and support for all recent browser versions and mobile platforms.
Testers can easily use TestComplete’s record and playback feature, like Katalon Studio. They can insert checkpoints into test steps to verify results. As a product of SmartBear, TestComplete can be integrated easily with other products offered by SmartBear.
Website: https://smartbear.com/product/testcomplete/overview/
License: From $9,114 per user per year
5. SoapUI
SoapUI is not a test automation tool for Web or mobile app testing; but it can be a tool of choice to test API and services. It is a headless functional testing tool specifically designed for API testing.
SoapUI supports both REST and SOAP services. API automation testers can use either the open-source or pro version. The pro edition has a user-friendly interface and several advanced features such as assertion wizard, form editor, and SQL query builder. SoapUI is a tool of ReadyAPI suite, offered by SmartBear.
The tool provides a quite comprehensive feature set for API testing with many advanced capabilities, including:
- Generating tests easily using drag and drop, point-and-click
- Powerful data-driven testing with data from files and databases
- Asynchronous testing
- Scripts can be reused easily
- Creating mock services with RESTful mocking
Moreover, SoapUI has recently brought the API Explorer feature from SoapUI Pro into the open-source version, allowing developers and testers to instantly debug their API’s responses. The version 5.5 of SoapUI in February 2019 added the Endpoint Explorer dialog to help users send exploratory requests and analyze responses without creating a project.
Website: https://www.soapui.org/
License: SoapUI is packaged and distributed in two editions, open source and Pro. The latter starts at $659 per year.
Please note that SoapUI and Postman are two representatives of several top API testing tools in this list only. To learn more about the comparison between these tools, please refer to this post.
6. IBM Rational Functional Tester (RFT)
IBM Rational Functional Tester is a test automation tool designed for testing of applications that are developed using different languages and technologies such as Web, .Net, Java, Visual Basic, Siebel, SAP, PowerBuilder, Adobe Flex, and Dojo Toolkit. It is also a data-driven testing platform for functional and regression testing.
RFT provides a feature called ‘storyboard testing’ which helps visualize and edit tests using natural language and application screenshots. RFT’s ScriptAssure capability allows testers to generate test scripts resilient to changes in the user interface of the AUT. RFT can also integrate with other IBM’s application lifecycle management tools, such as IBM Rational Team Concert and Rational Quality Manager.
Website: https://en.wikipedia.org/wiki/Rational_Functional_Tester
License: Commercial — Contact Sales
7. Tricentis Tosca
There are a few continuous testing platforms that provide comprehensive toolsets to support most, if not all, testing activities ranging from test design and test automation to test reports and analytics. Tricentis Tosca is one of them.
This tool has many features such as dashboards, analytics, integrations, and distributed executions to support continuous integration and DevOps practices. Moreover, it offers a friendly user interface and a rich feature set for designing, implementing, executing, managing, optimizing API tests.
Some other highlights of the tool:
- Can be easily integrated to be a crucial part of DevOps processes
- API tests can be performed across browsers, mobile devices, and platforms
- Multiple protocols and standards are enabled, including HTTP(s) JMS, AMQP, Rabbit MQ, TIBCO EMS, SOAP, REST, and IBM MQ
- A good set of test reporting and analytics capabilities
Tricentis claims to significantly reduce regression testing (within the magnitude of minutes from weeks). This claim, however, needs to be carefully verified by testing teams.
In the latest version 13.0 in January 2020, Tricentis Tosca supports API security configuration in the API Connection Manager. It also allows users to use the signature security option to sign multiple parts of a message.
Website: https://www.tricentis.com/
License: Commercial — Contact Sales
8. Ranorex
Having been around for many years, Ranorex provides a comprehensive and professional set of features for Web, mobile, desktop, and API testing. Leveraging its experience in desktop-based test automation, Ranorex has advanced capabilities for UI element identification, editing, and management.
Like Katalon Studio, Ranorex makes automation testing easy for testers with its friendly and intuitive GUI, record/playback, and script generation.
Testers can integrate Ranorex with Selenium Grid to enable distributed testing along with parallel test executions.
The newest release Ranorex Studio 9.2.1 (December,2019) allows users deploying apps to Android 8 and higher via Wifi. The WebDriver endpoints now can support Selenoid. Users can also blacklist DOMs for the CEF plugin.
Website: https://www.ranorex.com/
License: €690 for the Add-on and from €2,290 for Premium License.
9. Postman
Postman is another automation tool designed for API testing. Users can install this tool as a browser extension or a desktop application on Mac, Linux, Windows. It is popular not only among testers for API test automation but also developers who use the tool to develop and test APIs. It is, in fact, a development environment to develop and test APIs.
Some highlights of the tool:
- Comprehensive feature set for designing, debugging, testing, documenting, and publishing APIs
- Friendly and easy-to-use user interface
- Supporting both automated and exploratory testing
- Accepting Swagger and RAML API formats
- Requests and respondents can be packaged and shared with team members
Starting from version 7.2 released in June 2019, Postman extended their support for GraphQL request and schemas, GraphQL query autocompletion function and GraphQL variables.
Website: https://www.getpostman.com/
License: Free — $216 per user per year
10. Apache JMeter
JMeter is an open-source tool designed for test loading and performance measurement — two features of which JMeter is known. However, the tool is now also used for API and services testing, especially for API performance. JMeter is the third most popular tool for test automation, cited by 25% of respondents in the Test Automation Challenges survey.
Highlights of the tool include:
- Lightweight with a simple and easy-to-use user interface
- Test results can be replayed
- Support CSV files to set values for API parameters
- Support integration with CI tools such as Jenkins. JMeter is often used as a part of CI and DevOps toolchains
The most release JMeter 5.2 in November 2019 has been stacked with multiple new features, enhancements, and many bug fixes, such as new protocol, JMESPath extractor, JDBC improvements, StringtoFile, HTTP Samplers.
Website: https://jmeter.apache.org/
License: Open-source
— — — — —
This list of top tools here is not exhaustive; but it represents the best tools that are mature, popular, and providing capabilities using AI/ML to address the challenges that organizations are now facing when it comes to deliver “Quality at Speed.” This list also includes the tools that solely support API and services testing essential for successful Agile and DevOps transformation.
Your choice of testing tools should not only meet your current needs but should also focus on potential trends and improvements. A decent tool should support basic optimization, automation of test case and data generation, smarter solutions, and analytics.
More tools will be introduced as there are opportunities for growth; even though the level of test automation in organizations is low at between 14% and 18% (WQR report). Moreover, it’s going to be exciting to see how the existing and upcoming tools apply AI/ML to address the challenges ahead. API and services testing is also a trend that should see further development in the future.
References:
- Top Software Test Automation Software of 2019 — Gartner Peer Insights
- 100+ Best Software Testing Tools — QASymphony’s research
- Top 10 API Testing Tools (Details & Updates Done for You!)
- What is API Testing?
Top comments (0)