Comparison of Automated Testing Tools: Coded UI Test, Selenium and QTP

Software Testing is a way to validate and verify the working of a particular product or application. It can be incorporated at various points of time in the development process depending upon the methodology and tools used. Testing usually starts after the crystallization of requirements. At a unit level, it starts concurrently with coding; whereas at an integration level, when coding is completed. Testing is used for finding out the bugs in our application. It helps in finding out the failure of the software before it crashes the application. The purpose is to satisfy the stakeholders and ensure the quality of an application with testing.

This article is published from the DotNetCurry .NET Magazine – A Free High Quality Digital Magazine for .NET professionals published once every two months. Subscribe to this eMagazine for Free and get access to hundreds of free tutorials from experts

Manual and Automated Testing

There are mainly two broad types of software testing - Manual Testing and Automated Testing.

Manual Testing

Manual Testing involves testing the software without any automation script or any tool. Testers check the application or software by taking up the role of an end user. They try to find out if there is any unexpected behavior or failure in the application. Test Management can be taken care of by using test plans and test cases.

Automation testing

Automation testing process involves testing with the help of automation scripts and executing the scripts to run the application with the help of some automation tool. Once the script is ready then these tests can run quickly and efficiently.

Since the cost of automated testing is in the form of efforts and time required to create the scripts, not all tests can be converted to automated test. There should be a valid reason to pay that cost.

Reasons for Automation

1. Regression testing to confirm that new changes have not affected the application adversely. It considers already existing test cases for execution. This is an efficient process when we need to provide feedback to the developer immediately.

2. The test cases need to be iterated multiple number of times often with varying datasets to cover multiple workflow paths.

3. When we require support for agile methodologies.

4. Customized reports are required for monitoring.

Getting Started with Automated Testing

Once the need for automated testing has been established, it involves creation of relevant test scripts. Test script creation can be done only by a skilled testers having knowledge of testing, the suite of tools as well functionality under development. Such resources are costly and their time is a premium. Considering this fact, it is often not possible to budget the automation of all tests. Some of the major decision points while identifying cases for testing automation are

1. System modules where requirements do not change frequently

2. Ample time is at hand to describe a test via scripts

3. The application/software module is critical enough to justify the upfront cost of automation

4. After functional testing we want to do performance testing with multiple virtual users using the same test script.

With the scope of automation decided, next step is to pick the testing tool. The following checklist can help with the selection.

1. The tool should be able to easy to work with. It should execute test cases in unattended manner. It should provide interface to write scripts, efficient IDE and ease of test execution.

2. The tool should provide support to various technologies. It should support testing using different browsers, languages, and types of applications.

3. It should integrate with a software that does Application Lifecycle Management so that it can be used for running automated Build Verification Tests as well as the reports can be integrated with other reports created by ALM software.

Automated testing frameworks today

Today we will compare three automated testing tools namely Selenium, QTP (Quality Test Professional) and Coded UI Test (CUIT) with Visual Studio 2012.

We will consider above mentioned main aspects of automation and see how these tools provide the support for each category.

General information about these tools

Selenium was created by Jason Huggins. It is an open source testing tool. Later Simon Stewart started WebDriver (to overcome some limitations of Selenium). Both the tools are now merged to get one awesome testing tool. HP QTP (Quick Test Professional) was originally written by Mercury Interactive. It is a part of HP Quality Centre Suite (QC). Coded UI Test (CUIT) was introduced by Microsoft along with Visual Studio 2010. It integrates with Team Foundation Server.

Ease of Use

- Recording and Playback Functionality 
Each of the testing tools has the ability of recording the actions and playback the recorded actions. Selenium provides the plug-in named Selenium IDE with Mozilla Firefox with which the actions can be recorded. QTP provides record button to record a new test. Recording for CUIT can be done using two different tools. Microsoft Test Manager can be used to record actions that can be then converted to CUIT. The fast forward playback is available to run the test case in a semi-automatic mode even before converting to script. With Visual Studio CUIT provides Coded UI Test Builder to record the actions. In my opinion all the 3 tools are very easy for record and playback.

- IDE and tools with which the tester can write the scripts
With Selenium IDE there is no special tool and specific technology to write the script. We can insert commands with ‘Table or Source View’ when required.

QTP provides Keyword View to display test steps graphically or Expert View which shows VB Script lines.

For CUIT we can easily use Visual Studio IDE to write scripts. 
 

Selenium IDE comes as a plug-in with Mozilla Firefox. With this we can create a test suite which comprise of various test cases. With Selenium IDE is, you can convert recorded Selenium IDE scripts into different languages and after conversion you can run it in Selenium RC. Selenium RC has two components, one is "Selenium Server" and another is "Selenium Client".

With QTP IDE for the first time 3 add-ins are provided ActiveX, Visual Basic and Web. Various links to best practices, new features for the current version are available with start page. We can either open existing test case or create a new one.

For CUIT we have a very elaborative IDE as the recording can be done using Visual Studio. All the features of Visual Studio are applicable. The script writing support with Visual Studio is an excellent way of writing and debugging. In my opinion CUIT scores more points in this area.

- Ease of Test Case execution
With Selenium IDE we have the option of executing the entire test suite already recorded or a test case at a time.

Depending upon the add-ins loaded in QTP IDE the record and run window shows tabs. Windows Application tab is always available. The tests can be executed with run button which in turn opens the run dialog box. We can specify the location for run specific results and provide parameters if any.

CUIT can either be executed with Visual Studio or by using Microsoft Test Manager (MTM). With MTM we can provide various settings for the test case execution so as to gather a lot of information while executing the test case behind the scene. Test Execution can be done is more or less in a very straight manner with the 3 tools. CUIT provides various test settings to execute test cases so as to capture different data when we need to create a bug (commonly called as rich bug).

All the 3 tools can execute test cases without human intervention. 

Platform Support

-Language Support
Selenium uses Selenese, a high-level, cross platform language to write Selenium commands which is a domain specific language. There are 3 basic categories for the commands - named actions, accessors and assertions. To write tests there are a lot of programming languages like C#, Java, Perl, PHP, Python or Ruby.

QTP scripts can be written with VBScript which is a high-level language with support to everything except polymorphism and inheritance.

For CUIT we can write the script with Visual Studio with which we can use all object programming concepts if required. Writing VBScript code is very easy from tester’s perspective. Even though CUIT supports object oriented programming, the testers may not prefer as it involves a lot of skill for writing or customizing the created script.

- Support for various application types
Selenium supports only Web applications.

QTP supports almost any kind of applications.

CUIT supports Windows Applications, Web Application, WPF applications, SharePoint, Office Client Applications Dynamics CRM Web Client applications. 
 

Selenium scores fewer points in this regard as it supports only web application. QTP supports almost all kinds of applications as against CUIT.

- Support for various browsers
Selenium supports all versions of IE, Firefox, Safari and Opera and a few more browsers.

QTP supports IE & Firefox. But both do not provide full cross browser support.

CUIT supports only IE8, IE 9 and IE 10 (IE 10 supported only on desktop). There is no support to IE6, IE7, Chrome, Opera or Safari.

Selenium is the clear winner in this respect

- Support for Data Driven Testing
Selenium IDE supports xml data source using user extensions.

Data Driven testing is implemented as Excel workbook that can be accessed by QTP. There are 2 types of data sheet global and local. Global sheet is a single one which can be accessed from every action in a test. There can even be a local data sheet associated with every action.

Coded UI Test supports any data source supported by .NET framework which can come in the form of a .CSV file, XML file or any other data source like SQL Server table, Access table etc.

In my opinion CUIT provides better ways of data driven testing.

- Exception Handling
Selenium IDE does not support error handling particularly unexpected errors (as it supports only HTML language). Selenium RC will provide support for it (it supports languages with .NET, Java, Perl, Python, PHP, Ruby).

QTP provides VBScript with the help of which we can use On Error statements. As CUIT supports high level languages like C# or VB.Net we can use try catch construct here. In my opinion all the 3 tools have their limitations here.

With CUIT it is suggested to capture the base exception and write code accordingly.

- Validations or AssertionsSelenium assertions can be used in 3 modes assert, verify and waitFor. When an "assert" fails, the test is aborted. When a "verify" (Soft Assertions) fails, the test will continue execution, logging the failure. This facility can be used with TestNg framework. The "waitFor" commands wait for some condition to become true. They will succeed immediately if the condition is already true. However, they will fail and halt the test if the condition does not become true within the current timeout period.

For QTP there are checkpoints: to verify application under test. These are of 10 types – Standard, Table, Image, Bitmap, Database, Text, Text Area, Page, Accessibility, and XML. A checkpoint is a verification point that compares the current value with the expected value. If the current and expected value match it generates a PASS status otherwise FAIL status.
We can use Coded UI Test Builder to add assertions for your UI controls. We need to edit the assertion condition as required (equal to, in between, contains etc.), provide expected value and generate code for it. Mouse hover events can be recorded manually if required. 
 

In my opinion QTP and CUIT are better tools to be used in this aspect.

- Support for Objects
Object properties are not supported by Selenium. Selenium objects can be managed by using UI element user extensions. QTP comes with in-built object repository.

QTP objects have user friendly names.

Coded UI Test code is written with 3 main parts for UI controls. UIMap.designer.cs, UIMap.cs and UIMap.uitest. The first 2 are different physical files for same partial class while the third is a XML equivalent of all the actions recorded with CUIT Builder. Any changes required can be incorporated with the help of the partial class file. We can also edit the UIMap with the help of Coded UI Editor and find out object’s properties. The CUIT can be completely hand coded if required. Coded UI Test includes a rich API library to code against and a resilient record and playback tool. Coded UI Test can be extended to support custom controls.

CUIT is the clear winner in this regard.

Integration with Application Lifecycle Management and going beyond

- ALM Integration
Selenium being an Open Source software can be integrated with other Open Source products for Application Lifecycle Management like QMetry. This in turn can provide platform for software development lifecycle platform in the form of Atlassian Jira (project tracking tool), FogBugz or Bugzilla (bug tracking tool).

QTP being a part of Quality Centre it supports requirement traceability matrix. QTP integrates seamlessly with QC. Test management and mapping the manual testing process with automation becomes a lot easier with this integration

For CUIT and MTM we can provide all the ALM support Team Foundation Server provides. It supports work item tracking, source control or version control, build automation, various reports. The support is in-built; we do not have to do anything extra.

QC is still not complete life cycle management tool. It does not provide support for efforts management, build management or support to different process templates. It supports only test management, bug management and requirement management.

CUIT is a winner here as it seamlessly integrates with Team Foundation Server (TFS). TFS in turn supports work item tracking, source or version control, requirements management, project management, build automation and various reports for monitoring.

· Monitor with Customized Reports
Test results can be made available with each tool. Coded UI Test supports all the reports supported by Team foundation Server as well as has the option of creating any custom report. The custom report can be created in any of the 3 ways, by using Report Project with Business Intelligence Development studio (BIDs), by using Microsoft Excel or by using Report Builder facility to create reports on the fly.

- Going beyond
Selenium being Open Source a lot of plugins available. Selenium IDE has plug-ins for customization, for adding new functionality to API, changing existing functionality.

QTP provided plug-ins for ActiveX controls, web application and VB objects. Other than these plug-ins for other objects like Microsoft .NET, multimedia plug-ins and Windows Mobile are also available. These QTP plugins available at an additional cost.

Apart from hand coding complete CUIT there is another feature available. CUITe Coded UI Test enhanced is a thin layer developed on top of Microsoft’s Coded UI engine which helps reduce the code. It also increases readability and maintainability. It is very easy to install and will be referred with CUITe.dll in the project. CUITe provides simple object repository. 
Each of the tools keeps on adding features as per need. Selenium being open source there are a lot of plug-ins.
 

I have evaluated these tools from different angles and each has got its strength and weaknesses. You may choose a tool depending upon your need and the support the tool provides.

Summarizing the Comparison

The following table provides a bird’s eye view for the categories and the tools’ support for it.

Legend

 

Conclusion

To conclude, we did a quick overview of what is automation testing and when it a good time is to start thinking about test automation in a software development cycle.

We looked at three popular automation tools, Selenium, CUIT and QTP and gauged their strengths and weaknesses. Final selection of tool is almost always based on budgeting and team strengths (tool familiarity), however for Web Application testing all three have compelling strengths. For desktop application testing the choice gets reduced to two with Selenium dropping out.

Editorial Note: Abhishek from Telerik extended this article to add Telerik Test Studio to the mix. Read his article at http://telerikhelper.net/2014/07/10/comparison-of-automated-testing-tools-throwing-in-telerik-test-studio/