Choosing the right test metrics and product quality

What to measure using test metrics?

Metrics determine business development. It is important for software testing companies to focus on effective indicators in order to track changes and success (or vice versa) of their actions. However, often metrics that employees consider valuable do not correspond to the business and its global goals. As the founder of VPE Forum Melanie Ziegler noted, it is impossible to get income by increasing activity: business develops thanks to results. Software testing is no exception. It affects how a business views the quality assurance function and how it evaluates the results of its work.

What are testing metrics?

Testing metrics are numerical indicators that are used when it is necessary to evaluate the progress, quality, performance and operability of the testing process.

Focusing on this data, specialists can adjust the workflow, increase the efficiency and effectiveness of testing, and make the best decisions to optimize work. For example, “gram” is a metric for measuring weight. The software also has its own “meters”: e.g. “How many errors are there per 1000 lines of code?”

Businesses need measurable results in order to see an improvement in quality and understand how this is related to an increase in the return on investment. As you know, “we can’t control what we can’t measure”. Let’s give an example with HelloFresh, where initially the key metric was the number of subscribers to a box of recipes. The QA team then tracked the impact of poor quality on the number of subscribers. The areas that were affected by this metric became a key area of testing. Thanks to the rethinking of the indicators, the company managed to change its strategy and reduce the number of critical errors in the release.

There are a lot of metrics. For example, they can describe the success of SDLC, product quality, or measure the effectiveness of the entire software testing outsourcing team. The calculation of indicators takes place according to special formulas. By the way, the percentage of completed test cases is calculated as follows: the number of completed test cases is divided by the total number of test cases written, and their quotient is multiplied by 100. It’s like comparing the weekly mileage of a car with the ideal mileage recommended by the manufacturer.

Difficulties with metrics

If there are special formulas, what could be the difficulties? A good testing strategy implies that the team uses the right performance indicators. Testing by its nature seeks to provide more information about the product in order to bring clarity and manage risks.

The problem with metrics is that they determine all further work, and choosing them incorrectly can lead to adverse consequences. For example, if a QA specialist aims to find as many errors as possible, in pursuit of numbers, he/she may neglect critical areas. That is, functions that are not very valuable for business will be tested, and indicators will improve due to this. It will turn out to be a kind of imaginary picture of software quality, because the number of errors was the only criterion by which one could judge someone’s efforts. But the numbers themselves don’t tell the whole story.

Due to the fact that measurement can affect the way of checks, it is important for specialists who provide software testing services to think carefully about metrics. The priority should be indicators that are valuable for business and the improvement of which will bring real benefits to the company. So that in the pursuit of beautiful numbers, the team members do not forget about the most important thing – quality. After all, as economist Charles Goodhart said, “When a measure becomes a target, it ceases to be a good measure”.

It is worth being guided by sound logic, why it is important to measure something. For example, if the execution of an automated test suite takes more than two hours, it is advisable to divide it into two pipelines. First of all, run the script with the most important test cases. And then all that is left (for example, regression tests).

Testing metrics that are important to pay attention to

User satisfaction

The more the customer buys, the more success is guaranteed to the business. Accordingly, the first thing the QA team during the quality assurance audit should start from is the user’s reaction to the product and error messages. If testers measure this quality indicator and strive to improve it, the business will be successful. After all, 92% of customers leave the app if they are not satisfied with its quality.

Satisfied customers are regular customers. They usually return if they liked the service or product they purchased. If quality issues arise, almost 13% of dissatisfied customers share negative experiences with more than 20 people. Which significantly affects the company’s image.

Process metrics

These indicators can have a big impact on the software product quality. For example, you can measure the time from the moment a requirement is created to the moment the code implementing this requirement is deployed in a production environment. Or the time spent on the development of the function from the moment of obtaining permission to start working on it.

Finally, you can measure the time it takes to solve problems. Another option to increase the efficiency of processes is to pay attention to where the work stops. This is a bottleneck for which it is worth finding ways to optimize.

Coverage indicators

Test coverage is a test quality indicator that tells you which part of the code or how many requirements have been tested. Here, experienced QA recommends a top-down approach: start by covering the module and functionality, then analyze the data coverage for each functionality.

According to a review of academic studies that examined the relationship between application quality and code coverage, levels from 70% to 90% assume reliable software. In addition, the test coverage metric helps eliminate ambiguity that occurs in large teams and leads to consistent testing methods.

Code quality metrics

Automatic checks using tools such as SonarQube help to assess the code quality. According to them, it is easy to track how much technical debt is in the system. The accumulated problems and vulnerabilities should be sorted by degree of importance and decide what to pay more attention to. 

To measure the code quality, you need to focus on its key features:

• Reliability (high-quality code works without failures);
• Maintainability (software should be easy to maintain);
• Testability (there are no barriers to software testing);
• Portability (with high-quality code, the program can work in various environments).

Taking into account these features, specialists engaged in software testing services determine five main indicators of code quality: maintainability index, complexity metrics, depth of inheritance, connection of classes and code lines.

Error or incident indicators

Tracking crashes and downtime due to software errors can give an idea of the application quality. This indicator determines the end user’s impression of the product.

When reporting problems, it is worth noting which of them are serious and which are suggestions for improvement. It is important for business to know which aspects of quality are more important than others. For certain business models, productivity may be paramount, for example, for online stores with peak traffic on pre-holidays or sales days.

It is worth paying attention not only to the number of errors, but also to analyze the numbers, taking into account useful categories to improve the testing process. 

Research testing metrics

Research testing involves exploring the capabilities of the program. In simple words, what can and cannot be done with the application. During research checks, the software is tested without any plans. So how to measure it?

There are two types of metrics that were proposed by Michael Bolton and James Bach to evaluate exploratory testing. According to the first one, it is necessary to record in each session the time spent on configuration and execution, and the number of defects found. According to the second metric, it is important to take notes of the time it took to get acquainted with the charter of research testing, comparing it with the time it took to observe the interesting behavior of the product.

The following indicators are also used in quality assurance services:

• research or scheduled testing hours;
• average research session time;
• separation of the research session (installation, testing, possibility, troubleshooting);
• sessions;
• number of defects per session.

Test automation metrics

Metrics help to improve the automated testing process and track its status in the company. Over time, the software product becomes more complex, because the number of lines of code increases, new functions are added, errors are corrected, and so on. As a result, the test coverage may decrease, and the cost of the product and the delivery time of the software, on the contrary, increase.

Indicators can give an overall picture of the automatic testing state. If the work is being done in the right way, the introduction of automation should correct the negative dynamics by increasing the test coverage and speeding up the delivery time.

Performance indicators

It is extremely important for enterprises to know the state of peak traffic, as well as to evaluate the performance of software both in normal mode and at times of heavy load. Knowing the maximum operating capacity of the application, it is easier to prevent its downtime and failures after release.

Load tests consider a wide range of performance-related indicators. When evaluating the efficiency of work, response time, error rate, throughput, and the presence of bottlenecks are often measured.

Indicators of “Happiness”

This metric concerns the effectiveness of the software testing outsourcing team and helps answer the questions: “Will the tasks of a specialist who goes on vacation be completed?”, “Do employees have emotional burnout?” and so on. These metrics are assigned to each team member and help to understand the employee’s workload. Based on this data, it can be reduced or vice versa, you can include additional tasks.

This is very important, because the overall mood of the team directly affects the effectiveness of its work and the product quality. These indicators are not easy to measure, but they are worth it, because the cohesion and well-being of the team help the business to work better.

Conclusion

Rapidly growing competition in the service market has led to a demand for shorter production cycles. As a result, QA specialists are forced to accelerate together with the development team in order to deliver high-quality software on time. In order to perform the work correctly and quickly, it is important to use testing metrics that can provide valuable information about the state of the product. 

However, using the right metrics is not an easy task. For example, a person cannot say that a new car is wonderful until he/she drives it. Judging only by the picture, then such an assessment can hardly be called objective. Only with a reasonable and conscious approach to evaluating important indicators, software testing outsourcing can guarantee the product quality and the business success.

It is important to understand that metrics need to be updated regularly depending on changes in the project. Over time, technologies will change, as will the specialists involved in working on the product. Regardless of what is being tested and verified, the goal remains the same – to provide high-quality software that will bring value to users.