Key metric as a compass for startup development. Average cost to fix a defect. Efficiency of tests and test suites

The term "metric" has far from the only meaning. Let's touch on them in general, focusing on the most relevant for today - Yandex.Metrica.

What is a metric

The concept has several interpretations in dictionaries:

abbreviated form of the term "metric tensor";
a function that is designed to determine distances in metric space;
Software metric - the value expressed in numbers of any properties of a particular software;
in poetry: the teaching of the ancient Greeks about rhythm, the structure of poetic speech;
in music theory: a special section devoted to the study of the quantity that measures various rhythmic structures - the musical meter;
synonym for birth certificate, metric register, noble genealogy book;
a numerical value that influences the choice of route for information in computer networks.

Also "Metrica" is the name Russian network construction hypermarkets, a book by the mathematician Heron of Alexandria.

"Yandex.Metrica"

Today the word most often appears in the name of one of the Yandex services. IN this key Metrica is a web analytics tool widely used in ecommerce. Specifically, it provides analysis:

site audience;
behavior of visitors on it;
revenue and resource conversion;
traffic sources;
speed of the site, as well as its accessibility;
the effectiveness of a particular advertising campaign.

Yandex.Metrica reports

"Metrica" is a service that generates reports:

General picture of site traffic.
Distribution of visitors by (requests in search engines, link from social networks, advertising).
Demographic characteristics of those visiting the site.
Interaction of visitors with individual elements of the site - clicking on links, downloading content, viewing certain pages.
Reproducing the route of visitor actions, analyzing the completion of certain forms.
Information about the software and devices from which the views were made.
About the results of checking the availability of a resource by the system, about its load.
For online markets: detailed information about all orders placed on the site.

All of the options listed are free. There is only one in Metrica paid feature- "Target call". This is summary statistics of calls from potential consumers who have become aware of the number specific user from a number of external sources.

Metrics are a concept that defines a variety of phenomena of reality. In light of modern realities, many primarily associate the word with the Yandex service of the same name.

Metrics are quickly becoming the most important factor in measuring and managing performance in relation to organizational goals. However, organizations are on a dangerous path if they do not fully understand what metrics are and really say.

Managing productivity and process efficiency is the focus of many companies today. Regardless of what such practices are called - performance management of an enterprise, corporation or business (Business, Corporate or Enterprise Performance Management), - they are all based on metrics that describe the progress of certain processes in the company. Unfortunately, the mass of words used often obscures the understanding of what metrics are, how they should be defined, and how their relevance and usefulness can be established.

Metrics

Metrics are the essence of a measurement system. You can't manage what you can't measure. Anyone who questions the veracity of this statement is moving completely randomly, if at all in this case we can talk about movement. In the context of process performance management, metrics are measurements of process elements relative to some reference point. Metric systems are several metrics combined to reflect functions or processes.

The metrics themselves only describe process parameters. These processes may occur in the areas of research, production, finance, operations, marketing, services, or any other. Obviously, the more detailed the process is defined, the more accurate the measurement results will be. In general, metrics can be defined for all but the most amorphous processes.

Standards for reference points are of two types: internal and external. Internal benchmarks set target values to be achieved in business processes. They represent "as is" conditions, but can also be used to measure progress toward a desired "as should be" state.

External benchmarks allow you to compare a company's internal metrics with those of other companies to understand how the company stacks up against the best practices in the industry. External standards are important for understanding relative labor productivity in a particular industry.

Ideally, businesses establish benchmarks using statistical methods. In some cases (for example, when introducing a completely new process), managers arbitrarily set the initial level - perhaps in collaboration with consultants and experts in the relevant field. But the level thus established will gradually be modified taking into account actual measurements and statistical data.

Performance Management

Metrics should be chosen to provide the greatest amount of information while using the least amount of resources. Selecting metrics that align with business objectives is at the core of the success of a process performance management strategy. When metrics are used across the enterprise, the factors that drive business performance become clear. Without good metrics Performance management initiatives can do more harm than good.

Many companies feel too busy current problems survival to address the "root" quality issues. "We don't have time for theoretical discussions," they say. These companies refuse to understand that if they don't make an effort to regain control, there will soon be no business left to save. But the introduction of Six Sigma approaches and other quality initiatives have enlightened many companies and allowed them to regain control of their business. The basis of their success were metrics that ensure transparency of business processes. The metrics made it possible to detect, identify and eliminate the root causes: poor quality, bottlenecks, duplicative or conflicting processes, restrictions and other factors. General Electric is example of this, how extraordinary the results can be: the company received a nearly 300% return on investment thanks to the Six Sigma initiative, which showed the importance of metrics for a corporation. Pressures from competition, financial efficiency and increasing regulation are forcing companies to squeeze everything out of their processes.

Compliance and correctness

It should be axiomatic that all metrics come from business goals. For each goal, a company must have a metric to measure its performance in relation to that goal. Hence the first required characteristic metrics, compliance, which shows whether the metric supports business goals. If not, then it is useless and should not be used. An enterprise metrics development project will not succeed if it is a bottom-up approach. A top-down approach is required, deciding what metrics are needed to support the corporate business plan. An enterprise should have a set of well-defined metrics requirements associated with the corporate business plan, from the C-suite level down.

There is a second necessary characteristic - the correctness and validity of the metric measurement process itself. A metric may be appropriate from the point of view of the original task of its creation, but in reality turn out to be incorrect. It may simply be measured incorrectly and not show in reality what is expected of it. This poses the same risk to business as faulty instrumentation on an airplane. Faulty indicators sometimes prove fatal to a company. The correctness of measurement is by no means a trivial issue, especially in the case of metric systems. Therefore, experts have developed a special process for confirming the correctness of metrics. It consists of five points:

1. Correlate the metric with the goals that need to be achieved. Is the connection valid? Does it make sense? What other metrics are relevant to this goal?

2. Evaluate each component of the composite metric (functions, weights, measurements) from the point of view of correctness and acceptability. If you decompose the composite metrics into their components, you will see which of them have a strong impact on this metric. It is very useful to check that all components are correct possible scenarios. If weights are used, you need to check whether they need to be adjusted over time. If so, when were they last adjusted, and what process was used to obtain and validate the weights? Is this process documented?

3. Check the quality and applicability of all input data for the metric. Just like analyzing a metric, you need to evaluate the input data used component by component. “What goes around comes around” - this proverb is more relevant here than ever. Of course, this consideration should also lead to an analysis of the metrics that are located earlier in the process and generate data.

4. Check the applicability of the metric result for other metrics. The applicability of the result of a metric earlier in the process affects the current metric. This means that, having proven the correctness of the metric at this point in the process, you will have to check whether this metric is applicable to metrics further down the process.

5. Perform metric sensitivity tests. Does the input data have a linear or exponential effect on the metric value? Does the data influence the weights assigned to the metric components? Are the results repeatable?

At first, some of these steps may seem repetitive. To some extent this is true. The important thing to understand here is that the quality of the metric is not limited to one step, it is important throughout the entire process. Analysis of a single metric must be done in the context of the metric chain for that process.

Metrics life cycle

Separately, it is worth mentioning the changes in metrics. It would seem that once a metric has been developed and tested, why does it need to be continually validated? But what is valid today may become irrelevant tomorrow. Businesses adapt, and adaptation affects the assumptions on which the business operates. A change in one area will have an effect throughout the enterprise. Metrics used in areas where changes have occurred will influence subordinate metrics used elsewhere. Impacts can be felt either indirectly—for example, through a metric for downstream processes—or directly, with the results of the metric serving as a key component in the metrics system.

A good strategy for using metrics is to use the concept of metrics management life-cycle (MML). The MML defines procedures and controls for design review, development, monitoring, adjustment/modification, and finally reversal of each metric. This way you can maintain metric consistency and accuracy, ensure that indicators are valid and accurate, and that actions taken based on metrics are correct.

Metric selection

Having summed up the experience of several projects, we can give some universal recommendations for choosing metrics within an organization.

1. Metric cost. The cost of a metrics selection project is often ignored. The cost of each metric is the sum of the costs of continually collecting, compiling, storing, discussing, analyzing, managing, and maintaining the collection of metrics and its repository. The expected cost of each metric not only influences the development of the initial metric selection strategy, but also remains important factor throughout the entire life cycle of the metrics.

Gaining benefits through the use of metrics is associated with time and cost savings, as well as improvements in quality and efficiency. Ideally, before starting a metric selection program, benchmark standards should be established so that return on investment (or some other acceptable unit) can quantify the profit achieved. If there are no such reference point standards, the estimates may not be acceptable.

2. Tolerances in metrics. It is clear that there are tolerances in the measurement of any metric. If defined too loosely, tolerances can become a source of problems. On the other hand, if tolerances are defined too strictly, this can lead to excessive and unnecessary warnings or even process stops.

3. Behavioral factors . The purpose of metrics is to help answer questions like “who, what, where, when (how often), why and how.” Therefore, when defining a measurement system, specificity and unambiguity are important to avoid manipulating metrics to distort results. The more specific you are about what is a valid and reliable metric, the less flexibility there will be in interpreting the results. In addition, experts offer a number of tips on how to avoid behavioral problems.

The first tip is to use multiple metrics (two or more, complementary in nature) in relation to a single process. This will ensure control and balance of metrics. An example is the triple constraint of cost, time and quality in project management: you cannot influence one of them without simultaneously affecting the other two.

The second is to apply metrics to the process so that they reflect how changes in downstream processes affect the process. this process. For example: "Process A cannot be reversed to avoid costs without affecting negative influence on the availability of material in process B downstream. This will eliminate process B."

Metrics portfolio

A metrics portfolio is a repository, an archive of data in which all metrics with their attributes used by the enterprise are uniquely identified.

Typically, a metrics portfolio should include the following attributes:

metric name;
the metric context, which identifies the group to which the metric belongs, such as a department, department, or group/work center (note that some metrics can be used in multiple contexts);
description of the functionality of the metric;
input information, processes and data that go directly into the metric (including those taken from metrics earlier in the process);
formulas describing the internal processing process carried out by a given metric;
output data, information obtained as a result using metrics, as well as direct downward dependencies of the metric.

Experience shows that a portfolio of metrics is very convenient way working with metrics. When new metrics are proposed, you can review the portfolio to see if the metric exists in the same or similar form. If it exists in an identical form, then it is clear that the proposed metric is just a different application of the old one or a different context. Again, it is necessary to ensure that the metrics maintain their independence. If the proposed metric is similar to an existing one, then you should consider combining the two options into one metric that can later be used in different contexts. In this way, we will, if possible, avoid the appearance of duplicates and similar variants. Businesses are moving towards real-time process management and reporting. As this happens, proactively working with a portfolio of metrics will become increasingly important for standardization, coordination and governance. correct use metrics in the enterprise.

Metrics Project

Implementing a program that aims to create a portfolio of metrics can present a significant challenge. People resist change, it's human nature. The average metrics creation program consists of the following steps.

1. Harmonization of definitions. Without this step, it is extremely difficult to communicate between different contexts used in an enterprise. The word "definition" is used here in terms of business processes and their attributes: metric, dimension, object (organizational unit, unit of data), activity, task, etc. To "agree" on the definition, it is necessary to identify the context in which the word is used . Then identify antonyms and synonyms and make these definitions consistent within the context. Finally, begin to reflect into other contexts. Thus, what is consistent are the definitions given to words or standardized groups of words that make them compatible or translatable from one context to another. big number other contexts.

2. Development of the structure and attributes of a portfolio or archive of metrics. This is a source of documentation and definitions of metrics. The archive allows you to save the measurement history in the database, which may be important later. Storing this information in a central archive that is accessible to everyone helps ensure consistency and reliability of metrics.

3. Business process modeling. This should be a separate project, with little attention paid to metrics. The resulting model will be a critical component of the infrastructure needed to identify and collect relevant metrics. Whether the IDEF1x methodology, the Business Process Management Initiative (BPMI), or something else is used, the modeling process should be standard and formal throughout the organization. Otherwise, the company will face a costly, arduous process of mapping one methodology to another, and most likely a failure in its metrics creation efforts.

4. Identification of favorable application opportunities. Once you've completed the first three steps, you'll have many options for collecting and applying metrics. During the identification process, one should not lose sight of both the costs and benefits associated with the use of metrics. There will be many opportunities that will need to be assessed and prioritized according to criteria such as potential savings or ROI.

5. Launch of a pilot project. The pilot project should be small but have a significant impact. This project will begin the formal process of developing and using metrics across the enterprise and demonstrating their benefits. The first step of the pilot project should be to measure the current state of the process characteristic so that the effectiveness of the metrics program can be demonstrated.

6. Development of BI policy. A portfolio of metrics can facilitate data analysis across the enterprise. Part of a metrics development project typically involves identifying the most effective methods, processes and formats for presenting metrics to users throughout the organization. This is why the development of metrics usually initiates subsequent BI projects.

CIO and enterprise metrics

It is clear that obtaining metrics is a direct function of information technology. CIOs are well aware of the pressures being placed on their businesses. They understand that if IT can't meet the need to create metrics or is too slow to adopt new technologies, the CIO will be in trouble. At the same time, the creation of metrics has an impact on the IT infrastructure of the enterprise. There are two main areas: time and resources; qualification.

1. Time and resources. First of all we're talking about about the time costs associated with creating infrastructure and creating a portfolio of metrics. IT resources play an important role in solving problems related to metrics. You will need funds, as well as personnel, to collect, consolidate, and present metrics information for your organization. In addition, as soon as the metrics begin to be calculated, there will be a stream of queries to the enterprise databases. As requests accumulate, patterns emerge that indicate similarities between the types of requests and the groups supplying them. CIOs will need to orchestrate review of these requests across the enterprise. One result is the development of reusable components for multiple metrics with overlapping functions.

2. Qualification. Although IT professionals must have a good understanding of a company's business, their primary interest lies in the technology field. They have a general understanding of the business and its ecosystem, but typically their knowledge of finance, operations, and other specific areas of the business is not deep enough. However, experience shows that an enterprise metrics project leads to improved business proficiency for the CIO. Companies that have implemented Six Sigma principles in their organizations have consistently included IT managers on project teams. And this is quite justified, because working with metrics requires an even closer connection between IT and business. This is important to achieve not only the rather general goal of making it easier for IT professionals to understand business objectives and strategies. A more specific goal is also being pursued: a closer connection to business objectives will help IT specialists acquire a thorough knowledge of KPI metrics (key performance indicators).

By developing a portfolio of metrics for your organization, you are taking steps whose ultimate goal - visibility into the progress of enterprise processes - is not easily achieved. And at first, the vast majority of such projects encounter problems due to misunderstanding of personnel, explicit and implicit resistance, etc. But once enterprise users see the benefits of process visibility made possible by a portfolio of metrics, interest in this project will increase enormously. And if at this moment the appropriate training of personnel is carried out, then the project will have a very high chance of success.

You can't manage what you can't measure. This phrase is found in many guides on IT management, and indeed on any management. It is impossible to manage production without measuring instruments allowing you to make the right decisions in a timely manner and respond to a changing situation. The IT department is the same production, comparable in complexity to an average plant, and it requires appropriate tools to manage it. Process metrics are management tools. How effectively a manager can manage an organization depends on how the system of metrics is built.

Difference between metrics and indicators

Before moving further, let's agree on terms. A metric is a measurable parameter. An indicator is a measurable parameter for achieving a certain goal. A target value and a desired trend must be defined for the indicator.

Classification of metrics

The activities of any IT organization can be divided into three segments

Services
Processes
Infrastructure

Each of these segments should be managed and therefore measured.

Service metrics

Shows how our services are provided. These metrics correspond to the service parameters agreed upon in the SLA. It is the change in these metrics that the customer first feels. They are formulated in terms understandable to the customer and must correlate with the subjective perception of the customer. Examples of such metrics: time of report generation, number of clients served per unit of time, etc. It is for the values of service metrics that the IT organization is responsible to the customer. It is obvious that the meaning of service metrics depends on both the operation of processes and the infrastructure. Big time downtime (service metric) can be caused by both excessive load on the communication channel (technological metric) and insufficient speed incident resolution (process metric).

Technology metrics

Technology metrics reflect the health of the infrastructure. These include the current load of communication channels, free disk space, number of failures in the disk array, etc. Control of these metrics is most often assigned to monitoring and event management systems.

Process metrics and their classification

Process metrics show the efficiency of the internal processes of an IT organization. Any process has an input and an output, in addition, the process uses resources and is subject to control influences.

When building a system of metrics, you need to remember these four components and measure each of them. Input metrics - measure the load on the process. For example, for an incident management process, the number of incidents is an input metric. It is important that for a process manager, input metrics are a purely informational indicator; they cannot be influenced, but only reacted. Output metrics, or performance metrics, show how much the process achieves its goal. Resource metrics show the load and sufficiency of resources used by the process. Metrics controls show how controllable the process is, and how effective control actions are. CobiT proposes its own classification of metrics: performance indicators, controllability indicators. In addition, a maturity model is proposed for each process. The decomposition of metrics into four components is comparable to the classification of metrics proposed in CobiT. Performance indicators are output metrics, rationality indicators are resource metrics, process maturity is a controllability metric. It is obvious that when planning target values for various metrics, they should be balanced among themselves. Because the process can be very effective and completely irrational and vice versa. For example, we can resolve all incidents in half an hour, with the help of a thousand people, or we can cope with a dozen people, but resolve incidents in a month.

Reporting corrective actions

Indicators are not interesting in themselves; they are needed to implement management influences on the process. Therefore, responsibility for achieving target indicators should be assigned to the management of the process and to employees assigned to roles in the process. A reporting system for indicators should be built. It is important that appropriate indicators are presented at each level of management. It is unlikely that the IT director will be interested in analyzing the failure statistics of one of the disk arrays. The reporting system should be structured in such a way that at each level, each manager controls and is responsible for 3-9 indicators. Large quantity difficult to keep under constant control.

Motivation issues

Process performance indicators are often used to set personal goals and motivate employees. At first glance, this is logical and correct solution. However, a person whose salary depends on meeting target values will be too tempted to “tweak” the indicators in his favor. Such an “edit” leads to a violation of the main principle of using indicators - objectivity and reliability, as if a doctor made a diagnosis and prescribed treatment based on incorrect information about the patient’s temperature. How to avoid this? It is possible to create a developed internal audit service that would guarantee the relevance of indicators. But, often, the costs of maintaining such a service are unjustified. There is no clear recipe here. We have to look for a reasonable balance between trust and control.

Building a system of IT indicators

Processes do not operate in a vacuum. Each process is aimed at achieving a specific goal, is connected to other processes and contributes to the provision of services. The goal of each process should be set based on what aspect of the service it supports. Thus, the primary goals of the services are the parameters recorded in the SLA. The objectives of the processes and the indicators that measure them must be determined by decomposing these parameters. For example, the SLA may specify the target availability of a service. It is obvious that service availability is influenced by such indicators as time to resolve incidents, number of incidents, success of changes, etc. The target values of these indicators should be determined from the target value of service availability. It is also clear that the significance of each of these indicators is not the same and may vary depending on the conditions. Based on the significance of each indicator, it should be assigned an appropriate weight. By decomposing it into several stages, it is possible to build a system of goals and indicators of the organization. Such a system provides the organization’s management with a tool for operational control and quick diagnostics, the basis for decision-making at both the operational and tactical and even strategic levels.

On what basis should we build a system of indicators?

What should you be guided by when building a measurement system for IT processes? First of all, of course, CobiT. This methodology proposes a process model consisting of thirty-four processes. It is believed that any activity within an IT organization can be classified into one of these processes. For each process, a set of performance and efficiency indicators is provided. The problem is superficial and general description metrics When using CobiT recommendations, methods for measuring target values and algorithms for calculating indicators will have to be thought out independently. As for ITIL®, it is more specific. You can find enough in it detailed list indicators for each process with ways to measure them and desired trends. However, ITIL®, as you know, does not describe all possible IT processes. For example, for indicators of the Software Development process, you will have to turn to other sources. In the topic of process measurement, it is impossible not to touch upon the Balanced Scorecard (BSC). This tool, originally developed for enterprise management, can be successfully applied to an IT management system. The main postulate of BSC is that when building a system of organizational goals, it is dangerous to allow a bias in one direction or another. For example, paying all attention to financial efficiency, without paying attention to customer loyalty or the organization of internal processes. The same is true for IT. For example, skewed to the side technological aspect infrastructure can lead to disruption of internal processes or deterioration of relationships with the customer. The bias towards the service aspect is also harmful. Thus, when building a system of IT goals and measurements, it is necessary to highlight several perspectives that should be given balanced attention. For example, you can offer the following perspectives:

Services
Internal processes
Infrastructure
Finance
Staff.

In each specific case, the weight of goals for these perspectives may change. You can also use a different perspective system.

In this article I want to look at some of the most important QA metrics in my opinion. These will be such indicators, coefficients and indicators that will allow you to capture the overall picture of what is happening on the project in terms of quality and determine steps to improve it. Metrics will concern 5 different areas: requirements, software quality, testing team efficiency, quality of QA work and feedback. It is important to measure and track indicators simultaneously in various sections of the software development process in order to detect common, root problems, and be able to configure and optimize the entire process

Group 1 - Requirements for the software being developed

This group of metrics will allow us to evaluate how well we have worked out the requirements (user story) for the software, identify vulnerabilities and the most complex, potentially problematic software features, and understand where special control is required:

1. Test coverage requirements

In other words, this is the number of tests per requirement.

Metric purpose: identify weak spots in test coverage, highlight risks.

Of course, this metric will only work if the requirements are well decomposed and more or less equivalent. Of course, this is not always possible, but if you can make the requirements sufficiently atomic, then this metric will show the deviation of the coverage of each requirement from the average level. How more value differs from 1, the fewer/more tests are written for one requirement than usual.
The most important thing to pay attention to are the requirements for which the coefficient will be equal to or close to 0. For these, you need to consider adding tests.
If the requirements are not atomic, then this metric will only ensure that there is at least 1 test for each requirement. For this, the coefficient must always be greater than 1.

2. Degree of interconnectedness of requirements

The metric is calculated as the average number of connections of each requirement with other requirements.

Metric purpose: provide a basis for assessing the timing of testing and taking into account possible risks. Knowing the degree of mutual influence of requirements on each other, you can, for example, plan Extra time and cases for end-to-end testing, work on regression checks, look towards integration, etc.

The value of this metric will range from 0 to 1. 1 means that each requirement is related to each one, and 0 means that there is no relationship.
It is difficult to introduce any restrictions on the values of this coefficient; a lot depends on the specifics of the functionality, architecture, and technologies. However, from my own experience I can say that it is good when the degree of connectivity does not exceed 0.2-0.3. Otherwise, modification within the framework of one of the requirements will lead to a chain of changes, and therefore possible errors, in a significant part of the product.

3. Requirements stability factor

Metric purpose: show how many already implemented requirements have to be redone from release to release when developing new features.

Of course, completely isolated functionality does not exist, but the number of new requirements should prevail over the changing ones, and the coefficient should preferably be less than 0.5. In this case, we introduce 2 times more new features than we rework existing ones.
If the coefficient is higher than 0.5, especially if it is greater than 1, then this most likely means that we previously did something that turned out to be unnecessary. The team focuses not on creating new business values, but on reworking previously released features.
The metric also gives an idea of how easily the system’s functionality can be scaled and new features added.

Group 2 - Quality of the product being developed

As the name suggests, this group of metrics demonstrates the quality of the software, as well as the quality of the development itself.

1. Defect density

The percentage of defects per individual module during an iteration or release is calculated.

Metric purpose: highlight which part of the software is the most problematic. This information will help in assessing and planning work with this module, as well as in risk analysis.

Causes large quantity defects for any one specific module (coefficient greater than 0.3) may be different: poor quality requirements, developer qualifications, technical complexity, etc. In any case, this metric will immediately draw our attention to the problem area.

2. Regression coefficient

Metric purpose: show where the team's efforts go: are we doing more creation and debugging new features or most of the time we are forced to patch existing parts of the software

The closer the coefficient is to 0, the fewer errors were introduced into the existing functionality when implementing new requirements. If the value is greater than 0.5, then we spend more than half of the time restoring previously working software functions

3. Reopened defect rate

Metric purpose: assess the quality of development and correction of defects, as well as the complexity of the product or individual module

This metric can be calculated for the entire software, individual module or functionality. The closer the resulting value is to 0, the less old errors are repeated during development.
If the coefficient turns out to be more than 0.2-0.3, this may indicate either the technical complexity of the module and the highly related requirements in it, or a clumsy architecture, or that the previous fix was made poorly.

4. Average cost of fixing a defect

The ratio of the amount of costs incurred by the team when working with all defects (for example, as part of a release) to total number defects.

Metric purpose: show how expensive it is for us to detect and correct each defect. This will make it possible to calculate the benefits of reducing the number of mistakes made and evaluate the feasibility of the appropriate techniques.

Of course, there are no correct values here; everything will be determined by the specifics of a particular situation

5. Number of defects in a specific developer’s code

Metric purpose: highlight possible difficulties in the development team, which specialists lack experience, knowledge or time and need help.

If, for example, 50% of all defects are accounted for by 1 developer, and there are 5 of them in the team, then there is clearly a problem. It does not follow from this that this programmer does not work well, but it signals that you must understand the reasons for this situation.
The metric, among other things, can be an indicator of a particularly difficult module\functional\system to develop and support.

Group 3 – QA Team Capability and Effectiveness

The main purpose of this group of metrics is to express in numbers what the testing team is capable of. These indicators can be calculated and compared on a regular basis, analyze trends, observe how the team’s work is affected by certain changes.

1. Velocity of the QA team

It is calculated as the ratio of implemented story points (or requirements, or user stories) over several, for example, 4-5 iterations (Sprint) to the number of selected iterations.

Metric purpose: numerically express the capabilities and speed of the team’s work for further planning of the scope of work and analysis of development trends

The metric allows you to monitor the speed of QA work and observe what internal processes or external influences on the team can affect this speed.

2. Average defect lifetime

The total time during which defects found within an iteration or release were open to the sum of defects.

Metric purpose: show how much time on average it takes to work with one defect: to register it, correct it and reproduce it. This indicator will allow you to estimate the time required for testing and highlight areas of the software with which the greatest difficulties arise.

Typically, the lifetime of a defect is the entire time from its creation (Created status) to closure (Closed), minus all possible Postponed and Hold. Any bug tracker allows you to calculate and upload this information for an individual sprint or release.
Also, the average defect lifetime can be calculated for various modules and software functions, or, most interestingly, separately for each of the testers and developers from the team. This gives you a chance to identify particularly complex modules or a weak link in the software team.

Group 4 - Quality of work of the testing team

The purpose of this set of metrics is to evaluate how well testers perform their tasks and to determine the level of competencies and maturity of the QA team. Having such a set of indicators, you can compare the team with itself in different moments time or with others, external groups testing.

1. Efficiency of tests and test cases

Metric purpose: show how many errors on average our cases can detect. This metric reflects the quality of the test design and helps to monitor the trend of its change.

It is best to calculate this metric for all test sets: for separate groups functional checks, regression suite, smoke testing, etc.
This indicator of the “lethality” of tests allows you to monitor the effectiveness of each of the kits, how it changes over time and supplement them with “fresh” tests.

2. Rate of errors missed per production

Number of errors discovered after the release \ total number of errors in the software discovered during testing and after release

Metric purpose: demonstrate the quality of testing and the efficiency of error detection - what proportion of defects were filtered out and what proportion went through to production.

The acceptable percentage of errors that were missed in production will, of course, depend on many factors. However, if the coefficient is >0.1, this is bad. This means that every tenth defect was not detected during testing and led to problems in the software already distributed to users.

3. Real work time of the QA team

The ratio of the time spent by the team directly on QA activities to the total number of hours.

Metric purpose: firstly, to increase the accuracy of planning, and secondly, to monitor and manage the performance of a particular team.

Target activities include analysis, design, assessments, testing, work meetings and much more. Possible side effects are downtime due to blockers, communication problems, unavailability of resources, etc.
Naturally, this coefficient will never be equal to 1. Practice shows that for effective teams it can be 0.5-0.6.

4. Accuracy of time estimation by area\types\types of work

Metric purpose: allows the use of a correction factor for subsequent assessments.

The degree of assessment accuracy can be determined for the entire team or individual testers, for the entire system or individual modules BY.

5. Share of unconfirmed (rejected) defects

Metric purpose: show how many defects were created “idlely”.

If the percentage of defects that were rejected exceeds 20%, then the team may experience a desynchronization in understanding what is a defect and what is not.

Group 5 - Feedback and User Satisfaction

And finally, a group of metrics showing how the product was accepted end users how well it met their expectations. But not only software feedback is important: another important task of this group of metrics is to show whether users are satisfied with the process of interaction with the IT team in general and QA in particular.

1. User satisfaction with IT services

Regular survey of user satisfaction with IT services with scoring.

Metric purpose: show whether users trust the IT team, whether they understand how and why its work is organized, and how well this work meets expectations.

The metric can serve as an indicator that it is necessary to focus on optimizing the process or making it clearer and more transparent for users.
The satisfaction indicator can be calculated based on the results of a survey following the release. We collect all the estimates and count GPA. This score can then be recalculated after changes are made to the process.

2. User satisfaction with the product

Regular survey of users about how satisfied they are with the product.

Metric purpose: determine how well the product being developed meets user expectations, whether we are moving in the right direction, whether we correctly determine the importance of features and choose solution options.

To calculate this metric, we also conduct a user survey and calculate the average score. By calculating this indicator on a regular basis (for example, after each release), you can monitor the trend in user satisfaction.

3. Stakeholder engagement

Number of initiatives and proposals to improve the process and product received during the iteration (release) from stakeholders

Metric purpose: determine the degree of participation of external stakeholders in the work on the product. Having such a metric in hand, you can navigate where you need to get feedback so as not to one day encounter contempt and hatred, problems and misunderstanding.

We released new book"Content marketing in in social networks: How to get into your subscribers’ heads and make them fall in love with your brand.”

Not so long ago, we found out who and, and today I propose to consider the main issue regarding website conversion -.

Alternatively, you can set goals to:

viewing the contact page (that is, visiting specific page Online);
sending an application for training (in this case, the goal is usually configuredto submit the form);
communication with an operator via online chat;
purchase or order a service.

And, accordingly, than larger number performing such actions, the higher the conversion, profit and, as a result, the happier the advertiser.

Goals in Yandex.Metrica help you evaluate the effect

Let's say we are launching floristry courses. We make a beautiful website, invest money and run advertising.

Unknown. All we know is the amount of traffic.

Then we set up analytics counters and set up a goal for submitting an online application form. The picture is starting to become clearer. For example, we received the following numbers:

Impressions – 5655.
Clicks – 501.
CTR – 8.86.
The cost of a click is 30 rubles.
Total cost for advertising campaign– 15030 rubles.

During the month of work, 9 applications were received, from which we conclude:

CPA (from English “cost per action” - price per action) amounted to 1670 rubles;
CR (coefficient) is 1.8%.

Let the profit per student be (precisely profit, that is, the price of the course minus the cost) 3,000 rubles. Then, using the formula:

we will get

ROI = (9*3000 – 15030) / 15030 * 100% = 79.64%

Really interesting math?

Undoubtedly.

Why is goal setting so important?

Let's get back to our lambs. That is, goals.

If we considergoals in Yandex.Metrica, then we can divide them into the following types:

number of website pages viewed;
visiting a specific website page that is important for the advertiser;
user execution certain action Online(“event” goal in Yandex.Metrica).

In addition, there is such a thing ascompound goal in Yandex.Metrica (consisting of several steps).For example, for an online store it might look like this:

the user clicked the “Buy” button under one of the products;
the user has entered the cart;
the user filled in his contact information and payment fields;
The user clicked the “Place an order” button.

Or at specific example we have a compound goal:

If we look at each stage, we will see the following picture:

Setting goals is extremely important when we want to understand at what stage, why, and most importantly, how many visitors abandon the purchase.

Maybe they are initially not satisfied with the prices - not only before placing an order, but there is practically no talk about visiting the shopping cart. Or maybe, when entering the cart and seeing the number of fields that need to be filled out, the user decides that he would rather buy somewhere else. Everything is individual.

Again, this knowledge is useful for the notorious remarketing, the very mechanism that allows you to return users who have already been to the site to the site.

However, even after setting up all the goals, we cannot say that we are tracking everything.

Surely, everyone has encountered a situation when they entered a request, clicked on an ad, got to the site, received necessary information, became interested and called the number indicated in the header of the site, where a nice girl helped place an order.

Yes.

Will we find out about this?

Only if the user himself tells us about it to celebrate. But, you see, this is almost fantasy.

That is, there are no applications, there is an effect, but neither the agency nor the advertiser knows about it. And dancing with a tambourine begins.

In this case, the ideal option would be to connect call-tracking – a call tracking mechanism. Now exists great amount services for every taste, color and budget. There would be a desire.