Test reliability

This dashboard shows data collected from the continuous integration pipeline artefacts produced by test executions of one or more projects (in the following “project” and “source code repository” are treated as synonymous).

This dashboard is focussed on number of tests failures and errors, their trends and the elapsed time betweeen consecutive failures or errors in a project.

dashboard of reliability

Test failures or errors

The first chart displays the percentage of test failures or errors out of the total of executed tests. You can switch between them by using the filter “Type”, located top-left of the dashboard.

Collected data

To compute this chart Argos uses the same artefacts needed by “Executed Tests”, namely:

  • pytest reports in json and xml.

Failure percentage of executed tests

Usage

You can use this panel:

  • to monitor the trend of failures and errors and spot when there are some spikes;

  • to compare percentages and trends between projects.

Benefits

With this panel you can:

  • identify projects that have an unusually high number of test errors or failures.

  • Identify projects whose trends are not as expected.

  • Identify projects that show high variability of errors and failures; this might indicate projects where quality of testing process is poor.

Mean failure/error free time (mfft,meft)

This chart shows, for each project, the mean time during which the project is free of failures (or errors).

The smaller this number is, the worst is the “health” of the project. For example, if you have two projects: one with mfft of 15 minutes and the other with mfft of 2 days, the latter is in better health than the former, since it has a mean of 2 days where no failure appears.

Collected data

To compute this chart Argos uses the same artefacts needed by “Executed Tests”, namely:

  • pytest reports in json and xml.

Argos computes the mfft (meft) in this way:

  • for each pair of consecutive executions of a project, Argos computes the time difference;

  • if the second execution is failed (or has errors) then the time difference is set to 0;

  • the average of these numbers is reported as mfft (meft).

Mean failure free time

Usage

You can use this panel:

  • to view the amount of time where no failures/error appear on a project;

  • to compare the time failure/error free of different projects.

Benefits

With this panel you can:

  • identify projects where poor quality of testware is apparently not being dealt with.

  • Identify projects where tests never fails (and therefore that might be potentially useless).

Failures/Errors in time period

This chart shows the mean of the percentage of test failures/errors for each project in the selected time period.

Collected data

To compute this chart Argos uses the same artefacts needed by “Executed Tests”, namely:

  • pytest reports in json and xml.

Average percentage of failures in five projects

Usage

You can use this panel:

  • To monitor the failure rate of different projects.

Benefits

With this panel you can:

  • identify projects where the percentage of failures are not as expected.

  • Identify projects that on average have a reliability that is much worse than others.

Unit level

This panel displays the mean of percentage of failures and errors for unit tests among all the projects.

Collected data

To compute this chart Argos uses the same artefacts needed by “Executed Tests”, namely:

  • pytest reports in json and xml.

Percentage of failures and errors per test level

Usage

You can use this panel:

  • to compare failures and errors for each specific test level.

Benefits

With this panel you can:

  • understand which level suffers more by failures and errors.