I was inspired while watching a talk by Kevin Ottens about refactoring OpenGL code in Qt to take a look at gcov & lcov. gcov is used to analyze code coverage – which lines of code have actually been executed while running an application. lcov is a tool that can produce HTML reports from the gcov output.
If you have a suite of unit tests that you run on your code, you can use these tools to see which the lines of code are covered by your tests and which are not.
I couldn’t find a decent primer on how to set this up properly for my Qt projects on macOS so I could run it from Qt Creator, so I thought I’d write up how I did it.
The example I’m using here is available on Codeberg.
Versions
Before I get started, here are the versions of various tools I used:
- Qt – 5.8 dev branch from git
- Qt Creator IDE – 4.2.0
- clang/gcov – Apple LLVM 7.0.2 (clang-700.1.81) (built-in)
- lcov – 1.13 (installed using homebrew)
The Example
First let’s take a look at the simple example we’re going to work with.
1 2 3 4 5 6 7 8 9 10 11 12 |
#ifndef ASMEXAMPLECLASS_H #define ASMEXAMPLECLASS_H #include <QVector> class asmExampleClass { public: int addSomeStuff( const QVector<int> &inVec ) const; }; #endif |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
#include <QDebug> #include "asmExampleClass.h" int asmExampleClass::addSomeStuff( const QVector<int> &inVec ) const { if ( inVec.isEmpty() ) { qInfo() << "I have nothing to add"; return 0; } int sum = 0; for ( int num : inVec ) { sum += num; } if ( sum == 42 ) { qInfo() << "Of course it is"; } return sum; } |
This is just a simple class with one method which takes a QVector of int, adds them, and returns the result. Note that there are a couple of special cases – one for an empty vector and one for a specific sum.
Test Harness
Next, let’s take a look at the testing code.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
#ifndef ASMTESTSUITE_H #define ASMTESTSUITE_H #include <QObject> #include <QVector> #include <QtTest/QtTest> class asmTestSuite : public QObject { Q_OBJECT public: explicit asmTestSuite(); static QVector<QObject*> mSuites; }; #endif |
The asmTestSuite class is used as a convenience. All it does is add the derived class to a static list of tests to run when the derived class is instantiated. Each of our test suites will derive from this class.
1 2 3 4 5 6 7 8 |
#include "asmTestSuite.h" QVector<QObject*> asmTestSuite::mSuites; asmTestSuite::asmTestSuite() { mSuites.push_back( this ); } |
Then in main.cpp, we simply run each test suites, keep track of how many failed, and return that number:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
#include <QtTest/QtTest> #include "asmTestSuite.h" int main( int argc, char** argv ) { QCoreApplication app( argc, argv ); int failedSuitesCount = 0; QVector<QObject*>::iterator iter; for ( iter = asmTestSuite::mSuites.begin(); iter != asmTestSuite::mSuites.end(); ++iter ) { int result = QTest::qExec( *iter ); if ( result != 0 ) { failedSuitesCount++; } } return failedSuitesCount; } |
Unit Tests
Finally we have a test suite for our asmExampleClass.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
#include <QtTest/QtTest> #include "asmTestSuite.h" #include "asmExampleClass.h" class TestExampleClass : public asmTestSuite { Q_OBJECT private slots: void test_addSomeStuff(); }; // test adding list of numbers void TestExampleClass::test_addSomeStuff() { asmExampleClass example; int result = example.addSomeStuff( { 1, 32, 44, 51 } ); QCOMPARE( result, 128 ); } static TestExampleClass sInstance; #include "asmExampleClass_test.moc" |
For each test we want to run, we would add a new method named test_something. The Qt test library will automatically call all slots which are named like this. For details about writing tests for Qt, please see the Qt Test Overview.
Running The Tests
Now that we have a complete system for setting up and running unit tests, let’s take a look at the output when we run it.
1 2 3 4 5 6 7 |
********* Start testing of TestExampleClass ********* Config: Using QtTest library 5.8.0, Qt 5.8.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 7.0.2 (clang-700.1.81) (Apple)) PASS : TestExampleClass::initTestCase() PASS : TestExampleClass::test_addSomeStuff() PASS : TestExampleClass::cleanupTestCase() Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 0ms ********* Finished testing of TestExampleClass ********* |
Great! Our test passed. Ship it!
What does it look like if it fails? Here is what happens if I change the QCOMPARE to check for 42 instead of 128:
1 2 3 4 5 6 7 8 9 10 |
********* Start testing of TestExampleClass ********* Config: Using QtTest library 5.8.0, Qt 5.8.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 7.0.2 (clang-700.1.81) (Apple)) PASS : TestExampleClass::initTestCase() FAIL! : TestExampleClass::test_addSomeStuff() Compared values are not the same Actual (result): 128 Expected (42) : 42 Loc: [../CodeCoverageExample/test/asmExampleClass_test.cpp(23)] PASS : TestExampleClass::cleanupTestCase() Totals: 2 passed, 1 failed, 0 skipped, 0 blacklisted, 1ms ********* Finished testing of TestExampleClass ********* |
It gives us some useful information about which test failed, what the actual vs. expected results were, and where to find the test. This makes it easy to track down what’s going wrong.
Code Coverage
Now that we can run the unit tests, let’s look at a way to improve our coverage.
If we look at the asmExampleClass::addSomeStuff() method that’s being tested in this example, we can see that our unit tests don’t cover all paths – for example if you pass in an empty QVector. This is obvious in our simple example, but if you have a large, complex code base this can be difficult to spot. That’s where the code coverage tools come into play. It will help us by identifying which lines are not being executed in the tests.
Code Coverage Setup
Set up is relatively straightforward (once you know how!). It involves three things:
- changing our compile & link flags
- adding a script to run our code coverage tools
- changing Qt Creator’s run command to run the script
Compiler & Linker flags
With clang, the only flag we need to add is --coverage. The key is that it needs to be added for both the compiler and the linker. If you fail to add it as a linker flag you will run into link errors that look something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
Undefined symbols for architecture x86_64: "_llvm_gcda_emit_arcs", referenced from: ___llvm_gcov_writeout in asmExampleClass.o ___llvm_gcov_writeout in main.o ___llvm_gcov_writeout in asmTestSuite.o ___llvm_gcov_writeout in asmExampleClass_test.o ___llvm_gcov_writeout in moc_asmTestSuite.o "_llvm_gcda_emit_function", referenced from: ___llvm_gcov_writeout in asmExampleClass.o ___llvm_gcov_writeout in main.o ___llvm_gcov_writeout in asmTestSuite.o ___llvm_gcov_writeout in asmExampleClass_test.o ___llvm_gcov_writeout in moc_asmTestSuite.o "_llvm_gcda_end_file", referenced from: ___llvm_gcov_writeout in asmExampleClass.o ___llvm_gcov_writeout in main.o ___llvm_gcov_writeout in asmTestSuite.o ___llvm_gcov_writeout in asmExampleClass_test.o ___llvm_gcov_writeout in moc_asmTestSuite.o "_llvm_gcda_start_file", referenced from: ___llvm_gcov_writeout in asmExampleClass.o ___llvm_gcov_writeout in main.o ___llvm_gcov_writeout in asmTestSuite.o ___llvm_gcov_writeout in asmExampleClass_test.o ___llvm_gcov_writeout in moc_asmTestSuite.o "_llvm_gcda_summary_info", referenced from: ___llvm_gcov_writeout in asmExampleClass.o ___llvm_gcov_writeout in main.o ___llvm_gcov_writeout in asmTestSuite.o ___llvm_gcov_writeout in asmExampleClass_test.o ___llvm_gcov_writeout in moc_asmTestSuite.o "_llvm_gcov_init", referenced from: ___llvm_gcov_init in asmExampleClass.o ___llvm_gcov_init in main.o ___llvm_gcov_init in asmTestSuite.o ___llvm_gcov_init in asmExampleClass_test.o ___llvm_gcov_init in moc_asmTestSuite.o |
We add the coverage flag in our qmake .pro file like this:
1 2 3 4 |
mac { QMAKE_CXXFLAGS += --coverage QMAKE_LFLAGS += --coverage } |
Coverage Shell Script
I wrote a little shell script to process the coverage information and open the results in a browser.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
#!/bin/sh # ${1} is the directory containing the .gcno files (%{buildDir} in Qt Creator) LCOV=lcov GENHTML=genhtml BROWSER="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" SRC_DIR="${1}" HTML_RESULTS="${1}/html" mkdir -p ${HTML_RESULTS} # generate our initial info "${LCOV}" -d "${SRC_DIR}" -c -o "${SRC_DIR}/coverage.info" # remove some paths "${LCOV}" -r "${SRC_DIR}/coverage.info" "*Qt*.framework*" "*Xcode.app*" "*.moc" "*moc_*.cpp" "*/test/*" -o "${SRC_DIR}/coverage-filtered.info" # generate our HTML "${GENHTML}" -o "${HTML_RESULTS}" "${SRC_DIR}/coverage-filtered.info" # reset our counts "${LCOV}" -d "${SRC_DIR}" -z # open in browser and bring to front "${BROWSER}" "${HTML_RESULTS}/index.html" open -a "${BROWSER}" |
There are a couple of things to point out in this script that you might need to change for your own project. The first is the path to the commands (lines 5-6) may need to be added if they are not in your PATH environment variable. You might also want to change browsers – I have it set up for Chrome.
The other thing that you might want to change is the list of files which are ignored. Line 18 has a list of patterns of files to ignore – I’ve excluded the Qt frameworks, some Xcode stuff, the test harness code, and all the moc files.
Qt Creator Run Command
The final piece of the puzzle is to execute the coverage shell script whenever we run the test in Qt Creator. For this we need to modify the run command in our project settings.
Using the Projects tab on the left, we select the Run section from the kit we are using.
In the Run section, we set the Command Line Arguments:
> output.log && (%{sourceDir}/test/scripts/runCoverage.sh ./)
What this does is redirect the output of the tests to a log file (in the build directory) and run our runCoverage.sh script if there weren’t any errors. If you’ve put the script in a different place, you will have to modify the path here.
Because running the code coverage tools can take a while, you may want to set up two run commands – one for running the regular tests without coverage and one for running them with coverage. You can use the Add dropdown to create a new run configuration.
Final Results
Now that all of that is in place, when we run the testing program, it will run the tests, process the coverage results, create an HTML report, and open it in your browser.
This shows how many lines of code and functions there are in the sample, and how many were executed.
Drilling down into the actual code file, we can see the lines that were not executed:
We forgot to test a couple of cases. If we go back to asmExampleClass_test.cpp and add a new test to check for the empty QVector:
1 2 3 4 5 6 7 8 9 |
// test adding list of numbers with empty vector void TestExampleClass::test_addSomeStuffEmptyVector() { asmExampleClass example; int result = example.addSomeStuff( {} ); QCOMPARE( result, 0 ); } |
and re-run our code coverage, we can see that we’ve now tested that code path.
Repeat until you’ve found, tested, and corrected your edge cases and you’re happy with your coverage.
Go forth and test!
In my case, my .h and .cpp are in separate folders and maybe this is the reason why the coverage report is showing only the headers. Do you know if there’s a different setup for this situation?
Unfortunately I don’t have time to experiment with this right now, but I would look at the options to lconv.
There are several related to directories – maybe you need to pass in other paths somehow so it can find the files? –base-directory?
Thanks for this. I sometimes find inexplicable GUI windows crashes in Qt difficult to trace down. Your code has made that process a much easier and a far less odious task. Thanks again!!
Great! Glad it was useful.
Thanks for the great tutorial. It saved me a lot of time.
You’re welcome!
Glad it helped.