C++ Insights code coverage on Windows
In my last post, I wrote about the switch from Travis CI to GitHub Actions (C++ Insights: From Travis CI to GitHub Actions. In the what's next section, I dreamed a little about getting code coverage information from the Windows build. Here is what I ended up with.
The start of the journey: MSBuild and clang-cl
While MSVC offers code coverage analysis, I couldn't get that information out in a gcov-like format. My next attempt was to use what I already knew, Clang. Clang can perform code coverage analysis on Linux and macOS. It seemed like a logical choice to use it on Windows as well. The idea was fueled by this article code-coverage-with-clang-on-windows.html. Under Windows, clang-cl.exe got a new option --coverage. That is exactly what I was looking for, and it is in a single flag. What else could I dream of? Adding --coverage in the CMakeLists.txt was a piece of cake. I also went crazy and added the flag only for the Windows platform. Yes, I know that was a little overstating, but I was happy.
Ok, I stopped being so happy after the first compilation attempt. MSVC or better, MSBuild told me that it doesn't know the option /-coverage. It seems reasonable. I don't know it either. I concluded that due to the crazy setup on Windows, using MSBuild together with clang-cl.exe to invoke the Clang-compiler but map and filter all the Windows options, passing --coverage was not supported. I also tried to pass clang-cl.exe as a linker to CMake. No success. Should you know better, please let me know!
Changing road's: Using only clang-cl
As the road with MSBuild turned out to be a dead-end, I came up with the brilliant idea to use only clang-cl. Ok, it turned out that I used MSBuild for a reason. It was the easiest to set up. It took me a couple of attempts to figure out how I have to configure clang-cl to work without MSBuild and which flags I have to pass to generate code coverage information. It compiled!
It is all about the right tools in place
Now, success was in the air. I was so certain that I was only minutes away from pushing this great change to GitHub. Boy, was I wrong! I always remind my students that there is another step after compiling and linking! The beloved lld-link.exe told me at the end of the build:
1 | |
As always, the linker was right. That file didn't exist. Not even the path lib/windows was there. This is probably the time to tell you more about the difficulties of the Windows build for a clang-AST-based tool.
The official Clang binaries for Windows do ship without the necessary libraries and programs to create a clang-AST-based tool. It does not have the AST libraries like Linux and macOS do. It also misses llvm-config, which is needed to configure C++ Insights to link properly with the LLVM libraries. Back when Windows support was added by grishavanika and when adding AppVeyor to the CI pipeline, I started using the ZigLang binaries. I'm grateful that they do a Windows build with llvm-config and the AST libraries.
However, I never noticed before I did try to get the code coverage working that they do ship without clang_rt.profile-x86_64.lib.
Luckily, thanks to the good architecture of LLVM, it is possible to compile compiler-rt for an existing Clang build, as long as there is llvm-config to configure the project accordingly. And ZigLang provides this! So I ended up setting up another GitHub Action building compiler-rt for the ZigLang binaries.
Let's start small.
This time I decided to try it out with a smaller example. I successfully compiled the code Marco showed in his post. And it worked!!! Fantastic! I once more was convinced that pushing this incredible change was now a matter of minutes! I mean, what can go wrong at this point?
Getting the code coverage information
Well, while I now had a binary that did collect code coverage information, I needed to get the information out in gcov-format to upload it to codecov.io.
A quick search revealed that there is no lcov for Windows. At least no officially. Some projects are out there using MinGW to compile a potentially patched lcov or gcov version.
Luckily I had the answer in front of me all the time. Remember Marco Castelluccio's post? In it here explained a tool called grcov developed in Rust for Firefox's code coverage analysis. It is available for Windows and did work like a charm!
After some brief struggles with the yaml-syntax again and dependency caching, I had code coverage for Windows with GitHub Actions working!
Multiple code coverage reports for codecov.io
There is one more thing I would like to mention: codecov.io. I don't remember why I chose them at the time, but I'm still happy with my decision. When I was thinking about code coverage from the Windows build, I also thought about how to see which platform contributed to which coverage, or better, on which platform is the statement not covered by a test.
I was and still am surprised about how little codecov.io talks about that. Initially, I wasn't sure they'd support it after all. All I found mentioned was that multiple uploads from the same build are merged by codecov.io. This is already a good thing, but how do we know which platform lacks a test? The, to me, relatively hidden answer was flags. I, and of course, you, can add a flag to a coverage report when uploading it. These flags appear in the Build tab (here for an example). The default is that the aggregated result from all uploads from a build is shown. When looking at an individual file, the flags are now at the top right of the diff-view. All are on by default, but we can enable and disable them to see the individual platform. For C++ Insights, you can, for example, see that Insights.cpp shows not 100% coverage. Playing with the filters, you see that the line if(gUseLibCpp) is used only on Linux.
I like this feature very much.
I hope this post helps you set up code coverage for your own project.
Support the project
You can support the project by becoming a GitHub Sponsor or, of course, with code contributions.
Andreas