Coveralls in a monorepo
A short rantdevcoverage
One of my recent tasks at work was to implement a workflow to upload coverage reports to Coveralls after each test run in CI. I expected it to be an easy task since we already have our CI set up to run all the tests. Now it’s just a matter of generating coverage reports and uploading them using an API right?
Hell no, I was completely unprepared for the shitshow that lay ahead.
Here’s a quick breakdown of the tools used:
- coveralls-python for Python, which reads from the .coverage file (actually a SQLite database) generated by coverage.py.
- goveralls for Golang. Go has its own coverage generator with a unique output format.
- csmacnz.Coveralls for .NET. We use xUnit as the test runner, which generates an XML file (more accurately, a Cobertura XML file) using Coverlet.
- sbt-jacoco for Scala. JaCoCo is a test runner, and the sbt-jacoco plug-in promises a built-in integration with Coveralls.
Coveralls has a very simple one-page API reference page. To display all the data correctly, these are the requisite information:
- Coveralls token
- File path and coverage
- Git metadata (branch name, commit hash, commit message)
- CI event (push, pull request)
- Parallel flag
- Service number (build number, i.e. a unique number for each unit-test run)
Guess what? There’s no standardised way to specify these input. Sure, some libraries behave similarly for certain inputs, e.g. passing the token in through a
COVERALLS_REPO_TOKEN is almost universal (
COVERALLS_TOKEN), but in the bid to support every CI imaginable “out of the box”, each library implements their unique flavour of if-else hell:
- CI detection:
coveralls-pythonautodetects that GitHub Actions is used by snooping for a
GITHUB_ACTIONSin the envs. However, if you pass in a
COVERALLS_REPO_TOKEN, it will stop working.
node-coverallsclaims to support GitHub Actions, but passing in
COVERALLS_SERVICE_NAMEas per their README produces an error. Turns out they don’t.
- Git branch: Name of git branch is derived differently.
sbt-jacocoexpects you to provide it through the envs.
coveralls-pythontakes it from
csmacnz.Coverallsruns a git command in a subprocess. Unfortunately, there’s no guarantee that these are the same. Like Travis CI, GitHub’s very own
actions/checkout@v2checks out a merge commit from the HEAD to the upstream branch, while passing the HEAD as the
GITHUB_REF. Also, if your runner is not using Git 1.18 or higher, there won’t even be a
.gitfolder at all!
- Service number:
GITHUB_RUN_IDby default, but
node-coverallswill require the
COVERALLS_SERVICE_NUMBERto be passed in.
- Base path: Not all libraries allow specifying the base path, but this should be the single most important information, since this is likely the most inconsistent thing between test runners.
- Oh, and for some reason
node-coverallsdoes not allow you to alter the LCOV format. There’s an integrity check logic, so if you need to modify the base path, you would have to move the report file and even rename the working directory folder first.
- Cherry on top: I discovered this the last, but
sbt-jacocodoes not support the parallel flag at all.
Honestly, I don’t know. I don’t even know if the points I raised here are valid concerns at all. I do however believe that my overall experience would have been a lot more comfortable if
- there is a standard format for code coverage reports, and most, if not all, languages have a test runner that generates coverage reports in that format. The LCOV format comes close to this ideal.
- Coveralls releases their own, or curate and officially endorse, libraries for each language.
I don’t see either happening anytime soon. But thankfully, setting all these up is a one-time thing.