Measuring unit test coverage in multi-module Android projects using Jacoco: Part 1

Photo by Christian Kaindl on Unsplash

• Part 1: test reports generation [this article]
• Part2: coverage verification & enforcement

Not so long time ago our Android team decided to push our processes related to unit testing to the next level. We’ve already had some verbal agreements but most of the time everything was a bit spontaneous (not to say chaotic). We didn’t have any formal rules and metrics and as some famous man said — “you can’t improve something if you cannot measure it”. So I started searching for a way to automate our processes related to writing tests. Quick Googling led me to Jacoco: there are a lot of articles and blog posts on this topic but most of them appeared to be not quite compatible with Android specifics and especially with multi-module setup. So I decided to sum up all of my findings and share the results with the community in this blog post. Welcome!

Let’s start by adding Jacoco plugin to your project if you haven’t done it yet. To begin with, create a new jacoco.gradle file in the root project folder, it will contain most of the Jacoco-related logic once we’re done:

apply plugin: 'jacoco'
jacoco {
    toolVersion = "0.8.5"
}

Now we’re ready to start filling it with useful stuff. Essentially, Jacoco provides two main and quite independent features:

1. Test reports generation (HTML, XML, etc.). jacocoTestReport Gradle task is responsible for that. This feature is pretty straightforward — it simply generates human-readable reports for all Kotlin classes so you can use them to get an idea of your current coverage (measured in %) in a particular file, folder, package, or module. That’s what we’re going on to discuss in this first article.
2. Test coverage verification. This one is more interesting: it allows you to define some coverage threshold (like 80%) and to execute a check against your code which will say whether this threshold is reached or not. Depending on your needs, you can make your remote pipeline fail in Gitlab/GitHub or even prevent local builds from compilation if this check fails. This is what Jacoco’s jacocoTestCoverageVerification Gradle task is for. I am going to discuss this topic in the next article.

In most cases, feature #1 should look like a reasonable thing to begin with, so let’s get into more details on how to set up test reports generation using Jacoco in your Android project.

Nowadays, many multi-module projects contain both traditional Android (the ones which have either com.android.library or com.android.application plugins applied in their gradle file) and pure Java/Kotlin (java-library plugin) modules. Our project is no exception here. From unit testing and Jacoco standpoints, these modules are a little bit different and require separate setup strategies. So it would be great to have a way to differentiate all of that at the Gradle level. Let’s add the following Groovy function to the jacoco.gradle we’ve created earlier:

private static boolean isAndroidModule(Project project) {
    boolean isAndroidLibrary = project.plugins.hasPlugin('com.android.library')
    boolean isAndroidApp = project.plugins.hasPlugin('com.android.application')
    return isAndroidLibrary || isAndroidApp
}

Now we’re ready to set up Jacoco’s Gradle tasks depending on the current module type:

afterEvaluate { project ->
    if (isAndroidModule(project)) setupAndroidReporting()
    else setupKotlinReporting()
}

“Kotlin reporting” is fairly easy to set up:

def setupKotlinReporting() {
    jacocoTestReport {
        dependsOn test
        reports {
            csv.enabled false // change if needed
            xml.enabled false // change if needed
            html {
                enabled true
                destination file("${buildDir}/coverage-report")
            }
        }
    }
}

Android reporting is more complicated:

def setupAndroidReporting() {
tasks.withType(Test) {
        // Whether or not classes without source location should be instrumented
        jacoco.includeNoLocationClasses true
    }
task jacocoTestReport(
            type: JacocoReport, 
            dependsOn: ['testDebugUnitTest']
    ) {
        reports {
            csv.enabled false
            xml.enabled false
            html {
                enabled true
                destination file("${buildDir}/coverage-report")
            }
        }
        // Change as needed
        def fileFilter = [
                '**/*App.*',
                '**/*Application.*',
                '**/*Activity.*',
                '**/*Fragment.*',
                '**/*JsonAdapter.*', // adapters generated by Moshi
                '**/di/**',
                '**/*Dagger.*'
        ]
        def debugTree = fileTree(
                dir: "$buildDir/tmp/kotlin-classes/debug",
                excludes: fileFilter
        )
        def mainSrc = "$projectDir/src/main/java"
sourceDirectories.from = files([mainSrc])
        classDirectories.from = files([debugTree])
        executionData.from = fileTree(
                dir: project.buildDir,
                includes: [
                        'jacoco/testDebugUnitTest.exec',
                        'outputs/code-coverage/connected/*coverage.ec'
                ]
        )
    }
}

A couple of notes:

• We don’t write a lot of Android Tests (the ones which are located in/androidTest directory instead of usual /test) and we don’t really want such tests to affect our unit test coverage so I’ve excluded them from this setup. They can be added to the generation/verification process fairly easily though, the main change would be to enable testCoverageEnabled flag somewhere inside the setupAndroidReporting + you will also need to add one more dependency to dependsOn section:

dependsOn: ['testDebugUnitTest', 'createDebugCoverageReport']

• You may want to customize defFilter so it works best for your particular project. For instance, I don’t want UI things like Fragments and Activities to affect overall coverage so I added them to exclusions. By the way, you may also want to have the same set of exclusions for both Android and Kotlin modules. In our case, we don’t need any exceptions for Java/Kotlin modules.
• Java or mixed Java/Kotlin projects might need to tweak and extend source directories from the above. Pure Kotlin projects (like ours) should work fine with the setup from above in most cases.

We’re done with the basic setup for jacoco.gradle file. Let’s make its features available to actual modules. The simplest solution would be to add the following line to every module’s build.gradle file:

apply from: '../jacoco.gradle'

But this approach is not flexible enough as it leads to copy-pasting. And what’s more important — it’s super easy to forget to add this line when you’re creating a new module. If you’re not happy with these limitations you can try adding the following code to the top-level gradle file of your project:

subprojects {
    afterEvaluate { project ->
        project.apply from: '../jacoco.gradle'
    }
}

This will automatically apply all the Jacoco stuff from jacoco.gradle file to all Gradle submodules. You can further extend this approach by adding extra checks inside afterEvaluate block if you need to, here’s a simple example:

if (project.projectDir.toString().contains("somefolder/")) {
    // Apply jacoco only to modules inside "somefolder" folder
    project.apply from: '../jacoco.gradle'
}

That’s basically it, now we’re ready to generate HTML reports for all of our modules. Just open the Terminal tab in your Android Studio and execute the following command:

./gradlew jacocoTestReport

Alternatively, you can generate reports for individual modules by explicitly specifying the module name:

./gradlew your-module-name:jacocoTestReport

Your reports will be generated in <module>/build/coverage-report/ directory, they basically look like this:

Generated HTML report example

That’s basically it. Now you have everything needed to estimate and measure unit test coverage across all modules of your project. jacocoTestReport could also help you to find weak spots which require some extra attention from its maintainers.

But this is only the first step on the way to better unit testing. In the next article, I’d like to keep extending our jacoco.gradle to also includejacocoTestCoverageVerification feature: I’ll show how I made it work with our multi-module setup + how it can be integrated into your CI/CD pipelines. Stay tuned :-)

Bonus: ignore list. Let’s say you have some modules you’d like to exclude from all the measurements (to save time & resources for instance). Here’s how it can be done:

1. Create a newjacoco-ignore-list.gradle file next to the jacoco.gradle(it will evolve into a separate Jacoco config in the second article + it might be more convenient for you and other devs from your team to keep all configurable stuff independent from Jacoco implementation/integration details):

ext {
    jacocoIgnoreList = [
            "module-name-1",
            "module-name-2",
            ...
    ]
}

2. Extendjacoco.gradle a little bit:

apply from: '../jacoco-ignore-list.gradle'
...
afterEvaluate { project ->
    def ignoreList = jacocoIgnoreList
    def projectName = project.name
    if (ignoreList.contains(projectName)) {
        println "Jacoco: ignoring project ${projectName}"
        return false
    }
if (isAndroidModule(project)) setupAndroidReporting()
    else setupKotlinReporting()
}

3. Enjoy!