Detecting Hidden Gaps in Test Coverage Using Branch Reporting

March 19, 2015

At MichiganLabs, we believe that unit testing is a vital part of developing any application. Tests help document the expected behavior of your code, give instant feedback on mistakes while refactoring, and help prevent you from writing sphagetti code (since it would be difficult to test). So how do you know when you have written enough tests? This is where coverage measurement tools come in.

Coverage tools will track what parts of your code are executed by your test cases and give you a report of what remains untested. These tools exist for virtually every language. The one I use for Python is called coverage.py. However, these tools sometimes have nuances that could trick you into thinking you have a higher amount of coverage than you really do.

Here’s a sample of some code I wrote using the Flask-RESTful framework:

class UserResource(Resource):
 @marshal_with(user_fields)
 def get(self, user_id=None, username=None):
 if user_id:
 user = User.get_by_id(user_id)
 elif username:
 user = User.get_by_username(username)
 if not user:
 abort(404)
 return user

I also wrote some test cases, which do something like this:

GET /users/1, assert response == 200
# Verify that I can retrieve a valid user by username
GET /users/joshfriend, assert response == 200
# Verify that I can retrieve a valid user by user ID
GET /users/9001, assert response == 404
# Verify that trying to retrieve a nonexistent user returns 404

I even made sure I was executing every line of code by using coverage.py:

$ coverage run --source app/ -m py.test tests/
$ coverage report --show-missing

And the result looked OK. 100% coverage! Right?…

tests/api/test_user_api.py ........................
Name Stmts Miss Cover Missing
------------------------------------------------------------
app/api/user 88 0 100%

… Not really. Looking back at the code, I realized I forgot something important about Python. The if statement that tests to see if a user ID was provided does not check explicitly for None. Thus, a user ID of 0 will be ignored (, None and empty containers are considered False to an if statement). Let’s add another test case to see what happens:

GET /users/0, assert response == 200

Since the user_id provided is ignored and no username was supplied, the user variable is never assigned. The test case will throw an exception:

UnboundLocalError: local variable 'user' referenced before assignment

Fortunately, coverage.py has a convenient way of looking for this kind of gap in test coverage by recording the branches traversed in the code instead of only recording which lines were executed:

$ coverage run --branch --source app/ -m py.test tests/
tests/api/test_user_api.py ........................
$ coverage report --show-missing
Name Stmts Miss Branch BrMiss Cover Missing
--------------------------------------------------------------------------
app/api/user 88 0 30 1 97%
$ coverage html

Note the extra command to generate an HTML report. The console reporter only reports the number of branch statements that are missing coverage, not where they are. The HTML report is an easy way to see that info. The generated report is stored in htmlcov/index.html. For the example code above, the coverage report looks like this:

Branch Coverage

This made it obvious that I was not exercising every possible path through the code. The fix to make the new test case pass was really simple:

class UserResource(Resource):
 @marshal_with(user_fields)
 def get(self, user_id=None, username=None):
 if user_id is not None:
 user = User.get_by_id(user_id)
 else:
 user = User.get_by_username(username)
 if not user:
 abort(404)
 return user

Bear in mind that just because your coverage tool now reports 100% branch coverage, it does not mean your test coverage is comprehensive. Branch coverage monitoring is a good tool to help you get there but be sure to think about other edge cases, such as raised exceptions when calling external library functions. Multiple conditions or the use of the all() builtin inside an if statement can also be tricky since the coverage tool will only record whether or not the execution path went through the if or the else section, not which conditions of the if statement made it take that route.

Josh Friend
Josh Friend
Development Practice Co-Lead

Stay in the loop with our latest content!

Select the topics you’re interested to receive our new relevant content in your inbox. Don’t worry, we won’t spam you.

Product Strategy with Hudson Rowland
Business

Product Strategy with Hudson Rowland

January 4, 2021

From day one, MichiganLabs has built our company around our clients. This month, Hudson Rowland, delivery practice lead, shares insights about what it’s like to be a MichiganLabs client.

Read more
Josh Hulst on Michigan Software Labs and his journey as the Managing Partner
Team

Josh Hulst on Michigan Software Labs and his journey as the Managing Partner

March 18, 2020

Read the full interview with Josh Hulst from MobileAppDaily. Josh was featured in the March 2020 edition of the publication.

Read more
“Learning To Code” Actually Means Different Things To Different People...And That’s Okay
Development

“Learning To Code” Actually Means Different Things To Different People...And That’s Okay

October 15, 2020

Depending on whom it is coming from, the phrase “I’d like to learn how to code” can mean wildly different things. To help shed light on the subject, I will attempt to put never-before coders into two distinct categories.

Read more
View more articles