Detecting Hidden Gaps in Test Coverage Using Branch Reporting

March 19, 2015

At MichiganLabs, we believe that unit testing is a vital part of developing any application. Tests help document the expected behavior of your code, give instant feedback on mistakes while refactoring, and help prevent you from writing sphagetti code (since it would be difficult to test). So how do you know when you have written enough tests? This is where coverage measurement tools come in.

Coverage tools will track what parts of your code are executed by your test cases and give you a report of what remains untested. These tools exist for virtually every language. The one I use for Python is called coverage.py. However, these tools sometimes have nuances that could trick you into thinking you have a higher amount of coverage than you really do.

Here’s a sample of some code I wrote using the Flask-RESTful framework:

class UserResource(Resource):
 @marshal_with(user_fields)
 def get(self, user_id=None, username=None):
 if user_id:
 user = User.get_by_id(user_id)
 elif username:
 user = User.get_by_username(username)
 if not user:
 abort(404)
 return user

I also wrote some test cases, which do something like this:

GET /users/1, assert response == 200
# Verify that I can retrieve a valid user by username
GET /users/joshfriend, assert response == 200
# Verify that I can retrieve a valid user by user ID
GET /users/9001, assert response == 404
# Verify that trying to retrieve a nonexistent user returns 404

I even made sure I was executing every line of code by using coverage.py:

$ coverage run --source app/ -m py.test tests/
$ coverage report --show-missing

And the result looked OK. 100% coverage! Right?…

tests/api/test_user_api.py ........................
Name Stmts Miss Cover Missing
------------------------------------------------------------
app/api/user 88 0 100%

… Not really. Looking back at the code, I realized I forgot something important about Python. The if statement that tests to see if a user ID was provided does not check explicitly for None. Thus, a user ID of 0 will be ignored (, None and empty containers are considered False to an if statement). Let’s add another test case to see what happens:

GET /users/0, assert response == 200

Since the user_id provided is ignored and no username was supplied, the user variable is never assigned. The test case will throw an exception:

UnboundLocalError: local variable 'user' referenced before assignment

Fortunately, coverage.py has a convenient way of looking for this kind of gap in test coverage by recording the branches traversed in the code instead of only recording which lines were executed:

$ coverage run --branch --source app/ -m py.test tests/
tests/api/test_user_api.py ........................
$ coverage report --show-missing
Name Stmts Miss Branch BrMiss Cover Missing
--------------------------------------------------------------------------
app/api/user 88 0 30 1 97%
$ coverage html

Note the extra command to generate an HTML report. The console reporter only reports the number of branch statements that are missing coverage, not where they are. The HTML report is an easy way to see that info. The generated report is stored in htmlcov/index.html. For the example code above, the coverage report looks like this:

Branch Coverage

This made it obvious that I was not exercising every possible path through the code. The fix to make the new test case pass was really simple:

class UserResource(Resource):
 @marshal_with(user_fields)
 def get(self, user_id=None, username=None):
 if user_id is not None:
 user = User.get_by_id(user_id)
 else:
 user = User.get_by_username(username)
 if not user:
 abort(404)
 return user

Bear in mind that just because your coverage tool now reports 100% branch coverage, it does not mean your test coverage is comprehensive. Branch coverage monitoring is a good tool to help you get there but be sure to think about other edge cases, such as raised exceptions when calling external library functions. Multiple conditions or the use of the all() builtin inside an if statement can also be tricky since the coverage tool will only record whether or not the execution path went through the if or the else section, not which conditions of the if statement made it take that route.

Looking for more like this?

Sign up for our monthly newsletter to receive helpful articles, case studies, and stories from our team.

Lessons Learned from our Associate Developer
Team

Lessons Learned from our Associate Developer

September 13, 2023

One of our Associate Software Developers, Rohit, reflects on his time at MichiganLabs working on a short-term project, what he learned about real-world development, and the software consultancy business model.

Read more
5 takeaways from the Do iOS conference that will benefit our clients
iOS

5 takeaways from the Do iOS conference that will benefit our clients

November 30, 2023

Read more
Chicago Roboto 2022 Retrospective
Android Development

Chicago Roboto 2022 Retrospective

August 11, 2022

Scott Schmitz shares some notes of interest from talks at Chicago Roboto 2022, an Android community conference, that took place August 1-2.

Read more
View more articles