Detecting Hidden Gaps in Test Coverage Using Branch ReportingMarch 19, 2015
At MichiganLabs, we believe that unit testing is a vital part of developing any application. Tests help document the expected behavior of your code, give instant feedback on mistakes while refactoring, and help prevent you from writing sphagetti code (since it would be difficult to test). So how do you know when you have written enough tests? This is where coverage measurement tools come in.
Coverage tools will track what parts of your code are executed by your test cases and give you a report of what remains untested. These tools exist for virtually every language. The one I use for Python is called
coverage.py. However, these tools sometimes have nuances that could trick you into thinking you have a higher amount of coverage than you really do.
Here’s a sample of some code I wrote using the Flask-RESTful framework:
class UserResource(Resource): @marshal_with(user_fields) def get(self, user_id=None, username=None): if user_id: user = User.get_by_id(user_id) elif username: user = User.get_by_username(username) if not user: abort(404) return user
I also wrote some test cases, which do something like this:
GET /users/1, assert response == 200 # Verify that I can retrieve a valid user by username GET /users/joshfriend, assert response == 200 # Verify that I can retrieve a valid user by user ID GET /users/9001, assert response == 404 # Verify that trying to retrieve a nonexistent user returns 404
I even made sure I was executing every line of code by using
$ coverage run --source app/ -m py.test tests/ $ coverage report --show-missing
And the result looked OK. 100% coverage! Right?…
tests/api/test_user_api.py ........................ Name Stmts Miss Cover Missing ------------------------------------------------------------ app/api/user 88 0 100%
… Not really. Looking back at the code, I realized I forgot something important about Python. The if statement that tests to see if a user ID was provided does not check explicitly for
None. Thus, a user ID of 0 will be ignored (
None and empty containers are considered False to an if statement). Let’s add another test case to see what happens:
GET /users/0, assert response == 200
user_id provided is ignored and no username was supplied, the
user variable is never assigned. The test case will throw an exception:
UnboundLocalError: local variable 'user' referenced before assignment
coverage.py has a convenient way of looking for this kind of gap in test coverage by recording the branches traversed in the code instead of only recording which lines were executed:
$ coverage run --branch --source app/ -m py.test tests/ tests/api/test_user_api.py ........................ $ coverage report --show-missing Name Stmts Miss Branch BrMiss Cover Missing -------------------------------------------------------------------------- app/api/user 88 0 30 1 97% $ coverage html
Note the extra command to generate an HTML report. The console reporter only reports the number of branch statements that are missing coverage, not where they are. The HTML report is an easy way to see that info. The generated report is stored in
htmlcov/index.html. For the example code above, the coverage report looks like this:
This made it obvious that I was not exercising every possible path through the code. The fix to make the new test case pass was really simple:
class UserResource(Resource): @marshal_with(user_fields) def get(self, user_id=None, username=None): if user_id is not None: user = User.get_by_id(user_id) else: user = User.get_by_username(username) if not user: abort(404) return user
Bear in mind that just because your coverage tool now reports 100% branch coverage, it does not mean your test coverage is comprehensive. Branch coverage monitoring is a good tool to help you get there but be sure to think about other edge cases, such as raised exceptions when calling external library functions. Multiple conditions or the use of the
all() builtin inside an if statement can also be tricky since the coverage tool will only record whether or not the execution path went through the if or the else section, not which conditions of the if statement made it take that route.
Stay in the loop with our latest content!
Select the topics you’re interested to receive our new relevant content in your inbox. Don’t worry, we won’t spam you.
Product Strategy with Hudson RowlandJanuary 4, 2021
From day one, MichiganLabs has built our company around our clients. This month, Hudson Rowland, delivery practice lead, shares insights about what it’s like to be a MichiganLabs client.Read more
Josh Hulst on Michigan Software Labs and his journey as the Managing PartnerMarch 18, 2020
Read the full interview with Josh Hulst from MobileAppDaily. Josh was featured in the March 2020 edition of the publication.Read more
“Learning To Code” Actually Means Different Things To Different People...And That’s OkayOctober 15, 2020
Depending on whom it is coming from, the phrase “I’d like to learn how to code” can mean wildly different things. To help shed light on the subject, I will attempt to put never-before coders into two distinct categories.Read more