Are Bad Tests Worse Than No Tests At All?
Can you write a test that does more harm than good?
In the intro to his upcoming book The Art of Unit Testing, Roy Osherove describes a case where poorly-designed tests actually hurt his team's progress:
Worse yet, some tests became unusable because the people who wrote them had left the project and no one knew how to maintain the tests, or what they were testing. The names we gave our unit test methods were not clear enough. We had tests relying on other tests. We ended up throwing away most of the tests less than 6 months into the project.I'm on the fence. Anything that exercises your code in a automatic, repeatable way has got to be useful. On the other hand, poorly written tests cost time and effort to understand, refactor, and fix.
Are the drawbacks to bad tests worse than having no coverage at all?
I think the answer is that in the short term, even bad tests are useful. Trying to squeeze a extra life out of them beyond that, however, pays diminishing returns.
Just like other software, your tests should be built for maintenance, but in a crunch, you can punch something in that works. It's better to have bad tests than to have untested code.
(photo courtesy of Still Burning, some rights reserved)
3 Comments
Leave a comment
Well, if "bad" _only_ means that they were named poorly or rely on other tests, then yes, I agree it's better to have them than to not have them.
However, if the tests exist merely to say that tests exist, the logic is wrong, and gives a false sense of security, then I feel that those serve no benefit.
Instead of someone coming in with a blank slate, they have to analyze the existing tests, find flawed logic, find what's not _really_ covered, and refactor/redo the tests. Those are what I would consider "bad" tests, and not worth having.
Dave, roger that. You seem to have hit on a common theme.
This entry was reposted over at JavaLobby, where a commenter had pretty much the same response as you.
r1epsj8f30inps7d