TL;DR: You write tests that touch every line but verify nothing, creating false confidence in a broken system.
Problems 😔
Solutions 😃
Use mutation testing
Test real behaviors
Write assertive tests
Delete coverage-only tests
Refactorings ⚙️
https://maximilianocontieri.com/refactoring-011-replace-comments-with-tests
Context 💬
Many teams set a coverage threshold: 80%, 90%, or even 100%.
When you chase that number, you write tests that call methods without checking the actual results.
A test that calls calculateTax() but only asserts result is not None executes the line.
It doesn't verify the tax calculation is correct.
The dashboard turns green.
Defects survive in production.
This is vanity coverage: cosmetic metrics that hide real problems. Like brushes that make surfaces look smooth while the rot stays underneath.
Mutation testing reveals the truth.
When you mutate production code and no tests fail, your coverage numbers lied.
Sample Code 💻
Wrong 🚫
describe('BankAccount', () => {
test('deposit', () => {
const account = new BankAccount(100);
account.deposit(50);
// Only checking it didn't crash
expect(account).toBeDefined();
});
test('withdraw', () => {
const account = new BankAccount(100);
const result = account.withdraw(30);
// No assertion about the result!
});
test('transfer', () => {
const source = new BankAccount(200);
const target = new BankAccount(0);
// Just calling the method to "cover" the line
source.transfer(50, target);
});
});
Right 👉
describe('BankAccount', () => {
test('deposit increases balance', () => {
const account = new BankAccount(100);
account.deposit(50);
expect(account.balance()).toBe(150);
});
test('withdraw decreases balance', () => {
const account = new BankAccount(100);
account.withdraw(30);
expect(account.balance()).toBe(70);
});
test('withdraw raises on insufficient funds', () => {
const account = new BankAccount(50);
expect(() => account.withdraw(100))
.toThrow(InsufficientFundsError);
});
test('transfer moves money between accounts', () => {
const source = new BankAccount(200);
const target = new BankAccount(0);
source.transfer(50, target);
expect(source.balance()).toBe(150);
expect(target.balance()).toBe(50);
});
});
Detection 🔍
[X] Semi-Automatic
Run a mutation testing tool (PIT for Java, Stryker for JavaScript, mutmut for Python).
Count the surviving mutants.
If coverage is high but mutants survive, you have vanity coverage.
You can also search for assertion-free tests, single assertNotNull() assertions, or tests that still pass after you delete the entire production method body.
Exceptions 🛑
Smoke tests that call endpoints to verify the system starts are acceptable without detailed assertions.
These work when they complement a real test suite, not replace it.
Level 🔋
[x] Intermediate
Why the Bijection Is Important 🗺️
Your test suite must map each test to a real behavior in the MAPPER.
When a test covers a line without verifying observable behavior, you break that bijection.
Coverage tools only measure "lines executed."
Chase the number without bijection and your suite looks complete while missing real requirements entirely.
AI Generation 🤖
AI code generators sometimes produce vanity coverage.
Ask one to "add tests to reach 80% coverage," and it writes tests that call methods and assert trivially true facts.
The metric goes up.
Nothing gets verified.
AI Detection 🧲
AI can detect vanity coverage, but only if you ask the right questions.
Try: "Find tests with no real assertions" or "Find tests that pass when I delete the production method body."
Without those prompts, most AI tools see a green test and call it good.
Try Them! 🛠
Remember: AI Assistants make lots of mistakes
Suggested Prompt: Replace vanity coverage tests with tests that verify real behaviors and fail when production code is wrong
Conclusion 🏁
Coverage is a signal, not a goal.
When you treat it as a goal, you create vanity coverage that hides real defects.
Use mutation testing to discover what your suite actually verifies.
Write tests that describe real behaviors, not tests that execute lines.
Relations 👩❤️💋👨
https://maximilianocontieri.com/code-smell-104-assert-true
https://maximilianocontieri.com/code-smell-76-generic-assertions
https://maximilianocontieri.com/code-smell-175-changes-without-coverage
https://maximilianocontieri.com/code-smell-30-mocking-business
https://maximilianocontieri.com/code-smell-203-irrelevant-test-information
Mutation Testing
Test Coverage Is Not Enough
Stryker Mutation Testing
Quote
The most dangerous kind of waste is the waste we don't recognize.
Shigeo Shingo
Disclaimer 📘
Code Smells are my opinion.
Credits 🙏
Photo by Jamie Street on Unsplash
This article is part of the CodeSmell Series.
https://maximilianocontieri.com/how-to-find-the-stinky-parts-of-your-code
