TL;DR: Overly large auto-generated config can crash your system.
Problems 😔
- Config overload
- Hardcoded limit
- Lack of validations
- Crash on overflow
- Fragile coupling
- Cascading Failures
- Hidden Assumptions
- Silent duplication
- Unexpected Crashes
- Thread panics in critical paths
- Treating internal data as trusted input
- Poor observability
- Single point of failure in internet infrastructure
Solutions 😃
- Validate inputs early
- Enforce soft limits
- Fail-fast on parse
- Monitor config diffs
- Version config safely
- Use backpressure mechanisms
- Degrade functionality gracefully
- Log and continue
- Improve degradation metrics
- Implement proper Result/Option handling with fallbacks
- Treat all configuration as untrusted input
Refactorings ⚙️
Context 💬
In the early hours of November 18, 2025, Cloudflare’s global network began failing to deliver core HTTP traffic, generating a flood of 5xx errors to end users.
This was not caused by an external attack or security problem.
The outage stemmed from an internal "latent defect" triggered by a routine configuration change
The failure waxed and waned, until a fix was fully deployed.
The root cause lay in a software bug in Cloudflare’s Bot Management module and its downstream proxy logic.
The Technical Chain of Events
Database Change (11:05 UTC): A ClickHouse permissions update made previously implicit table access explicit, allowing users to see metadata from both the default and r0 databases.
SQL Query Assumption: A Bot Management query lacked a database name filter:
SELECT name, type FROM system.columns
WHERE table = 'http_requests_features'
ORDER BY name;
This query began returning duplicate rows—once for default database, once for r0 database.
Feature File Explosion: The machine learning feature file doubled from ~60 features to over 200 features with duplicate entries.
Hard Limit Exceeded: The Bot Management module had a hard-coded limit of 200 features (for memory pre-allocation), which was now exceeded.
The Fatal .unwrap(): The Rust code called .unwrap() on a Result that was now returning an error, causing the thread to panic with "called Result::unwrap() on an Err value". see code below
Global Cascade: This panic propagated across all 330+ data centers globally, bringing down core CDN services, Workers KV, Cloudflare Access, Turnstile, and the dashboard.
The estimated financial impact across affected businesses ranges from $180-360 million.
Sample Code 📖
Wrong ❌
let features: Vec<Feature> = load_features_from_db();
let max = 200;
assert!(features.len() <= max);
# This magic number assumption
# is actually wrong
for f in features {
proxy.add_bot_feature(f.unwrap());
# You also call unwrap() on every feature.
# If the database returns an invalid entry
# or a parsing error,
# you trigger another panic.
# You give your runtime no chance to recover.
# You force a crash on a single bad element.
}
# A quiet config expansion turns into
# a full service outage
# because you trust input that you should validate
# and you use failure primitives (assert!, unwrap())
# that kills your program
# instead of guiding it to safety
Right 👉
fn load_and_validate(max: usize) -> Result<Vec<Feature>, String> {
let raw: Vec<Result<Feature, Error>> = load_features_from_db();
if raw.len() > max {
return Err(format!(
"too many features: {} > {}",
raw.len(), max
));
}
Ok(raw.into_iter()
.filter_map(|r| r.ok())
.collect())
}
Detection 🔍
You can detect this code smell by searching your codebase for specific keywords:
.unwrap() - Any direct call to this method.expect() - Similarly dangerouspanic!() - Explicit panics in non-test codethread::panic_any() - Panic without context
When you find these patterns, ask yourself: "What happens to my system when this Result contains an Err?" If your honest answer is "the thread crashes and the request fails," then you've found the smell.
You can also use automated linters. Most Rust style guides recommend tools like clippy, which flags unwrap() usage in production code paths.
When you configure clippy with the #![deny(unwrap_in_result)] attribute, you prevent new unwrap() calls from entering your codebase.
Level 🔋
[x] Advanced
Why the Bijection Is Important 🗺️
Your internal config generator must map exactly what your code expects.
A mismatched config (e.g., duplicated metadata) breaks the bijection between what your config represents and what your proxy code handles.
When you assume "this file will always have ≤200 entries", you break that mapping.
Reality sends 400 entries → your model explodes → the real world wins, your service loses.
That mismatch causes subtle failures that cascade, especially when you ignore validation or size constraints.
Ensuring a clean mapping between the config source and code input helps prevent crashes and unpredictable behavior.
AI Generation 🤖
AI generators often prioritize "correct" logic over "resilient" logic.
If you ask an AI to "ensure the list is never larger than 200 items," it might generate an assertion or a panic because that is the most direct way to satisfy the requirement, introducing this smell.
The irony: Memory-safe languages like Rust prevent undefined behavior and memory corruption, but they can't prevent logic errors, poor error handling, or architectural assumptions. Memory safety ≠ System safety.
AI Detection 🧲
AI can easily detect this if you instruct it to look for availability risks.
You can use linters combined with AI to flag panic calls in production code.
Human review on critical functions is more important than ever.
Try Them! 🛠
Remember: AI Assistants make lots of mistakes
Suggested Prompt: remove all .unwrap() and .expect() calls. Return Result instead and validate the vector bounds explicitly
Conclusion 🏁
Auto-generated config can hide duplication or grow unexpectedly.
If your code assumes size limits or blindly trusts its input, you risk a catastrophic crash.
Validating inputs is good; crashing because an input is slightly off is a disproportionate response that turns a minor defect into a global outage.
Validate config, enforce limits, handle failures, and avoid assumptions.
That’s how you keep your system stable and fault-tolerant.
Relations 👩❤️💋👨
Disclaimer 📘
Code Smells are my opinion.
A good programmer is someone who always looks both ways before crossing a one-way street
Douglas Crockford
Douglas Crockford
This article is part of the CodeSmell Series.
