AI generated ETL ran clean on the first try, and it took six weeks to find out what it was missing. A late Monday file, a file sent twice, a file full of zeros and mangled names all got through. The ugly old job had a rule for every one of them, and nobody had written those rules down. Now I ask AI for the failure list before I ask for any code.
Data Quality and Observability: A Field Guide in Six Parts
Master data quality and observability in six parts: name the problem, rank the risk, define rules, place controls, monitor for drift, and respond calmly.
You Just Found Bad Data in Production. Now What?
Found bad data in production? Use this data incident response playbook: triage, contain, trace the source, fix, verify, tell people, and review calmly.
It Passed Every Test and Still Broke: Four Signals Validation Never Catches
Every test passed and the dashboard still broke. Here are four signals validation never catches: freshness, volume, schema drift, and distribution shift.
Where Should a Data Quality Check Live? Gates, Controls, and Quarantine
A check in the wrong place catches bad data too late. See where each data quality check belongs: ingestion, transformation, release gates, and quarantine.
“Clean Data” Is Not a Requirement: Writing Rules People Can Act On
Clean is a feeling, not a spec. Write data quality rules people can act on, with a testable check, a threshold, and a named owner, so every failure is owned.
Count Is Not Risk: Ranking Data Issues by What They Actually Cost
The biggest pile of errors is rarely your biggest problem. Try ranking data issues by impact and likelihood, not row count, and fix what actually hurts.