AI generated ETL ran clean on the first try, and it took six weeks to find out what it was missing. A late Monday file, a file sent twice, a file full of zeros and mangled names all got through. The ugly old job had a rule for every one of them, and nobody had written those rules down. Now I ask AI for the failure list before I ask for any code.
First Look at Apache Airflow 3.0: A Game-Changer in Workflow Orchestration
If you’ve spent any time wrangling data pipelines, you know that Apache Airflow is a staple in the orchestration world. Let us learn.
Apache Spark and Airflow in Action : My Experience
Recently, I’ve been working with a large client where we’re diving deep into Apache Spark and Apache Airflow. Let us learn!
Data Quality Asserts in T-SQL: Failing a Load When Rules Break
Write data quality asserts in T-SQL to reject invalid staging rows, preserve check results, and record each load attempt.
TRY_CAST vs CAST: Finding Rows That Will Not Convert Before a Load
Find text values that will not convert before loading, separate invalid and missing input, and use explicit date styles and culture parsing.
NOCHECK CONSTRAINT: Loading Data and Trusting Foreign Keys Again
Use NOCHECK CONSTRAINT carefully, distinguish enabled from trusted foreign keys, validate old rows, and restore optimizer assumptions.
Detecting Changed Rows With a Row Hash
Compare one fingerprint instead of twenty columns. I show a safe HASHBYTES recipe, the NULL trap that fools CONCAT_WS, and a load that touches only the rows that changed.