Every few weeks a backend team would rename a column and our pipelines would break silently.
Reyansh Iyer
Data Engineer | Spark, Airflow, dbt, Snowflake | Pipelines that don't break on Mondays
Every few weeks a backend team would rename a column and our pipelines would break silently.
What we did:
1. Producers define a contract (schema + semantics + owner) in YAML in their repo
2. CI checks on the producer side block breaking changes unless the version is bumped
3. We consume via a versioned view, never the raw table
4. Breaking changes need a 2-week deprecation window
Incidents from upstream changes: 11 last year, 1 this year. The cultural part (producers owning their data) was harder than the tooling.
5 reactions3 comments
