Data Quality

Data quality tooling watches whether the data is right, which is a different and less-served problem than whether the pipeline ran. A pipeline can succeed perfectly and deliver nonsense, and the usual way this is discovered is a person noticing a number looks wrong several weeks later, after decisions have been made from it.

The core mechanic is testing data the way you test code. Assertions about what should be true — this column is never null, this identifier is unique, this value falls in a known set, this table gains between eight and twelve thousand rows a day, this total reconciles with that one — run on every load and fail loudly when violated. Most quality problems are caught by a small number of unglamorous assertions of exactly this kind.

Freshness is the single highest-value check and the most commonly missing. A table that stopped updating is indistinguishable from a table with no new activity, and dashboards will happily show stale numbers forever without complaint. An alert when a table has not updated within its expected window catches a large share of real incidents before anyone downstream is affected.

Anomaly detection — learning normal ranges and flagging deviations — is useful as a supplement and dangerous as a foundation, because it is noisy. A detector that fires on every seasonal pattern trains people to ignore it, and an ignored alert is worse than no alert. Start with explicit assertions you can reason about and add statistical detection for things you cannot enumerate.

Lineage is what makes an incident tractable. When a number is wrong, the question is what fed it and what it feeds, and answering that by reading code under pressure is slow. Tools that map lineage automatically turn a half-day investigation into a five-minute one, and they also answer the question that arises before every schema change: what breaks if I alter this column?

Ownership is the part software cannot supply. Every important table needs a person who is alerted and who is expected to act, because an alert routed to a shared channel is routed to nobody.

The honest framing for this subcategory is insurance. It produces no new insight and prevents you from confidently acting on something false, which is the more expensive failure.