Stage 0

Compute statistics from the one untouched batch and save them to baseline_stats.csv

Stage 1

Ingestion

Stage 2

Numeric drift vs baseline

Stage 3

Schema check

Stage 4

Duplicate row check

Stage 5

Univariate outliers

Stage 6

Multivariate anomaly detection (Isolation Forest)

Stage 7

Combine into one row per batch, flag, close the loop

Stage 8

Alerting

Stage 9

Reporting

Data Quality Monitoring Pipeline

• Capture baseline statistics from a known-clean reference batch (Quantity, Price, Customer ID)

• Loop through each monthly batch file automatically

• Compare each batch's stats to baseline: mean drift, missing-value drift

• Check column count against expected schema (catches dropped/added columns)

• Flag exact duplicate rows (catches reprocessing or pipeline reruns)

• Detect univariate outliers (IQR) and multivariate anomalies (Isolation Forest)

• Combine all signals into one quality_status per batch: OK, WARNING, or CRITICAL

• Output a full results table, a chart, and a flagged-batches file for review