Description

You will own the data quality and observability stack for production ingestion pipelines. Your work ensures data freshness, reliability, and cost efficiency across the platform.

Responsibilities

• Implement pipeline-level metrics collection for records in/out, processing lag, throughput, and failure counts.
• Build and maintain monitoring dashboards in CloudWatch and Grafana for all production ingestion pipelines.
• Configure alerting thresholds tied to SLAs to trigger on pipeline lag, failures, and data quality breaches.
• Design and implement data quality rules (completeness, schema conformance, record counts, freshness) in collaboration with Data/Cloud Engineers.
• Produce automated weekly and monthly SLA reports showing ingestion success rates, data freshness, and trend analysis.

Required Skills

• 9-12 years of experience in data quality engineering, data observability, or data operations with a platform focus.
• Hands-on experience with AWS CloudWatch (metrics, logs, alarms, dashboards) and Grafana.
• Strong experience implementing data quality frameworks such as Great Expectations, dbt tests, Deequ, or Soda.
• Proficiency in SQL and Python for metrics collection, reporting automation, and DQ rule implementation.
• Familiarity with data lineage and cataloging tools like AWS Glue Catalog, Apache Atlas, DataHub, or OpenLineage.
• Understanding of data observability concepts including freshness, volume, schema change detection, and distribution anomalies.
• Experience with AWS cost monitoring tools (Cost Explorer, CUR, CloudWatch billing metrics).

Key Skills

Education

Bachelor's degree