Description

Key Responsibilities – Platform Engineering
- Implement metadata-driven ingestion frameworks and standard ingestion patterns
- Build batch, incremental, and CDC ingestion pipelines
- Ensure pipelines follow platform standards for configuration, schema evolution, tagging, and error handling
Observability & Operations
- Implement pipeline monitoring, metrics, and alerting
- Support operational validation, failure simulation, and recovery testing
Data Quality & Governance
- Embed data quality rules and quarantine patterns
- Apply Unity Catalog tags (domain, owner, sensitivity, freshness) at ingestion time
- Ensure governance metadata is written to governed schemas
CI/CD & Automation
- Build CI/CD pipelines using GitHub Actions
- Support configuration validation, environment promotion, and rollback
- Ensure secret management via AWS Secrets Manager
Required Qualifications
- 5–8 years data or platform engineering experience
- 3+ years hands-on Databricks experience
- Strong PySpark and SQL
- Experience with Delta Lake, Unity Catalog, Git-based development
Preferred Qualifications
- Ingestion framework experience
- Monitoring and alerting implementation
- Production platform support experience
Success Measures
- Stable, reusable ingestion frameworks
- Reduced custom pipeline development
- Reliable CI/CD and proactive operational monitoring

Key Skills

Education

Bachelor's degree