Description

Build and maintain metadata-driven ingestion frameworks and CI/CD pipelines on Databricks. You will own the lifecycle of batch, incremental, and CDC data pipelines, ensuring they adhere to platform standards for configuration, schema evolution, and error handling.

This role is on-site.

Responsibilities

• Implement metadata-driven ingestion frameworks and standard ingestion patterns for batch, incremental, and CDC data.
• Build and maintain CI/CD pipelines using GitHub Actions, including configuration validation, environment promotion, and rollback mechanisms.
• Implement pipeline monitoring, metrics, and alerting, supporting operational validation and failure simulation.
• Embed data quality rules and quarantine patterns, applying Unity Catalog tags at ingestion time.
• Ensure governance metadata is written to governed schemas and manage secrets via AWS Secrets Manager.

Required Skills

• 5–8 years of data or platform engineering experience.
• 3+ years of hands-on Databricks experience.
• Strong proficiency in PySpark and SQL.
• Experience with Delta Lake and Unity Catalog.
• Experience with Git-based development workflows.
• Ability to build reusable ingestion frameworks that reduce custom pipeline development.
• Experience with production platform support and proactive operational monitoring.

Preferred Skills

• Deep experience with metadata-driven ingestion frameworks.
• Proven track record implementing monitoring and alerting systems.

Key Skills

Education

Bachelor's degree