Description

You will lead the end-to-end architecture of Databricks-based data and analytics platforms on AWS.

Responsibilities

• Design scalable ELT/ETL pipelines using Databricks, Spark, and Delta Lake.
• Architect data ingestion, data quality, governance, and lineage frameworks.
• Implement CI/CD pipelines for Databricks notebooks and jobs.
• Optimize Spark workloads and cluster configurations for performance and cost.
• Provide architectural guidance and governance across engineering teams.

Required Skills

• 10+ years in data engineering or architecture.
• 4+ years specifically with Databricks.
• Strong hands-on experience with Apache Spark (PySpark/Scala).
• Expert knowledge of AWS architecture, including S3, IAM, Glue, Lambda, EMR, and Redshift.
• Experience implementing Delta Lakehouse architecture.
• Strong knowledge of CI/CD tools like Azure DevOps, GitHub Actions, or Jenkins.
• Proficiency in Data Management and Governance practices.

Key Skills

Education

Any Gradute