Description

You will design, build, and optimize large-scale data pipelines on the Databricks Lakehouse Platform on AWS, driving automated CI/CD and deployment practices.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using Databricks on AWS, developing workflows with PySpark/Spark and SQL.
• Implement CI/CD pipelines for Databricks using Git, GitLab, GitHub Actions, or AWS-native tools, managing automated deployments with Databricks Asset Bundles.
• Tune Spark clusters, workflows, and queries for cost efficiency and compute performance, implementing monitoring and observability.
• Implement and maintain data governance using Unity Catalog, enforcing access controls and data compliance.
• Translate business requirements into scalable data implementations while documenting data flows and technical architecture.

Required Skills

• 10+ years of professional experience in data engineering.
• Strong hands-on experience with Databricks, including Delta Lake, Unity Catalog, and Lakehouse Architecture.
• Proficiency in PySpark, Spark, and advanced SQL.
• Expertise with AWS cloud services, specifically S3 and IAM.
• Demonstrable skills in DevOps practices, including Git, GitLab, and CI/CD pipelines.
• Experience managing infrastructure via Terraform or CloudFormation.
• Ability to design low-latency data access strategies beyond Spark-only patterns.
• Experience handling multi-terabyte structured or time-series datasets.

Key Skills

Education

Any Graduate