Description

Design and optimize large-scale data pipelines on AWS Databricks, ensuring scalability and reliability for complex datasets.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Optimize Databricks workloads for cost and performance, applying Spark tuning and shuffle minimization.
• Implement data governance and security policies using Unity Catalog and IAM.
• Deploy jobs and notebooks via Databricks Asset Bundles with CI/CD pipelines.
• Partner with stakeholders to deliver tailored data solutions for analytics and BI.

Required Skills

• 10+ years of experience in data engineering with strong SQL and Python skills.
• Deep expertise in Databricks, including Delta Lake, Unity Catalog, and Lakehouse Architecture.
• Proven ability to implement robust PySpark solutions for batch-heavy pipelines.
• Knowledge of Medallion Architecture (Bronze/Silver/Gold) and table triggers.
• Hands-on experience with AWS services: S3, IAM, Glue Catalog, and networking.
• Experience with Git-based development workflows and CI/CD tools.
• Ability to handle and process tens of millions of records efficiently.
• Understanding of Databricks Workflows and orchestration.

Preferred Skills

• Experience with Spark Structured Streaming for real-time or near real-time solutions.
• Familiarity with Databricks Runtime configurations and advanced features.
• Exposure to GitLab pipelines or similar CI/CD systems.

Key Skills

Education

Any Graduate