Description

Lead the design and deployment of large-scale data solutions on Databricks. This role is hybrid.

Responsibilities

• Build batch and streaming pipelines using PySpark, Delta Lake, and Structured Streaming.
• Implement the Medallion architecture (Bronze, Silver, Gold) for clean, curated data.
• Integrate Databricks with AWS or Azure services like S3, Glue, ADF, and Event Hub.
• Drive data governance through Unity Catalog and Purview.
• Optimize performance, manage DBU costs, and lead a team of data engineers.

Required Skills

• 15+ years in data engineering experience.
• 7+ years experience specifically with Databricks and Spark.
• Strong expertise in PySpark.
• Expertise in Delta Lake.
• Proficiency with CI/CD practices.
• Experience with Spark.
• Demonstrated leadership, communication, and problem-solving skills.

Preferred Skills

• Experience with Azure or AWS cloud services.
• Background in data governance frameworks.

Key Skills

Education

Any Graduate