Description

You will engineer and optimize large-scale data processing pipelines, focusing on Databricks, Spark, and Delta Lake architectures.

This role is remote.

Responsibilities

• Engineer and optimize data pipelines using Spark, PySpark, and SQL for distributed processing.
• Implement data asset modeling, lifecycle management, and optimization strategies.
• Manage data governance and implement lakehouse patterns using Delta Lake.
• Tune cluster performance, debug complex issues, and implement observability solutions including logging and metrics.
• Maintain and evolve CI/CD processes using Git.

Required Skills

• 8+ years of experience in Data Engineering.
• 3-5 years of experience with Databricks.
• Expertise in Spark, PySpark, and SQL.
• Strong experience with Data Build Tool (DBT).
• Deep knowledge of Delta Lake, governance, and lakehouse concepts.
• Cloud experience, preferably with Azure.
• Proficiency in Git and mature CI/CD practices.

Key Skills

Education

Any Graduate