Description

You will lead data engineering efforts focusing on Data Vault implementation within the Databricks environment.

Responsibilities

• Design and implement Data Vault models using Databricks.
• Develop and optimize distributed data processing jobs using PySpark and Spark SQL.
• Own the design and execution of data pipelines, including partitioning strategies and performance tuning.
• Manage data orchestration using tools like Airflow or Azure Data Factory.
• Ensure data pipelines adhere to governance, security, and compliance frameworks.

Required Skills

• 8+ years in data engineering experience.
• 3+ years experience in a Data Vault Lead role using Databricks.
• Expert proficiency in PySpark and Spark SQL.
• Hands-on experience with cloud data platforms, specifically Azure Data Lake.
• Experience with orchestration tools (Airflow, Azure Data Factory).
• Familiarity with CI/CD tools, including Git, Jenkins, and Terraform.
• Solid understanding of data modeling and partitioning strategies.

Key Skills

Education

Any Graduate