Description

Lead the design and development of enterprise data solutions on Azure and Databricks. You will own the data engineering lifecycle, from architecture and pipeline development to CI/CD automation and infrastructure management.

Responsibilities

• Design and develop enterprise data solutions using Azure, Databricks, Spark, Python, and SQL.
• Build, optimize, and maintain Spark/PySpark data pipelines, resolving data skew, partitioning, caching, and shuffle issues.
• Provision and manage cloud and Databricks resources using Terraform, adhering to Infrastructure as Code practices.
• Implement CI/CD workflows with GitHub Actions for source control, testing, and automated pipeline deployment.
• Troubleshoot cluster and job failures, improving reliability and performance across Azure data services.

Required Skills

• 5+ years of experience as a Data Engineer.
• Hands-on experience with Azure Databricks, Spark, and Python.
• Strong SQL background and database management skills.
• Experience with Delta Live Tables (DLT) or Databricks SQL.
• Familiarity with data governance, lineage, or cataloging tools (e.g., Purview, Unity Catalog).
• Experience with Azure Functions, messaging services, or orchestration tools.
• Proficiency with Terraform for cloud infrastructure provisioning.
• Experience with GitHub and GitHub Actions for version control and CI/CD.
• Strong understanding of distributed computing concepts (partitions, joins, shuffles, cluster behavior).

Preferred Skills

• Experience monitoring and optimizing Databricks clusters or workflows.
• Familiarity with SDLC and modern engineering practices.

Key Skills

Education

Any Gradute