Description

You will analyze, design, and develop enterprise data solutions focused on Azure, Databricks, Spark, Python, and SQL.

Responsibilities

• Develop, optimize, and maintain Spark/PySpark data pipelines, addressing performance issues like data skew and shuffle optimization.
• Build and support Delta Lake tables and data models for analytical and operational use cases.
• Use Terraform to provision and manage cloud and Databricks resources, adhering to IaC practices.
• Implement and maintain CI/CD workflows using GitHub and GitHub Actions for pipeline deployment.
• Troubleshoot failures and improve reliability across Databricks jobs, clusters, and data pipelines.

Required Skills

• 5+ years of experience as a Data Engineer.
• Hands-on experience with Azure Databricks, Spark, and Python.
• Strong SQL and database background.
• Experience with Azure cloud data services integration.
• Experience with GitHub and GitHub Actions for version control and CI/CD automation.
• Knowledge of distributed computing concepts (partitions, joins, shuffles).
• Experience with Terraform for cloud infrastructure provisioning.
• Familiarity with data governance or cataloging tools (e.g., Unity Catalog).
• Understanding of data engineering principles and modern engineering practices.

Key Skills

Education

Any Gradute