Description

Design, implement, and optimize large-scale data pipelines within an Azure environment.

This role is hybrid.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Integrate data from multiple sources including Azure Blob Storage, ADLS, and relational/non-relational systems.
• Optimize Databricks workloads for cost efficiency, performance, and reliability.
• Manage data security, access controls, and governance using Unity Catalog.
• Deploy Databricks jobs, notebooks, and configurations using Databricks Asset Bundles and GitLab.

Required Skills

• 8+ years of experience in data engineering roles.
• Expertise in Databricks, including Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, and Delta Live Pipelines.
• Proficiency in Spark and PySpark for big data processing.
• Strong SQL skills for data transformation.
• Experience with Azure Cloud Services and Azure Blob Storage/ADLS.
• Hands-on experience with Databricks Asset Bundles and GitLab for version control and deployment.
• Solid understanding of relational databases.

Preferred Skills

• Familiarity with Spark Streaming and real-time data solutions.
• Azure Data Engineer Associate or Databricks Certified Data Engineer Associate certification.

Key Skills

Education

Any Graduate