Description

Design and implement end-to-end data engineering solutions using the Databricks ecosystem and Azure data services.

Responsibilities

• Execute native Spark migrations to Databricks and implement Medallion architecture.
• Build orchestration layers using Databricks workflows and Azure Data Factory.
• Optimize Databricks performance and cost through Auto Compaction, Z-Ordering, and vacuuming.
• Develop CI/CD pipelines for Databricks using Azure DevOps and GitHub.
• Process near real-time data using Auto Loader and Delta Live Tables (DLT) pipelines.

Required Skills

• 5+ years of experience in data engineering roles.
• Expertise in PySpark, Apache Spark, and Spark SQL.
• Strong proficiency in Python and libraries including Pandas, NumPy, and pymysql.
• Hands-on experience with Azure Data Factory (ADF), Azure Synapse Analytics, and Azure Storage.
• Deep knowledge of Delta Lake security and implementing Massive Parallel Processing layers.
• Experience with Azure Key Vault, SQL Pools, Azure Functions, and Logic Apps.
• Proficiency in SQL, including stored procedures, dynamic SQL, and complex control flow logic.
• Experience managing code deployment through Dev, QA, and Prod environments using Git and CI/CD.
• Ability to extract data via APIs and troubleshoot Python library installations.

Preferred Skills

• Experience with Oracle and managing data through stored procedures via Python.

Key Skills

Education

Any Gradute