Description

You will design and implement scalable data engineering solutions using the Azure Databricks ecosystem.

Responsibilities

• Execute native Spark migrations to Databricks and implement Medallion architecture.
• Build orchestration layers using Databricks workflows and Azure Data Factory.
• Develop CI/CD pipelines for Databricks via Azure DevOps and GitHub.
• Optimize data performance and costs through Auto Compaction, Z-Ordering, and vacuuming.
• Engineer near real-time data processing using Auto Loader and DLT pipelines.

Required Skills

• Extensive experience with PySpark, Apache Spark, and Spark SQL.
• Hands-on expertise with Azure Databricks, Azure Data Factory (ADF), and Azure Synapse Analytics.
• Proficiency in Python libraries including Pandas, NumPy, and pymysql.
• Strong SQL skills, including writing stored procedures, functions, and dynamic SQL.
• Experience managing Azure services: ADLS, Azure Key Vault, Azure Functions, and Logic Apps.
• Ability to implement Massive Parallel Processing (MPP) layers.
• Experience with data extraction via APIs and stored procedures.
• Proven track record in CI/CD pipeline design and deployment across Dev, QA, and Prod environments.
• Knowledge of Oracle and managing data through Spark Pools.

Key Skills

Education

Not specified