Description

Lead the design, development, and optimization of large-scale data solutions using Azure Databricks, Apache Spark, and Delta Lake.

This role is on-site.

Responsibilities

• Architect and build large-scale ETL/ELT data pipelines using Databricks (PySpark/Spark SQL).
• Optimize cluster usage, Spark code, and Delta Lake pipelines for high performance and cost efficiency.
• Implement Delta Lake features including ACID transactions, schema evolution, and time travel.
• Lead migration of on-prem or legacy workloads to Azure Databricks and design data models with Unity Catalog governance.
• Build and orchestrate data workflows via ADF, Databricks Workflows, or other tools while integrating with Azure services.

Required Skills

• 10+ years of experience in data engineering.
• 6+ years in data engineering, with at least 3 years on Azure Databricks.
• Strong hands-on expertise with PySpark, Spark SQL, Apache Spark internals, and performance tuning.
• Deep understanding of Delta Lake, data versioning, and Lakehouse architecture.
• Strong SQL skills: complex queries, performance tuning, and analytical functions.
• Experience integrating Databricks with ADF, ADLS Gen2, Azure SQL, Event Hub, and other Azure components.
• Hands-on experience with CI/CD pipelines using Git, GitHub Actions, Azure DevOps, or similar tools.

Preferred Skills

• Solid understanding of distributed computing and data security best practices.
• Ability to design scalable, maintainable, and reusable data frameworks.
• Experience conducting code reviews and mentoring junior engineers.

Key Skills

Education

Any Graduate