Description

You design and optimize large-scale data solutions using Azure Databricks and Apache Spark.

This role is on-site.

Responsibilities

• Architect and build large-scale ETL/ELT pipelines using Databricks with PySpark and Spark SQL.
• Optimize cluster usage, Spark code, and Delta Lake pipelines for performance and cost efficiency.
• Implement Delta Lake features including ACID transactions, schema evolution, and time travel.
• Lead migration of on-prem or legacy workloads to Azure Databricks.
• Design data models and ensure data integrity and governance using Unity Catalog.

Required Skills

• 6+ years of data engineering experience, including at least 3 years on Azure Databricks.
• Strong hands-on expertise with PySpark, Spark SQL, Apache Spark internals, and performance tuning.
• Deep understanding of Delta Lake, data versioning, and Lakehouse architecture.
• Strong SQL skills for complex queries, performance tuning, and analytical functions.
• Experience integrating Databricks with ADF, ADLS Gen2, Azure SQL, and Event Hub.
• Hands-on experience with CI/CD pipelines using Git, GitHub Actions, or Azure DevOps.
• Solid understanding of data modeling, distributed computing, and data security best practices.
• Ability to design scalable, maintainable, and reusable data frameworks.

Preferred Skills

• Experience with Azure SQL and Event Hub integration.
• Solid understanding of data security best practices.

Key Skills

Education

Any Graduate