Description

Responsibilities include:

Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, Scala, Spark, and SQL to support enterprise data initiatives.

Build and maintain data ingestion, transformation, and streaming solutions for processing large-volume data in Azure Databricks environments.

Implement and manage Lakehouse architecture, including Delta Lake, Unity Catalog, Medallion Architecture, and Databricks job orchestration.

Monitor, troubleshoot, and enhance the performance of Databricks’ jobs, clusters, and Spark workloads to ensure reliability and efficiency.

Collaborate with cross-functional teams to develop robust data warehouse and data lakehouse solutions that meet business and technical requirements.

Support DevOps pipelines, Azure services, and AKS environments, resolving production issues and ensuring seamless deployment of data engineering solutions.

Required Qualifications:

Bachelor’s degree in computer science, Information Technology, Engineering, or a related technical discipline.

8-10 years of experience in Data Engineering, with at least 8 years of hands-on expertise in Databricks, Scala, Spark, and SQL development.

Strong experience designing and implementing ETL/ELT pipelines, large-scale data transformations, data warehousing, and Lakehouse architectures using Azure Databricks.

Proven expertise with Delta Lake, Unity Catalog, Auto Loader, Medallion Architecture, Spark Structured Streaming, and Databricks job orchestration

Key Skills

Education

Any Graduate