Description

You will design, develop, and maintain scalable data pipelines using Databricks, Spark, and Azure cloud technologies.

This role is on-site.

Responsibilities

• Design and maintain scalable data pipelines using Databricks, Spark, and cloud technologies.
• Optimize pipeline performance, scalability, and reliability.
• Implement ETL processes, data warehouse solutions, and data validation methods.
• Troubleshoot data-related issues and provide root cause analysis.
• Collaborate with data scientists and analysts to meet data requirements.

Required Skills

• 5+ years of experience in data engineering focusing on Databricks and Spark.
• Strong command of SQL and proficiency in PySpark.
• Deep understanding of distributed computing and Spark for data processing.
• Experience with Azure Databricks Medallion Architecture, DLT, and Iceberg.
• Knowledge of Azure cloud platform, services, and Azure Data Factory.
• Experience with data warehouse and data lake architectures.
• Proficiency in Python and ETL processes.

Preferred Skills

• Familiarity with Kafka, Hadoop, and Hive ecosystems.
• Experience implementing CI/CD pipelines for data engineering workflows.

Key Skills

Education

Bachelor's Degree