Description

You will own the design, development, and maintenance of data pipelines using Databricks and Spark within an Azure environment.

Responsibilities

• Design and maintain scalable data pipelines using Databricks, Spark, and cloud technologies.
• Optimize pipeline performance, scalability, and reliability.
• Implement ETL processes, data warehouse solutions, and data validation methods.
• Troubleshoot data-related issues and provide root cause analysis.
• Collaborate with data scientists and analysts to meet data requirements.

Required Skills

• 5+ years of experience in data engineering focusing on Databricks and Spark.
• Strong command of SQL and proficiency in PySpark.
• Deep understanding of distributed computing and Spark for data processing.
• Experience with Azure Databricks Medallion Architecture, DLT, and Iceberg.
• Knowledge of Azure cloud platform, services, and Azure Data Factory.
• Experience with data warehouse and data lake architectures.
• Proficiency in Python and ETL processes.
• Bachelor's Degree in Computer Science, Engineering, or a related field.

Key Skills

Education

Bachelor's Degree