Description

You will design, develop, and maintain scalable data pipelines and ETL processes.

Responsibilities

• Implement and optimize Spark jobs, data transformations, and data processing workflows using Databricks notebooks.
• Manage data pipelines and infrastructure using Azure DevOps and CI/CD best practices.
• Ensure data integrity and data quality checks meet zero percent error requirements in production.
• Handle data extraction, transformation, and loading, including error handling and parallelized code.
• Understand and utilize Databricks features like Unity Catalog and Lakehouse flow.

Required Skills

• 5+ years of experience in data engineering or a similar role.
• Proven expertise with Azure Databricks.
• Strong proficiency in Python and SQL.
• Experience with Spark for data processing.
• Knowledge of ETL processes and data pipeline design.
• Familiarity with Data Warehousing concepts.
• Experience automating ETL processes for production optimization.

Key Skills

Education

Bachelor's degree