Description

Lead the design, development, and optimization of scalable data pipelines using Azure Databricks.

This role is on-site.

Responsibilities

• Design and build complex data pipelines leveraging Azure Databricks for large-scale processing.
• Write clean, maintainable PySpark and Python code for data ingestion, transformation, and analytics.
• Implement and maintain ETL workflows, integrating Databricks with other Azure data services.
• Monitor and optimize workflows for scalability, cost-effectiveness, and operational efficiency.
• Mentor junior team members on best practices and conduct code reviews.

Required Skills

• 7+ years of experience in data engineering and scalable solution design.
• Expert-level proficiency in Azure Databricks.
• Strong command of PySpark and Python for data manipulation.
• Proven track record of implementing and maintaining ETL workflows.
• Experience integrating Databricks with various Azure data services.
• Ability to troubleshoot pipeline issues and ensure high availability.
• Experience collaborating with cross-functional teams to align technical solutions with business objectives.

Preferred Skills

• Experience documenting data processes and architectures for compliance and knowledge sharing.
• Familiarity with agile ceremonies and design discussions.

Key Skills

Education

Any Graduate