Description

Lead the design, development, and optimization of scalable data pipelines using Azure Databricks. You will own the ETL workflow lifecycle, ensuring high performance, data quality, and reliability for enterprise analytics.

Responsibilities

• Design and build complex data pipelines leveraging Azure Databricks to support large-scale data processing.
• Write clean, maintainable PySpark and Python code for data ingestion, transformation, and analytics.
• Implement and maintain ETL workflows, integrating Azure Databricks with other Azure data services.
• Monitor and optimize data workflows for scalability, cost-effectiveness, and operational efficiency.
• Mentor junior team members on Azure Databricks best practices and conduct code reviews.

Required Skills

• 7+ years of experience in data engineering and scalable data solution design.
• Expert-level proficiency in Azure Databricks.
• Strong command of PySpark and Python for data manipulation.
• Proven track record of implementing and maintaining ETL workflows.
• Experience integrating Databricks with various Azure data services.
• Ability to troubleshoot pipeline issues and ensure high availability.
• Experience collaborating with cross-functional teams to align technical solutions with business objectives.

Preferred Skills

• Experience documenting data processes and architectures for compliance and knowledge sharing.
• Familiarity with agile ceremonies and design discussions.

Key Skills

Education

Any Graduate