Description

Key Responsibilities

• Architecting and implementing complex ETL/ELT processes using PySpark to handle high-volume streaming and batch data processing workloads.
• Optimizing existing Databricks notebooks and clusters for performance, cost-efficiency, and fault tolerance within the Azure cloud environment.
• Collaborating closely with cross-functional data science and analytics teams to integrate data models and ensure high data quality standards across the enterprise.
• Developing automated deployment pipelines and CI/CD workflows to streamline the delivery of data engineering solutions and production-ready code.
• Managing Azure Data Lake Storage, SQL Data Warehouse, and related integration services to support end-to-end data platform architecture.
• Mentoring junior engineers on coding best practices, debugging strategies, and the adoption of modern cloud-native big data technologies.

Requirements

• Experience: A minimum of 6 to 15 years of hands-on experience in data engineering, with at least 4 years specifically dedicated to Azure Databricks and PySpark development.
• Technical Skills: Advanced proficiency in Python and Spark SQL is mandatory, along with deep expertise in performance tuning, partition strategies, and managing Delta Lake tables.
• Cloud Infrastructure: Comprehensive understanding of Azure services, including Azure Data Factory, Azure Data Lake Gen2, Azure Synapse, and Key Vault integration.
• Analytical Thinking: Strong ability to solve complex data challenges, optimize query performance, and translate ambiguous business requirements into robust technical specifications.
• Soft Skills: Exceptional communication skills are required, as this role involves frequent interaction with stakeholders and participation in face-to-face technical interviews.
• Education & Certification: A degree in Computer Science or a related quantitative field is preferred, with active certifications in Azure Data Engineering (DP-203) being a significant advantage

Key Skills

Education

Bachelor's degree