Description

Lead the architecture and implementation of advanced analytics solutions using Azure Databricks and Fabric.

This role is on-site.

Responsibilities

• Design and implement end-to-end data pipelines and ETL processes using Azure Databricks and PySpark.
• Integrate Azure Databricks with Azure Data Lake, Synapse, Data Factory, and on-premise systems.
• Lead the development of data lakes and warehousing solutions, ensuring security and governance compliance.
• Mentor data engineers and drive best practices for code quality, scalability, and documentation.
• Collaborate with stakeholders to translate business requirements into effective technical solutions.

Required Skills

• 8+ years of experience in data ingestion and transformation pipelines.
• Deep expertise in Azure Databricks, including cluster management, notebook development, and Delta Lake.
• Proficiency in PySpark and big data technologies like Hadoop and Spark.
• Strong SQL skills and experience with NoSQL databases.
• Hands-on experience with Azure Data Lake Gen2, Azure Synapse, and Azure Data Factory.
• Experience building and optimizing query layers using Databricks SQL.
• Knowledge of CI/CD pipelines, Git, and cloud security best practices.
• Familiarity with Auto Loader and Delta Live Tables for seamless data ingestion.

Preferred Skills

• Experience with Microsoft Fabric for analytics and integration tasks.
• Proficiency in Power BI for dashboarding, reporting, and optimizing reports.
• Knowledge of Spark Streaming for real-time data processing and Azure Resource Manager templates.

Key Skills

Education

Any Graduate