Description

Design and implement large-scale batch and real-time data pipelines within the Azure cloud platform.

Responsibilities

• Build highly performant data ingestion pipelines from multiple sources using Azure Databricks and Azure Data Factory.
• Develop scalable, reusable frameworks for dataset ingestion and end-to-end data integration.
• Lead the design of ETL, data integration, and data migration processes to ensure data quality and consistency.
• Partner with architects, engineers, and stakeholders to deploy enterprise-grade data platforms.
• Optimize and tune code performance within the Databricks environment.

Required Skills

• 15+ years of professional experience in data engineering.
• Expertise in Azure Databricks, Spark SQL, and PySpark.
• Strong proficiency in Python scripting.
• Deep experience with Azure cloud components including Azure Data Factory, ADLS, Azure SQL DB, Azure Data Lake, Azure Data Catalogue, LogicApps, and FunctionApps.
• Hands-on experience with Azure Storage services and data warehousing.
• Solid understanding of SQL, T-SQL, and/or PL/SQL.
• Proven experience working in Agile environments using Jira.
• Experience implementing event-based and streaming technologies for data processing.

Preferred Skills

• Direct experience handling large-scale data ingestion projects specifically within Azure environments.

Key Skills

Education

ANY GRADUATE