Design and implement large-scale batch and real-time data pipelines within the Azure cloud platform.
Responsibilities
Build highly performant data ingestion pipelines from multiple sources using Azure Databricks and Azure Data Factory.
Develop scalable, reusable frameworks for dataset ingestion and end-to-end data integration.
Lead the design of ETL, data integration, and data migration processes to ensure data quality and consistency.
Partner with architects, engineers, and stakeholders to deploy enterprise-grade data platforms.
Optimize and tune code performance within the Databricks environment.
Required Skills
15+ years of professional experience in data engineering.
Expertise in Azure Databricks, Spark SQL, and PySpark.
Strong proficiency in Python scripting.
Deep experience with Azure cloud components including Azure Data Factory, ADLS, Azure SQL DB, Azure Data Lake, Azure Data Catalogue, LogicApps, and FunctionApps.
Hands-on experience with Azure Storage services and data warehousing.
Solid understanding of SQL, T-SQL, and/or PL/SQL.
Proven experience working in Agile environments using Jira.
Experience implementing event-based and streaming technologies for data processing.
Preferred Skills
Direct experience handling large-scale data ingestion projects specifically within Azure environments.