Description

Lead the architecture and implementation of end-to-end cloud-native data solutions on Azure, migrating legacy systems to modern PySpark-based workflows.

This role is on-site.

Responsibilities

• Architect and implement modern data solutions from ingestion to consumption for BI and analytics.
• Migrate legacy logic from on-premise layers including SSIS, stored procedures, Informatica, and Vertica into PySpark.
• Build orchestration layers using Databricks Workflows.
• Develop CI/CD pipelines for Databricks within Azure DevOps.
• Collaborate with Data Engineering, Data Management, and BI teams in a complex development environment.

Required Skills

• 8+ years of experience in data engineering, with at least 4 years focused on cloud-native end-to-end data solutions.
• Extensive hands-on experience with PySpark and Apache Spark.
• Deep expertise in Azure Databricks and Azure Data Factory (ADF).
• Strong understanding of columnar database architectures and cloud-native databases.
• Experience with ETL/ELT techniques and processing substrates like Kafka.
• Proficiency in migrating data from SSIS, Vertica, Apache Hudi, and file systems.
• Knowledge of Snowflake and Microsoft Azure SQL Data Warehouse.
• Any Graduate degree.

Preferred Skills

• Implementation experience with Apache Hudi.

Key Skills

Education

Any Graduate