Description

Build end-to-end data capabilities and maintain optimal pipeline architectures within the Azure ecosystem.

This role is on-site.

Responsibilities

• Build and maintain infrastructure for extracting, transforming, and loading data from diverse sources.
• Design and implement reusable frameworks using Apache Spark.
• Collaborate with product teams to define data migration and integration strategies.
• Create proof-of-concepts to demonstrate the viability of proposed technical solutions.
• Integrate source systems with information management solutions to automate migration processes.

Required Skills

• 5+ years of experience in data engineering and application development.
• Hands-on experience with Azure Data Factory, Azure Databricks, Azure Data Lake Storage (ADLS Gen2), and Azure SQL.
• Proficiency in Python programming for developing data pipelines.
• Strong experience implementing large-scale data processing pipelines and migrations.
• Expertise in Apache Spark, specifically PySpark.
• Solid foundation in data structures, algorithms, and design patterns.
• Bachelor’s degree in Computer Science, Information Systems, or equivalent experience.

Preferred Skills

• Microsoft Certified Azure Data Engineer or Azure Solutions Architect.
• Databricks Certified Associate Developer for Apache Spark 2.4/3.0.

Key Skills

Education

Any Graduate