Description

Build and maintain scalable data pipelines for AI/ML workloads in Azure.

This role is remote.

Responsibilities

• Design and maintain scalable data ingestion pipelines into Azure Data Lake Storage (ADLS) and Databricks.
• Develop and optimize data transformations using Spark (PySpark) and SQL.
• Implement batch and streaming pipelines to support analytics and AI/ML use cases.
• Integrate data from enterprise source systems (APIs, relational databases, SaaS) into centralized platforms.
• Monitor, troubleshoot, and optimize pipeline performance, reliability, and cost efficiency.

Required Skills

• 7+ years of professional experience in data engineering.
• Strong hands-on experience with Azure data services, including ADLS, Azure Data Factory, and Synapse Pipelines.
• Deep experience with Databricks, including Spark and PySpark development.
• Proficiency in Python and SQL.
• Proven experience building enterprise-scale data pipelines.
• Solid understanding of data architecture, data modeling, and ETL/ELT patterns.
• Experience with pipeline orchestration, automation, and monitoring.

Preferred Skills

• Experience supporting AI/ML or GenAI platforms and workflows.
• Familiarity with CI/CD pipelines for data engineering or ML workflows.
• Experience with real-time/streaming data technologies (e.g., Kafka, Azure Event Hubs).

Key Skills

Education

Any Graduate