Description

You will design, develop, and maintain scalable data pipelines for analytics and machine learning use cases.

This role is remote.

Responsibilities

• Build and optimize ETL/ELT pipelines using Azure Databricks and PySpark.
• Ingest, transform, and process structured and unstructured data from diverse sources.
• Implement data models and ensure data quality, validation, and governance standards.
• Integrate pipelines with downstream systems like Power BI and data science models.

Required Skills

• 10+ years of experience with Microsoft Azure (ADLS Gen2, Azure Data Factory, Azure Synapse Analytics).
• Strong proficiency in Python for data engineering tasks.
• Hands-on experience with PySpark and Spark SQL.
• Experience with relational databases (Azure SQL, SQL Server) and NoSQL stores (Cosmos DB).
• Strong SQL skills for data transformation and optimization.
• Deep understanding of data warehousing and lakehouse architecture.
• Experience implementing CI/CD pipelines, preferably with Azure DevOps.
• Experience with batch and near-real-time data processing patterns.

Key Skills

Education

Any Graduate