Description

You will design, develop, and maintain scalable data pipelines for analytics, reporting, and ML use cases.

This role is remote.

Responsibilities

• Build and optimize ETL/ELT pipelines using Azure Databricks (PySpark) and Azure Data Factory.
• Ingest and process structured and unstructured data from diverse sources.
• Implement data models and ensure data quality, validation, and governance standards.
• Integrate pipelines with downstream systems like Power BI and data science models.

Required Skills

• 10+ years of strong experience with Microsoft Azure.
• Proficiency in Python for data engineering tasks.
• Hands-on experience with PySpark and Spark SQL.
• Experience with relational databases (Azure SQL, SQL Server, PostgreSQL).
• Experience with NoSQL or big data stores like Delta Lake or Cosmos DB.
• Strong SQL skills for data transformation and optimization.
• Experience implementing CI/CD pipelines, preferably using Azure DevOps.
• Familiarity with version control using Git.

Preferred Skills

• Understanding of data warehousing and lakehouse architecture.
• Experience with Terraform for infrastructure as code.

Key Skills

Education

Any Graduate