Description

You will design and build scalable data pipelines using Azure Databricks and Delta Lake within the Azure cloud environment.

Responsibilities

• Develop and optimize ETL workflows for high-performance data transformation and cleansing using PySpark and Python.
• Write and tune complex SQL queries for analytics and reporting over large datasets.
• Implement CI/CD pipelines and deployment automation using Azure DevOps and Git.
• Integrate data sources across the Azure ecosystem, including Data Lake, Data Factory, and Synapse.
• Collaborate with cross-functional teams in Agile/Scrum projects to deliver sprint goals and ensure data accuracy.

Required Skills

• 5+ years of experience with Azure Databricks for building and managing data pipelines.
• Strong proficiency in PySpark and Python for data transformation and analytics.
• Hands-on expertise in the Azure ecosystem: Data Lake, Data Factory, Synapse, Key Vault.
• Experience with Delta Lake architecture and Big Data concepts.
• Proficiency in SQL for complex querying and data manipulation.
• Experience with CI/CD pipelines using Azure DevOps and version control (Git).
• B.Tech/B.E. or equivalent degree.

Preferred Skills

• Knowledge of data governance principles and data integration best practices.
• Familiarity with Agile methodologies and sprint-based delivery.

Key Skills

Education

Any Graduate