Description

You will design and build scalable data pipelines using Azure Databricks and Delta Lake.

This role is remote.

Responsibilities

• Perform data transformation, cleansing, and enrichment using PySpark and Python.
• Implement and optimize Big Data solutions within the Azure cloud ecosystem.
• Develop and manage ETL workflows with high performance and reliability.
• Write optimized SQL queries for analytics and reporting over large datasets.
• Leverage Azure DevOps for CI/CD, story tracking, and deployment automation.

Required Skills

• Strong proficiency in Azure Databricks for building and managing data pipelines.
• Hands-on experience with PySpark and Python for data transformation and analytics.
• Expertise in the Azure ecosystem, including Data Lake, Data Factory, Synapse, and Key Vault.
• Solid understanding of Big Data concepts and data lake architecture.
• Experience with Delta Lake, CI/CD pipelines using Azure DevOps, and Git version control.
• Proficiency in SQL for complex querying and data manipulation.
• Knowledge of data integration, ETL processes, and data governance principles.
• 5+ years of professional experience.

Preferred Skills

• Familiarity with Agile methodologies and sprint-based delivery.
• Strong analytical, debugging, and communication skills.

Key Skills

Education

Any Graduate