Description

You will build and maintain data pipelines in Azure Databricks and Delta Lake using PySpark. You own the full lifecycle from ingestion to transformation, ensuring accuracy, security, and compliance. You will optimize existing scripts and notebooks while collaborating with data architects to align models with project goals.

Responsibilities

• Develop and manage data pipelines in Azure Databricks and Delta Lake using PySpark.
• Ingest data from diverse sources and perform transformations to ensure data consistency.
• Implement and maintain CI/CD workflows using Azure DevOps for stories, bugs, and issues.
• Troubleshoot data-related issues through analysis and optimize pipelines for performance.
• Document technical specifications and best practices while collaborating with cross-functional teams.

Required Skills

• 9+ years of software engineering experience.
• Strong proficiency in PySpark for data processing and transformation.
• Hands-on experience with Azure Databricks and Delta Lake.
• Experience implementing CI/CD pipelines using Azure DevOps.
• Ability to elicit data requirements and architect effective data solutions.
• Experience working in Agile development environments and sprint delivery.
• Understanding of data security and compliance standards.
• Bachelor's degree in any discipline.

Preferred Skills

• Experience harmonizing data models with broader project objectives.

Key Skills

Education

Any Gradute