Description

You will build and maintain data pipelines in Azure Databricks and Delta Lake, owning the lifecycle from ingestion to transformation.

Responsibilities

• Develop and manage data pipelines using PySpark in Azure Databricks and Delta Lake.
• Ingest data from diverse sources and perform transformations to ensure consistency, accuracy, and compliance.
• Implement and maintain CI/CD workflows using Azure DevOps for stories, bugs, and issues.
• Troubleshoot data-related issues through analysis and optimize existing scripts and notebooks for performance.
• Document technical specifications and collaborate with data architects to align models with project goals.

Required Skills

• 9+ years of software engineering experience.
• Strong proficiency in PySpark for data processing and transformation.
• Hands-on experience with Azure Databricks and Delta Lake.
• Experience implementing CI/CD pipelines using Azure DevOps.
• Ability to elicit data requirements and architect effective data solutions.
• Experience working in Agile development environments and sprint delivery.
• Understanding of data security and compliance standards.
• Bachelor's degree in any discipline.

Preferred Skills

• Experience harmonizing data models with broader project objectives.

Key Skills

Education

Any Gradute