Description

You will build and maintain data pipelines supporting AI/ML workloads.

Responsibilities

• Manage and monitor end-to-end data pipelines and jobs.
• Maintain and manage Retrieval Augmented Generation (RAG) indexes.
• Integrate and onboard new potential data sources into the ecosystem.

Required Skills

• 5+ years of experience in Data Engineering or a related capacity.
• Strong proficiency in Python and PySpark.
• Expertise in SQL and data warehousing concepts, including dimensional data modelling.
• Hands-on experience with Azure DataBricks and the Azure cloud data platform.
• Familiarity with Delta Lake architecture and the Medallion pattern.
• Experience leveraging Azure services like Azure Data Factory, Synapse Analytics, and Event Hubs.
• Proficiency with CI/CD pipelines and DataOps practices.
• Experience consuming and building services using Rest APIs.

Key Skills

Education

Any Graduate