Description

You will build and maintain the data infrastructure and pipelines required for generative AI and analytics applications.

Responsibilities

• Design, construct, and maintain scalable data pipelines and management systems.
• Implement ETL processes to move data from various sources into data warehouses.
• Perform data modeling and schema design to meet application requirements.
• Optimize existing data infrastructure and automate processes to ensure reliability.
• Collaborate with data scientists to implement RAG, chunking, and embedding strategies.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Python.
• Hands-on experience with REST APIs, FastAPI, and Graph APIs.
• Experience with SQL and SQLAlchemy.
• Working knowledge of Azure services including DataLake, Azure SQL, Function App, and Azure Cognitive Search.
• Practical understanding of chunking, embeddings, vectorization, indexing, prompting, and RAG.
• Experience with DevOps practices, including managing code repositories and pull requests.

Preferred Skills

• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree