Description

You will build and manage data architectures and pipelines specifically designed to support agentic AI systems and complex agent-to-data interactions.

Responsibilities

• Design and develop data pipelines and ELT processes to move data from source systems to analytical platforms.
• Build data architectures including databases and data lakes to support agentic workflows.
• Implement data pipelines that facilitate feedback loops for human-in-the-loop systems.
• Manage vector databases for efficient embedding storage and retrieval.
• Collaborate with data scientists to preprocess data, train models, and integrate AI into applications.
• Optimize data storage and retrieval using effective partitioning schemes.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python and AI/ML frameworks.
• Experience with Spark and Azure Databricks.
• Hands-on experience training and fine-tuning LLMs with structured and unstructured datasets.
• Practical knowledge of vector databases and embedding models.
• Experience with Azure AI services including Azure OpenAI, Azure AI Search, and Azure Machine Learning.
• Familiarity with Azure Blob Storage and Azure Data Lakes.
• Understanding of Graph DB and core machine learning algorithms.
• Experience with GIS spatial data, including geo-location, topology, and correlation.
• Experience with Department of Transportation data domains.
• Ability to implement partition schemes using Spark.

Preferred Skills

• Expertise in integrating with AI agent frameworks.
• Experience developing AI Composite Agentic Solutions for forecasting and what-if analysis.

Key Skills

Education

Bachelor's or Master's degrees