Description

Key Responsibilities
•    Agent Development: Build and orchestrate autonomous AI agents with multi-step reasoning, tool usage, and workflow chaining using frameworks like LangChain, CrewAI, AutoGen, Semantic Kernel, or LlamaIndex.
•    LLM Integration & Optimization: Deploy, fine-tune, and serve open-source LLMs (e.g., Llama 3) using Databricks Model Serving; optimize latency, throughput, and cost.
•    RAG & Knowledge Systems: Design advanced RAG pipelines leveraging vector search, embeddings, semantic ranking, and enterprise data sources (structured + unstructured).
•    Context Engineering: Develop prompt strategies, memory frameworks, and metadata tagging to improve contextual accuracy and response quality.
•    UI & Experience Design: Build intuitive AI-driven applications using Databricks Apps (Streamlit/Dash) or modern web frameworks to enable business consumption.
•    Data Engineering for AI: Build reliable data pipelines (batch & streaming) supporting training, inference, and feature generation using Delta Lake.
•    Security & Governance: Implement enterprise-grade controls using Unity Catalog (row/column-level security, lineage, auditability) aligned with compliance standards.
•    LLM Guardrails & Responsible AI: Implement guardrails (e.g., NeMo Guardrails) for prompt injection prevention, hallucination mitigation, and safe output handling.
•    MLOps & AIOps: Establish CI/CD pipelines for AI models and agents, including versioning, monitoring, drift detection, observability, and incident response.
•    Performance & Cost Optimization: Optimize model performance, GPU/compute usage, and inference cost efficiency across environments.
•    Testing & Evaluation
•    Collaboration & Stakeholder Engagement
•    Documentation & Knowledge Transfer

Required Skills and Qualifications
Databricks & Lakehouse
•    Strong experience with Unity Catalog, Delta Lake, Vector Search, Databricks Workflows, and Model Serving
•    Hands-on with Lakehouse architecture patterns
LLMs & Generative AI
•    Experience with open-source LLMs (Llama, Mistral, etc.), prompting techniques, and fine-tuning approaches
•    Strong knowledge of RAG architectures and embedding strategies
AI Engineering & Frameworks
•    Expertise in LangChain, LlamaIndex, Semantic Kernel, AutoGen, or equivalent
•    Experience building agentic workflows and multi-agent systems
Programming
•    Advanced Python proficiency (APIs, web apps, orchestration, data processing)
•    Familiarity with REST APIs and microservices architecture
MLOps & Monitoring
•    Experience with MLflow, CI/CD pipelines, model lifecycle management, and observability tools
•    Knowledge of drift detection and model performance monitoring
Data Engineering Foundations
•    Experience with Spark, SQL, and large-scale data processing
•    Familiarity with streaming frameworks (Kafka, Structured Streaming)
Security & Governance
•    Expertise in AI security risks (prompt injection, jailbreaks, data leakage)
•    Experience implementing governance frameworks and compliance controls

Key Skills

Education

Bachelor's degree