Description

Key Responsibilities

 

• Harness Engineering: Build and maintain the LLM harnesses that power each AI feature — agent loops, tool/function calling, context construction, memory, retries, and failure handling. Use frameworks like LangChain, LlamaIndex, or LangGraph where they fit; write custom orchestration where they don't.
• Prompt Engineering: Design, iterate, and version prompts as first-class assets. Run structured prompt experiments, measure deltas with eval datasets, and keep prompt libraries clean and reviewable.
• RAG Pipelines: Build and operate Retrieval-Augmented Generation pipelines that extract information from documents and parse it into structured knowledge bases — chunking, indexing, retrieval, reranking, prompt assembly, and response handling — collaborating with the AI Architect to shape and refine the patterns.
• Embeddings & Vector Indexes: Generate embeddings and manage vector indexes (e.g., OpenSearch, Pinecone, pgvector). Tune indexes for retrieval quality and cost.
• Data Parsing & Curation: Build extraction and parsing pipelines for documents, structured records, and customer datasets so they are clean, labeled, and ready for downstream AI work.
• Accuracy Validation: Write and operate accuracy validation scripts. Maintain evaluation datasets and report quality metrics to the Strike Team and the AI Architect.
• Light ML Work: Apply traditional ML where appropriate — classification, clustering, lightweight fine-tuning — to complement LLM-based components.
• Analysis: Do the data analysis that informs AI design decisions — sample inspection, error analysis, prompt iteration.
  
   

KPIs & Success Metrics

 

• AI feature accuracy and quality against eval datasets meets the project bar.
• Prompt iteration velocity with measurable eval deltas on owned features.
• RAG and retrieval quality (relevance, groundedness) for owned pipelines.
• On-time delivery of AI-side Strike Team commitments.
• Eval coverage: golden and regression sets maintained for owned features.
  
   

Key Skills & Experience

 

• Education: Bachelor's or Master's degree in Computer Science, Data Science, Statistics, or a related technical field.
• Experience: 3+ years of AI or data science experience, with hands-on time in LLM-based application development.
• Technical Proficiency:
  
   

– Strong Python skills, including the standard data science stack (pandas, numpy, scikit-learn).

– Hands-on experience building LLM harnesses — agent loops, tool/function calling, structured outputs — against APIs like Anthropic, OpenAI, or Bedrock.

– Strong prompt engineering practice: structured iteration, prompt versioning, and prompt evaluation against datasets.

– Working experience with at least one orchestration framework (LangChain, LlamaIndex, LangGraph) and at least one vector database.

– Comfortable with embeddings, similarity search, and basic retrieval evaluation.

– Working knowledge of classical ML for analysis and lightweight modeling tasks.

– Comfort using AI coding assistants (Claude Code) for daily work.

 

• Engineering Excellence: Able to write clean, tested code that ships to production — not just notebooks. Familiar with Git, code review, and basic CI/CD.
• Analytical Mindset: Strong instinct for data analysis, error inspection, and iterative experimentation.
  
   

Preferred Skills & Experience

 

• Agent Frameworks: Hands-on experience with agentic frameworks (LangGraph, Claude Agent SDK, OpenAI Agents) or custom agent harnesses in production.
• Evals: Experience building structured eval harnesses (golden sets, regression suites, LLM-as-judge patterns).
• Cloud: Hands-on AWS experience (Bedrock, SageMaker, OpenSearch).
• Fine-Tuning: Any experience with model fine-tuning or distillation.
• Guidewire Knowledge: Familiarity with Guidewire products or the insurance domain is a plus.
• Domain Analysis: Prior experience working on document-heavy, regulated, or insurance/finance datasets

Key Skills

Education

Bachelor's or Master's degrees