Description

Key Responsibilities

• Build and operate agentic applications that interpret user intent, manage conversation context, call tools and APIs, and carry out multi-step analysis with appropriate user review.
• Develop the Angular user experience, .NET APIs, and Python agent services, integrating with existing reporting systems, authentication, and data access controls.
• Own the evaluation pipeline: curate representative datasets with domain experts, define scoring rubrics and acceptance criteria, and combine automated checks with human review.
• Evaluate answer correctness, supporting evidence, tool selection and execution, and end-to-end task success, including ambiguous requests, unsupported questions, and failure scenarios.
• Integrate evaluations and regression tests into CI/CD. Version prompts, evaluation datasets, and configuration; compare changes against baselines and enforce quality gates before release.
• Monitor production quality, latency, cost, and failures through traces and user feedback. Turn observed failures into reproducible test cases and measurable improvements.
• Implement reliable execution with timeouts, retries, cancellation, durable state, and recovery, while protecting sensitive data and enforcing authorization at tool and data boundaries.
• Work with product owners and domain experts to validate analytical findings, preserve source traceability, and communicate uncertainty and application limitations clearly.

Required Qualifications

• 8+ years of professional software development experience, including delivery and support of secure, scalable enterprise applications.
• Hands-on experience building agentic applications using large language models, tool calling, structured outputs, prompt and context design, and multi-step workflows.
• Build and operate agentic applications that interpret user intent, manage conversation context, call tools and APIs, and carry out multi-step analysis with appropriate user review.
• Develop the Angular user experience, .NET APIs, and Python agent services, integrating with existing reporting systems, authentication, and data access controls.
• Own the evaluation pipeline: curate representative datasets with domain experts, define scoring rubrics and acceptance criteria, and combine automated checks with human review.
• Evaluate answer correctness, supporting evidence, tool selection and execution, and end-to-end task success, including ambiguous requests, unsupported questions, and failure scenarios.
• Practical experience building evaluation and quality pipelines for AI applications, including test datasets, scoring criteria, regression analysis, and human review. Ability to explain how evaluation results informed a release or quality improvement.
• Strong full stack development skills with Angular, .NET Web APIs, and Python, including integration with existing enterprise services.
• Strong SQL, relational data modeling, and data analysis skills, with attention to numerical accuracy, missing data, and traceability of results.
• Experience with cloud-native application development on Azure, CI/CD, automated unit and integration testing, and end-to-end testing.
• Experience with authentication, authorization, secure API design, observability, and reliable asynchronous processing.
• Strong problem-solving and communication skills, with the ability to work independently and collaborate with product, engineering, and domain experts.

Preferred Qualifications

• Experience with retrieval-augmented generation (RAG), including document ingestion, retrieval, grounding, and retrieval quality evaluation.
• Experience with healthcare analytics, enterprise reporting, or other applications that require careful handling of sensitive data and verifiable analytical results.
• Experience taking AI prototypes into production and improving quality, latency, and operating cost using evaluation and production evidence

Key Skills

Education

Bachelor's degree