Description

Overview:

• We are seeking a highly technical and strategic Content Automation Specialist to transform how content quality is validated across Microsoft 365 and Copilot experiences.
• This role focuses on designing scalable evaluation systems, automated testing workflows, and AI-assisted quality frameworks that increase efficiency while maintaining human-centered quality standards. The ideal candidate understands where automation can replace manual review, where human evaluation remains critical, and how to build systems that continuously improve content quality at scale.
• This individual will partner with cross-functional teams to define testing strategies, build evaluation frameworks, develop automation opportunities, and shape the future of content quality operations.

Responsibilities:
Quality Systems Strategy

• Design scalable testing and evaluation frameworks for content, templates, prompt templates, and AI-generated experiences.
• Define quality measurement approaches and establish testing standards across content ecosystems.
• Create recommendations for balancing automation, AI evaluation, and human review.
  Automation & Tooling.
• Develop and implement automated testing workflows and validation systems.
• Build tools and processes that reduce manual effort while maintaining content quality.
• Identify opportunities to leverage AI for defect detection, content validation, and quality assurance.

Human Evaluation & AI Quality

• Design human-in-the-loop evaluation frameworks for assessing AI outputs.
• Establish quality rubrics, benchmarking criteria, and testing methodologies.
• Determine where human judgment adds measurable value and where automation can scale operations.

Data & Optimization

• Analyze testing metrics and operational data to identify improvement opportunities.
• Develop recommendations that improve quality, efficiency, and testing coverage.
• Measure the effectiveness of testing systems and continuously optimize workflows.

Cross-Functional Leadership.

• Partner with cross-functional teams on automation recommendations and evaluations of quality risks.
• Drive adoption of testing standards and scalable evaluation practices.
• Influence long-term strategy for content quality and AI evaluation programs.

Required Qualifications:

• Bachelor’s degree in computer science, Engineering, Data Science, Information Systems, AI, or related field.
• 2-4 years of experience in test automation, quality engineering, AI evaluation, or systems development.
• Strong experience with Python and automation tooling.
• Experience with Copilot, VSCode, and other AI tooling and infrastructure.
• Experience building testing frameworks, quality systems, or workflow automation solutions.
• Experience interpreting data and translating findings into strategic recommendations.
• Strong technical problem-solving and systems-thinking skills.

Top 3 Must-Have HARD Skills & years of experience for each:

• 2-4 years in content testing and automation work (including experience with Python and automation tooling).
• Demonstrated experience translating findings into recommendations, actions, next steps, and milestones - 2 years minimum.
• Demonstrated experience with AI tooling, preferable Copilot and VSCode, but other tooling experience is acceptable so long as transferable - 1 year (min); 2 years (preferred)

Preferred Qualifications:

• Experience evaluating generative AI systems or LLM-powered products.
• Experience designing human evaluation programs or human-in-the-loop workflows.
• Familiarity with content quality measurement and content operations.
• Experience working with machine learning, AI quality, or model evaluation frameworks.
• Understanding of Microsoft 365, Copilot, or productivity software ecosystems.
• Experience building internal tools, dashboards, or quality monitoring systems

Key Skills

Education

Any Graduate