AI Test & Evaluation Engineer (Python / Pytest)

Contract Rate: Up to £600 - £650/day (Inside IR35)

Location: Hybrid (1 day per week in London, Cardiff, or Glasgow)

Clearance: Active SC Clearance required

Role Overview

We are seeking an experienced AI Test & Evaluation Engineer to design, develop, and execute evaluation frameworks for AI platforms, including M365 Copilot agents. This role focuses on building automated test suites and validation pipelines to evaluate AI-specific quality attributes, performance, and reliability within complex technical environments.

Key Responsibilities

• Framework Design & Execution: Lead the end-to-end design, development, planning, and execution of test and evaluation frameworks tailored for AI solutions, including work on the Cyber RIIO tech stack and M365 Copilot agents.

• Automated Testing & Pipelines: Utilize robust Python skills to build scalable automated test suites and validation pipelines using PyTest.

• AI Quality Attribute Analysis: Rigorously test and evaluate AI-specific criteria including accuracy, relevance, faithfulness, hallucination rates, robustness, bias, fairness, security, privacy, and explainability.

• Data Science Project Delivery: Apply proven data science project delivery experience to ensure rigorous testing standards align with broader system architecture and delivery goals.

Key Requirements

• Security Clearance: Must hold valid, active SC Clearance.

• AI Evaluation Expertise: Proven experience designing and executing evaluation frameworks for AI, specifically M365 Copilot agents.

• Technical Skills: Strong Python programming skills combined with hands-on, practical experience using PyTest to build automated testing framework components.

• Analytical Skills: In-depth understanding of AI metrics and failure modes (hallucinations, bias, drift, security vulnerabilities, and model explainability).

• Delivery Track Record: Solid background in data science project delivery.

Location & Working Pattern

This is a hybrid position requiring 1 day per week in the office, with location options in London, Cardiff, or Glasgow. The remainder of the week can be worked remotely.