AI Evaluation Engineer
Posted on Jul 13, 2026 by CV-Library
London, United Kingdom
Retail Trade
Immediate Start
£430 - £500 Daily
Contract/Project
Role/Job title :AI Evaluation Engineer
Work Location :London
Type work : Contract
Mode of work : Hybrid
Responsibilities:
* Define and implement end-to-end evaluation strategy for generative AI conversational systems (LLMs, RAG pipelines, agents).
* Establish evaluation metrics
* Test Dataset & Benchmarking
* Benchmark models (GPT variants, open-source LLMs) and prompt strategies
* Implement automated evaluation pipelines
* Conduct human-in-the-loop evaluation for qualitative validation.
* Prompt & Response Quality Optimization
* RAG & Knowledge Grounding Validation
* Safety, Risk & Compliance Testing
Your Profile
* Strong understanding of LLMs, RAG architecture, prompt engineering
* Python – data analysis, evaluation pipelines
* Prompt evaluation tools (PromptTools, DeepEval, etc.)
Experience with:
- Evaluation Framework Design
- Establish Evaluation metrics & NLP quality assessment
- Test Dataset and benchmarking
- LLM output evaluation and scoring
- Safety, risk and compliance testing
- Python, data analysis, experimentation frameworks
- Tooling and automation
Familiarity with:
- Responsible AI principles
- Customer support / retail conversational flows
- Analytical mindset with ability to translate model behaviour into actionable improvements
Work Location :London
Type work : Contract
Mode of work : Hybrid
Responsibilities:
* Define and implement end-to-end evaluation strategy for generative AI conversational systems (LLMs, RAG pipelines, agents).
* Establish evaluation metrics
* Test Dataset & Benchmarking
* Benchmark models (GPT variants, open-source LLMs) and prompt strategies
* Implement automated evaluation pipelines
* Conduct human-in-the-loop evaluation for qualitative validation.
* Prompt & Response Quality Optimization
* RAG & Knowledge Grounding Validation
* Safety, Risk & Compliance Testing
Your Profile
* Strong understanding of LLMs, RAG architecture, prompt engineering
* Python – data analysis, evaluation pipelines
* Prompt evaluation tools (PromptTools, DeepEval, etc.)
Experience with:
- Evaluation Framework Design
- Establish Evaluation metrics & NLP quality assessment
- Test Dataset and benchmarking
- LLM output evaluation and scoring
- Safety, risk and compliance testing
- Python, data analysis, experimentation frameworks
- Tooling and automation
Familiarity with:
- Responsible AI principles
- Customer support / retail conversational flows
- Analytical mindset with ability to translate model behaviour into actionable improvements
Reference: 225366111
https://jobs.careeraddict.com/post/113553077
AI Evaluation Engineer
Posted on Jul 13, 2026 by CV-Library
London, United Kingdom
Retail Trade
Immediate Start
£430 - £500 Daily
Contract/Project
Role/Job title :AI Evaluation Engineer
Work Location :London
Type work : Contract
Mode of work : Hybrid
Responsibilities:
* Define and implement end-to-end evaluation strategy for generative AI conversational systems (LLMs, RAG pipelines, agents).
* Establish evaluation metrics
* Test Dataset & Benchmarking
* Benchmark models (GPT variants, open-source LLMs) and prompt strategies
* Implement automated evaluation pipelines
* Conduct human-in-the-loop evaluation for qualitative validation.
* Prompt & Response Quality Optimization
* RAG & Knowledge Grounding Validation
* Safety, Risk & Compliance Testing
Your Profile
* Strong understanding of LLMs, RAG architecture, prompt engineering
* Python – data analysis, evaluation pipelines
* Prompt evaluation tools (PromptTools, DeepEval, etc.)
Experience with:
- Evaluation Framework Design
- Establish Evaluation metrics & NLP quality assessment
- Test Dataset and benchmarking
- LLM output evaluation and scoring
- Safety, risk and compliance testing
- Python, data analysis, experimentation frameworks
- Tooling and automation
Familiarity with:
- Responsible AI principles
- Customer support / retail conversational flows
- Analytical mindset with ability to translate model behaviour into actionable improvements
Work Location :London
Type work : Contract
Mode of work : Hybrid
Responsibilities:
* Define and implement end-to-end evaluation strategy for generative AI conversational systems (LLMs, RAG pipelines, agents).
* Establish evaluation metrics
* Test Dataset & Benchmarking
* Benchmark models (GPT variants, open-source LLMs) and prompt strategies
* Implement automated evaluation pipelines
* Conduct human-in-the-loop evaluation for qualitative validation.
* Prompt & Response Quality Optimization
* RAG & Knowledge Grounding Validation
* Safety, Risk & Compliance Testing
Your Profile
* Strong understanding of LLMs, RAG architecture, prompt engineering
* Python – data analysis, evaluation pipelines
* Prompt evaluation tools (PromptTools, DeepEval, etc.)
Experience with:
- Evaluation Framework Design
- Establish Evaluation metrics & NLP quality assessment
- Test Dataset and benchmarking
- LLM output evaluation and scoring
- Safety, risk and compliance testing
- Python, data analysis, experimentation frameworks
- Tooling and automation
Familiarity with:
- Responsible AI principles
- Customer support / retail conversational flows
- Analytical mindset with ability to translate model behaviour into actionable improvements
Reference: 225366111
Share this job:
Alert me to jobs like this:
Amplify your job search:
Expert career advice
Increase interview chances with our downloads and specialist services.
Visit Blog