LayaEval logo LayaEval
⌘K
GitHub ↗
Open source · MIT · Python

A quiet check for every
answer your AI gives.

LayaEval sits around your agent. It watches what goes in and what comes out, and tells you — in milliseconds — if the answer is true, safe and complete.

$pip install layaeval
Browse 42 checks
Travel agentguarded by LayaEvalBook me a Paris hotel for Mar 4Message your agent…BEHIND THE SCENES1.8 ms check1. Read the messageAgent figures out what you want2. Use a toolsearch_hotels(city=Paris)3. Write a draft“Booked Hotel Lumière for Mar 4 ✓”4. LayaEval checks itAll 3 checks pass5. Send to userSafe to deliver

How it works

01

Plug in

Install with pip. Works next to any agent or RAG app — LangChain, LlamaIndex, CrewAI or your own code.

02

Pick checks

Choose from 42 ready checks. Each one asks a single plain question, like “did it make anything up?”.

03

Get a verdict

Every check returns pass or fail, a score and a confidence. Fast enough to run on every request.

Quickstart

Two checks, ten lines.

No account, no server. Everything runs on your own machine. Swap in any of the 42 checks below — they all work the same way.

$pip install layaeval
quickstart.py
1from layaeval import Groundedness, ToolSelectionCorrectness
2

3# 1. Did the answer stick to the sources?
4g = Groundedness(threshold=0.7)
5print(g(context="Paris is the capital of France.",
6        response="Paris is the capital of France.").passed)
7

8# 2. Did the agent pick the right tool?
9t = ToolSelectionCorrectness(threshold=0.7)
10print(t(task="Find a hotel in Paris",
11        available_tools=["search_hotels", "send_email"],
12        selected_tool="search_hotels").passed)
Tap any card to open it

RAG

Finding

Did search bring back the right pages?

3
RAG-001

Retrieval Relevance

Did we fetch documents that match the question?

RAG-002

Document Relevance

Is this one document useful for the question?

RAG-005

Retrieval Completeness

Did search find all the big pieces of evidence?

RAG

Enough to answer

Is there enough in the pages to answer at all?

4
RAG-003

Context Sufficiency

Do the sources hold enough to answer without guessing?

RAG-004

Context Coverage

How much of the needed info is in the sources?

RAG-012

Answerability

Can this question be answered from these sources at all?

RAG-013

Abstention Correctness

When sources are empty, did it honestly say “I don't know”?

RAG

Staying truthful

Does every line come from the pages, with nothing made up?

7
RAG-006

Groundedness

Is each sentence in the answer backed by the sources?

RAG-007

Strict Groundedness

A yes or no: is every claim proven? One gap fails it.

RAG-008

Hallucination Risk

Did the answer invent facts that aren't in the sources?

RAG-014

Context Utilization

Did the answer use the useful facts it was given?

RAG-018

Claim Support

Is this single claim backed by the evidence?

RAG-019

Contradiction Detection

Does the answer say the opposite of the sources?

RAG-020

Context Conflict

Do two fetched sources disagree with each other?

RAG

Answer quality

Is the reply on topic, complete and correct?

3
RAG-009

Response Relevance

Does the answer actually address what was asked?

RAG-010

Response Completeness

Did the answer cover every part of the question?

RAG-011

Factual Correctness

Does the answer match the known correct answer?

RAG

Citations

Do the little [1] tags point to the right place?

3
RAG-015

Citation Correctness

Does each citation point to a source that says that thing?

RAG-016

Citation Completeness

Does every important claim have a citation?

RAG-017

Citation Relevance

Is the cited source really about this claim?

Agent

Understanding the job

Did the agent get what you wanted, and finish it?

4
AG-001

Intent Resolution

What does the user want — info, an action, or a fix?

AG-002

Task Completion

Did the agent finish the job all the way?

AG-003

Task Adherence

Did the agent follow the instructions and limits?

AG-004

Goal Achievement

Did the agent reach what the user truly wanted?

Agent

Using tools

Right tool, right inputs, results actually used.

6
AG-005

Tool Selection

Which tool is the best one for this step?

AG-006

Tool Selection Correctness

Was the tool the agent picked the right one?

AG-007

Tool Call Accuracy

Was the tool called the right way?

AG-008

Tool Input Accuracy

Are the values sent to the tool correct?

AG-009

Tool Output Use

Did the agent use what the tool gave back?

AG-010

Tool Necessity

Was calling a tool really needed here?

Agent

Taking the path

Sensible order, no repeats, no going in circles.

3
AG-011

Tool Redundancy

Did the agent ask for the same thing twice?

AG-012

Tool Sequence

Were the tool steps done in a sensible order?

AG-013

Navigation Efficiency

Did the agent take the short way, without loops?

Agent

Final result

Is the last message true, complete and correct?

5
AG-014

Agent Groundedness

Is the final answer backed by what the tools returned?

AG-015

Agent Hallucination

Did the agent claim results that never happened?

AG-016

Response Relevance

Does the final message answer the user's request?

AG-017

Answer Completeness

Did the final reply cover every part of the task?

AG-018

Agent Correctness

Is the final result the correct one?

Agent

Staying safe

Follows the rules and can't be tricked.

4
AG-019

Agent Abstention

Did the agent refuse actions it shouldn't or can't do?

AG-020

Policy Compliance

Did the agent follow the house rules and safety limits?

AG-021

Prompt Injection

Is the user trying to trick the assistant into ignoring its rules?

AG-022

Indirect Prompt Injection

Is a hidden command tucked inside a web page or tool result?