How to Test AI Answers with an LLM Grader in Oracle

A RAG assistant must be tested like any other code, with questions whose answers you know. But AI answers vary in wording, so you cannot compare them with an expected string. Three tests work: check that each answer contains its key fact, check that its citations point to sources it was actually given, and have a second language model grade whether every statement is supported.

This guide builds all three in Oracle AI Database 26ai for the ASK function of a RAG assistant, and shows how a careless grading test can mislead you.

Code for This Guide

The examples are files 07 and 08 in the examples/ch12 folder and files 05 and 06 in the examples/ch28 folder of the Oracle AI code repository on GitHub, each with its output.

These examples come from AI Applications with Oracle Database 26ai and APEX 26.1, a book of 237 tested examples of AI in Oracle Database and APEX.

They test the ASK function and the RAG_LOG table from how to build RAG with PL/SQL in Oracle Database, with two sets of questions whose right answers are known: DOC_QUESTIONS from how to search PDF documents by meaning in Oracle, and EVAL_QUESTIONS from how to choose an embedding model.

Test 1: Does the Answer Contain the Key Fact?

For each of 14 document questions, the test lists a key fact the right answer must contain, such as "20", "7 days", or "30 June 2027". It then counts the answers that contain it, the answers with a citation, and the answers that gave up.

Example:

-- the 14 document questions: does the answer contain the key fact?
with keys (question_id, key_fact) as (
  values (1, '20'), (2, '7 days'), (3, '20 MB'), (4, '180 days'), (5, '3 days'),
         (6, '3 business days'), (7, 'ten months'), (8, '11'), (9, '15 minutes'),
         (10, '100'), (11, '30 June 2027'), (12, '5 GB'), (13, '14 September 2026'),
         (14, '40'))
select count(*) as questions,
       count(case when instr(a.answer, k.key_fact) > 0 then 1 end) as with_key_fact,
       count(case when regexp_like(a.answer, '\[\d') then 1 end) as with_citation,
       count(case when a.answer like 'I could not find%' then 1 end) as not_found
from   doc_questions q
join   keys k on k.question_id = q.question_id
cross  apply (select ask(q.question) as answer from dual) a;

Output:

   QUESTIONS    WITH_KEY_FACT    WITH_CITATION    NOT_FOUND
____________ ________________ ________________ ____________
          14               14               14            0

All 14 answers contain the key fact and cite a source. A key fact is a coarse test, since an answer can contain "20" and still be wrong, but it catches the failures that matter most: a source not retrieved, a model answering from its own knowledge, a prompt change that breaks the form. Run it after every change to the sources, chunking, retrieval, or prompt.

Test 2: Do the Citations Point to Given Sources?

Citations only help if they point to the right sources. RAG_LOG keeps the sources of every answer, so the test can mark which of them the answer cites.

Example:

-- the sources an answer cites, looked up in the sources it was given
with last_answer as (
  select answer, sources
  from   rag_log
  where  question = 'How many custom roles can an Enterprise account have?'
  order  by asked_at desc
  fetch  first 1 row only)
select s.n, s.source,
       case when instr(l.answer, '[' || s.n || ']') > 0 then 'cited' end as in_answer
from   last_answer l,
       json_table(l.sources, '$[*]' columns (n number path '$.n',
                                             source varchar2(60) path '$.source')) s
order  by s.n;

Output:

   N SOURCE                                       IN_ANSWER
____ ____________________________________________ ____________
   1 Atlas CRM 8.4 Administrator Guide, part 4    cited
   2 Atlas CRM 8.4 Administrator Guide, part 2
   3 Atlas CRM 8.4 Administrator Guide, part 5
   4 Article KB-104

Four sources were given, and the answer cited the one that contains the fact. An answer that cites a number it was not given, or cites nothing, was probably not taken from the sources; an application can check for both before it shows an answer.

Test 3: Grade Answers with a Language Model

Key facts test what an answer must contain. Whether it states anything it must not, claims its sources do not support, needs a reader. A language model can be that reader: given a reference and an answer, it judges whether the answer is correct and supported. This is called LLM-as-a-judge.

This example stores the answers to 20 English questions, then has Gemini grade each against the knowledge base article that is the question's right answer, with a JSON schema for the verdict and a reason.

Example:

-- 20 questions: answers from ASK, graded against the right article
create table judged_answers as
select q.question_id, q.question, q.article_id,
       ask(q.question) as answer,
       cast(null as varchar2(10)) as correct,
       cast(null as varchar2(10)) as unsupported,
       cast(null as varchar2(400)) as reason
from   eval_questions q
where  q.lang = 'en' and q.question_id <= 20;

update judged_answers j
set   (correct, unsupported, reason) = (
         select json_value(v, '$.correct'), json_value(v, '$.unsupported'),
                json_value(v, '$.reason')
         from  (select generate(
                  'You grade the answers of a support assistant. Reference article: '
                  || (select body from kb_articles a where a.article_id = j.article_id)
                  || chr(10) || 'Question: ' || j.question
                  || chr(10) || 'Answer: ' || j.answer
                  || chr(10) || 'Is the answer correct according to the reference? '
                  || 'Does it state anything that the reference does not support?',
                  'GEMINI',
                  json('{"generationConfig": {"temperature": 0,
                         "responseMimeType": "application/json",
                         "responseSchema": {"type": "OBJECT",
                           "required": ["correct", "unsupported", "reason"],
                           "properties": {
                             "correct":     {"type": "STRING", "enum": ["yes", "no"]},
                             "unsupported": {"type": "STRING", "enum": ["yes", "no"]},
                             "reason":      {"type": "STRING"}}}}}')) as v
                from   dual));
commit;

select count(*) as answers,
       count(case when correct = 'yes' then 1 end) as correct,
       count(case when unsupported = 'yes' then 1 end) as with_unsupported_claims
from   judged_answers;

Output:

Table JUDGED_ANSWERS created.

20 rows updated.

Commit complete.

   ANSWERS    CORRECT    WITH_UNSUPPORTED_CLAIMS
__________ __________ __________________________
        20         15                          4

Fifteen of 20 correct and 4 with unsupported claims: a worse picture than any earlier test. Reading the reasons, most of it falls apart. The "unsupported" claims came from the other sources the answer was given, such as an administrator guide or an API FAQ, which the judge never saw because it had only the one article. The judge was right about what it was shown; the test was wrong.

Grade Against What the Model Saw

The fix is to grade each answer against the sources the answering model actually received, as RAG_LOG recorded them.

Example:

-- the same answers, checked against the sources ASK gave the model (from RAG_LOG)
alter table judged_answers add (faithful varchar2(10));

update judged_answers j
set    faithful = (
         select json_value(generate(
                  'You check the answers of a support assistant. The assistant was given '
                  || 'these sources: '
                  || (select json_serialize(json_query(r.sources, '$[*].text' with wrapper)
                                            returning clob)
                      from   rag_log r
                      where  r.question = j.question
                      order  by r.asked_at desc
                      fetch  first 1 row only)
                  || chr(10) || 'Answer: ' || j.answer
                  || chr(10)
                  || 'Is every statement of the answer supported by the sources?',
                  'GEMINI',
                  json('{"generationConfig": {"temperature": 0,
                         "responseMimeType": "application/json",
                         "responseSchema": {"type": "OBJECT", "required": ["faithful"],
                           "properties": {"faithful": {"type": "STRING",
                                                       "enum": ["yes", "no"]}}}}}')),
                  '$.faithful')
         from   dual);
commit;

select count(*) as answers,
       count(case when unsupported = 'yes' then 1 end) as unsupported_by_article,
       count(case when faithful = 'no' then 1 end) as unsupported_by_sources
from   judged_answers;

Output:

Table JUDGED_ANSWERS altered.

20 rows updated.

Commit complete.

   ANSWERS    UNSUPPORTED_BY_ARTICLE    UNSUPPORTED_BY_SOURCES
__________ _________________________ _________________________
        20                         4                         2

Two answers remain flagged:

• A fair catch: an answer about rejected two-factor codes advised using backup codes, which the source only implies.
• The judge's own mistake: the refusal "I could not find this in the Atlas knowledge base" was marked unsupported, though it claims nothing.

A judge is a model too. It needs precise instructions, here that a refusal is always supported, and its verdicts need spot checks by a person before anyone acts on them. Used that way, it finds in minutes the few answers among hundreds that a person should read.

When to Run the Tests

Conclusion

Test a RAG assistant with questions whose answers you know. Check that each answer contains its key fact, check that citations point only to sources the model was given, and use a second model as a grader for unsupported claims, always grading against the sources the answering model actually saw. Give the grader precise rules, such as that a refusal is supported, and have a person read what it flags.