Rules first, then AI: thirteen found in full, one in part, one missed
The same fifteen hidden errors as test 1, the same model, the same laptop. We changed how the work is split: fixed rules for the mechanical checks, and the regulation given to the model as text instead of from its memory.
The result
Thirteen of the fifteen planted errors were found in full. One (#12) was found in part: the right documents, the wrong reason. One (#14, a Grade A limit) was missed. Our pass mark was set on 1 November 2026, before the first run: at least 13 of the 15 errors found, with no more than three false findings. It is a rough target we set for a demonstration, not a risk-based acceptance criterion and not a validation criterion. Test 1 found 12, with two false findings.
The 25 test documents are written in Estonian. You can download them with the answer key, inspect the planted errors and run your own comparison: Demo Pharma test set (ZIP). We do not publish the prompts and model settings. Before relying on the system, each client tests it on its own documents with a set of known errors.
How we scored: a finding counts in full only when both the documents and the reason are right. The model also repeated two findings that the rules had already made; we count those as duplicates, not as false findings. The bonus gap (no complaints SOP) was found by a rule.
What we changed
- A rule-based pre-check. A short script reads every document code, title, version and appendix, and checks that the processes EU GMP requires have a procedure. No AI involved.
- Regulation as text. The model got the exact clauses as text: Annex 1 §9.30 and §9.32–9.38, EU GMP 4.11–4.12, 8.5 and 9.1–9.2, and one sentence stating the Annex 16 principle that a batch is certified by a Qualified Person. It was told to quote them, not to rely on memory.
- A strict output format. One finding per line, each with a quote from the document and the clause, and the rule findings listed as already known.
Not done yet: the same test on a second model, and a test on a new document set that the model has not seen. Both come next, one change at a time.
The answer key
| # | Level | Planted error | Result |
|---|---|---|---|
| 1 | Easy | SOP refers to a complaints SOP that does not exist | Found (rule) |
| 2 | Easy | SOP refers to form FRM-QA-006; the forms stop at 005 | Found (rule) |
| 3 | Easy | Reference to version 2 of an SOP; version 3 is in force | Found (rule) |
| 4 | Easy | Wrong title given for a QC SOP | Found (rule) |
| 5 | Easy | Reference to Appendix 2; the SOP has only Appendix 1 | Found (rule) |
| 6 | Easy | Quality manual lists an SOP under the wrong title | Found (rule), missed in test 1 |
| 7 | Easy | Two SOPs point to each other for numbering rules; neither has them | Found (AI) |
| 8 | Medium | Deviation closure: 30 calendar days vs 20 working days | Found (AI) |
| 9 | Medium | Batch certified by "QA manager" instead of the QP | Found (AI) |
| 10 | Medium | Assay limits 90–110 % vs specification 95–105 % | Found (AI) |
| 11 | Medium | SOP uses an autoclave retired from service | Found (AI) |
| 12 | Medium | CAPA effectiveness sent to self-inspection, which does not cover it | Partly (AI): right documents, wrong reason |
| 13 | Hard | SOP based on the 2008 Annex 1 | Found (AI) |
| 14 | Hard | Grade A limit "1 CFU" vs "no growth" (Annex 1 2022, Table 6) | Missed, though Table 6 was in the prompt |
| 15 | Hard | Aseptic process simulation once a year instead of twice (Annex 1 §9.38) | Found (AI), quoting §9.38; missed in test 1 |
| B | Bonus | No complaints SOP at all (EU GMP Chapter 8) | Found (rule) |
The rules also flagged that there is no recall procedure (EU GMP Chapter 8). It was not in our key, but it is a real gap, so we do not count it as false.
What we learned
- Split the work. Rules found six of the seven simple reference errors and the bonus gap, in under a second. The seventh, two SOPs pointing to each other, needed the AI. The AI was left with the questions that need reading: roles, limits, deadlines, outdated references.
- Give it the text. Both false findings and the worst miss of test 1 came from regulation answered from memory. With the clauses supplied, this run produced no wholly false findings under our scoring, and the model quoted Annex 1 §9.38 correctly. One finding still gave the wrong reason. This is one run on one document set, and three things changed at once, so it does not show which change helped most.
- Tables are still a weak spot. Table 6 of Annex 1 was in the prompt, yet the model did not compare it with the monitoring SOP (#14). Next we give such limits as plain sentences, or check them with a rule.
- Still not a report. The model repeated two rule findings even though it was told not to. A short clean-up step fixes that; a person still reads the result and decides.
Thirteen in full on our own synthetic set is a good sign, not proof. Real quality systems are larger and messier, and a qualified person stays responsible for every decision. See what the draft Annex 22 says about this.
What comes next
- Limits from tables given as plain text, plus a rule for Grade A limits.
- The same test on a second local model.
- A larger demo set with errors we have not seen ourselves.
We will publish those results the same way, including what they miss.