Blog

Why Keeplore drops rules whose quote isn't in the transcript

Numair Khan · 24 October 2026 · 7 min read

A rule is kept only if its source quote appears in the employee's own words. If it does not, the rule is dropped.

Keeplore turns interview transcripts into rules for AI agents, and a model proposes each rule's quote, so we check it. This post describes that check as of 24 October 2026, three bugs we found and fixed while writing it, and what it still does not do.

The problem: a rule with an invented source looks verified

Keeplore asks a model to read an interview transcript and return rules. Each rule has a condition (when), an action (do), a reason (why) and a source_quote, which the schema describes as "exact words copied from the employee's messages".

Models do not always copy. Sometimes they return a clean, plausible sentence the person never said. A rule with a made-up quote is worse than a rule with no quote, because the quote is there so a reviewer can trust it. So we do not trust the model's claim. We check it.

The check, in pseudo-code

normalise(s) = decompose accents (NFKD) and remove the combining marks
               lowercase
               put spaces around each Chinese, Japanese or Thai character
               replace every run of characters that are not letters or digits (any script) with one space
               trim

said = normalise( join(employee_messages_only, " ") )

for each rule the model returned:
    q = normalise(rule.source_quote)
    keep it if  q is not empty
           and  (" " + said + " ") contains (" " + q + " ")
    otherwise drop it

That is the whole filter. A rule's condition, action and reason are not compared to anything. Only the quote is.

What the check catches

Employee: Officially it's no refunds on annual, but if they're past the window we do a prorated credit for the months left. Anything over five hundred I check with Dana in finance first.

The first quote leaves out the words before it. That is fine, because it is still a piece of what was said. The second is a paraphrase presented as a quote. The check cannot tell a careless paraphrase from a fabrication. Both fail.

Why we normalise

Models change punctuation. They swap straight and curly apostrophes, drop a trailing full stop, and join lines that were separate. Interview text has line breaks and double spaces. If we compared raw strings, we would drop real quotes for cosmetic reasons.

So both sides are reduced to letters and digits separated by single spaces, with accents folded. "It's" and "it’s" both become "it s". Case, punctuation, accents and whitespace stop mattering. The cost is in the limits below: normalising is also what makes the check looser than it looks.

Why only the employee's turns

The model sees the whole transcript, interviewer included. The check searches only the employee's messages.

An interviewer question can contain a rule-shaped sentence: "So for anything over five hundred you check with finance?" If the model quotes that and the employee only said "yes", the rule would look as if the employee had stated it. Excluding the interviewer's turns means a quote must come from the employee's own words.

What happens to the rules after the check

• Survivors are saved as pending. A manager sees each rule with its quote and approves, edits or rejects it. Only approved rules are ever returned by the query endpoint.
• Re-extraction replaces only pending rules. Rules a manager has already approved or rejected stay, and are sent back to the model with an instruction not to repeat them. Extraction runs again only when the employee has sent new messages since the last run.
• Contradictions with approved rules are flagged. After saving, Keeplore looks for approved rules in the same organisation that share words with the new ones, asks a model whether any new rule directly contradicts one, and flags it for the reviewer. It does not decide which rule is right, and it does not block anything.
• The prompt tells the model not to include passwords or personal customer data. That is an instruction to the model. The quote check does not enforce it.

Three bugs we found while writing this post

Describing the check line by line showed us three ways it was weaker than we thought. We fixed all three on 24 October 2026 and added tests for each.

1. Quotes that normalise to nothing were kept. The blank test ran on the raw quote, but the match used the normalised one. A quote made only of punctuation, or written in a script the old normaliser did not keep, became an empty string, and an empty string is found inside every transcript. Now a quote that is empty after normalising is dropped.
2. The match ignored word boundaries. It compared characters, so the quote "ten" was found inside "often". Now both sides are padded with a space and the quote must appear as whole words.
3. Accented and non-Latin letters were treated as separators. The old normaliser kept only a to z and 0 to 9, so "Müller" became "m ller", and "Müller" and "Möller" matched each other. Now accents are folded ("Müller" becomes "muller") and letters and digits in any script are kept. Chinese, Japanese and Thai have no spaces between words, so there each character counts as a word.

What it still does not catch

We would rather you hear these from us.

1. A real quote that does not support the rule. The check proves the words exist. It does not prove the rule follows from them. A model can quote a true sentence and draw the wrong conclusion. This is why a manager approves each rule. The check removes one kind of error. It does not remove the need to review.

2. Very short quotes prove little. There is no minimum quote length. "We do" passes if those two words appear in that order anywhere.

3. A quote can span two messages. The employee's messages are joined into one string. A quote made from the end of one message and the start of the next passes, even though the employee never said those words together.

4. Accent folding has a cost. "Müller" and "Muller" now match, and so do "Müller" and "Möller" once accents are gone. A model that writes "Mueller" is still dropped. We think folding is the right trade for names. Tell us if you disagree. We have tests for Dutch, Russian and Chinese text, but we have not run a real interview in a non-Latin script yet.

5. Dropped rules are silent, and we have not measured how many there are. Nobody sees a count of dropped rules. If the model paraphrases honestly, a valid rule disappears. We have no figure for how often that happens and are not going to guess one.

6. It says nothing about whether the employee was right. If the employee misremembers a threshold, the quote is real and the rule is wrong. The reviewer is the only check on that.

The same principle at query time

When an agent calls the query endpoint, Keeplore looks only at approved rules. It asks a model to answer using only those rules and to return an empty answer if none apply. The code then returns an empty answer unless the model cites at least one rule that really exists in the set it was given. The answer text itself is written by the model and is not checked against the rules, which is why the rules it used come back with it.

So if no approved rule applies, the answer is empty, and the agent can hand over to a person. The response gives each rule, its reason, and who it came from (name and role). The quote stays in Keeplore for the reviewer; the quote check is a gate at extraction, not at query time.

What we would like feedback on

1. Should there be a minimum quote length? If so, what? Too high and we drop honest short quotes like "no refunds". Too low and the check proves little.
2. Should quotes that span messages be rejected? We would need to check each message on its own.
3. Should dropped rules be visible to the reviewer? We would show them as "dropped: quote not found", so a reviewer can see what the model tried.
4. Is accent folding right for your language? If you build for Dutch, German, Polish or Greek text, tell us what you do.

Write to hello@keeplore.dev.

You can see the check at work on your own interview: each rule you review shows the quote it came from. The whole flow is in How Keeplore works.

Related

• How Keeplore works
• Keeplore vs RAG
• API docs