Knowledge queries & when to trust them
Caffeine and cold medicine — real questions, and the exact reasoning for trusting the answer.
Karpathy doesn't blindly trust the model and he doesn't blindly distrust it. He runs a quick three-part check: is the knowledge recent? Is it common on the internet? Is it high-stakes? "Not recent, very common, low stakes" is the sweet spot where a zip-file answer is fine — and he still glances at the box.
His two real examples both pass the same test. First: how much caffeine is in one shot of Americano? The model says ~63 mg.
Second: he has a runny nose, asks for meds, then asks whether DayQuil or NyQuil would help. The model reads out the ingredients and reasons about each — and here's the tell:
“I actually took out the box and I looked at the ingredients and I made sure that NyQuil's ingredients are exactly these… because I don't always fully trust what's coming out here.”
Andrej Karpathy·08:26
Before trusting a bare (tool-less) answer: 1) Recent? If yes, it may not be in the file. 2) Common online? If no, recall is weak. 3) High-stakes? If yes, verify regardless. Caffeine and cold medicine pass on all three — so a quick sanity check is enough.
The load-bearing points
- Use the gate: recency, commonness, stakes.
- The model's answer is a statistical recollection, not a lookup — even when it's right.
- Low-stakes + common + not-recent = trust with a glance.
Run the gate
Take three questions you'd ask an LLM this week. For each, score it on recency / commonness / stakes, and decide: trust it, verify it, or reach for a tool.
Show the point
Anything scoring “recent” or “high-stakes” should not rest on bare recall — that's your cue for search (Part III).
Check the box
Ask about the ingredients or spec of a physical product you own, then compare against the actual label.
Show the point
This is exactly Karpathy's NyQuil move — calibrating how much to trust recall on this kind of question.
Correctness of one answer doesn't change what the model is: a probabilistic recollection. On anything touching health he verifies against a primary source out of habit, because the model gives no guarantee.