Tool use: the Python interpreter
When it can't do it in its head, it writes a program and runs it — if it has the tool.
Give the model a calculator-shaped problem and a good app won't fake it — it writes a program, emits special tokens that say "run this," pauses, gets the result back as text, and continues. The catch runs through this whole part: tool availability is uneven. A model without the interpreter will cheerfully hallucinate the answer instead.
Ask "30 × 9" and the model just answers 270 — that's easy enough to do "in its head" (pure next-token prediction). Ask it to multiply two big numbers and it can't — so it reaches for the tool.
“Instead of the LLM giving you an answer directly, it has the ability now to write a computer program… Please go off and run it and give me the result of running that computer program.”
Andrej Karpathy·53:07
Grok (at filming) had no interpreter and did a big multiplication "in its head" — landing "remarkably close," but wrong (…0,6,0 instead of …1,2,0). Gemini did one correctly in-head, then failed on a harder one. A model without the tool will still answer — it just makes it up.
The load-bearing points
- Good apps write & run code for calculator-shaped problems.
- Special tokens tell the app to execute, then feed the result back as text.
- Tool availability is uneven across models.
- A model without the tool will hallucinate the number rather than admit it.
Force the tool
Ask your model to multiply two ~7-digit numbers. Does it open a code tool, or just print digits? Verify the result on a calculator.
Show the point
If there's no “analyzing / running code” step, be suspicious — it may have guessed and be subtly wrong.
Tool census
Try the same hard-arithmetic prompt across two or three providers and note which actually run code.
Show the point
This is Karpathy's real point: keep a mental map of which model has which tool, because it decides whether you can trust the answer.
It writes a program and emits special tokens meaning “run this.” The app executes the code, captures the output, feeds it back into the context as text, and the model continues from there.