You're talking to a zip file
A lossy, probabilistic compression of the whole internet — and nothing else, by default.
Everything else in this course sits on one picture. A new chat is an empty token stream — you and the model take turns writing tokens into it, and that stream is the context window, the model's working memory. The thing writing back is a ~1 terabyte file on a disk: about a trillion neural-net parameters that compressed the internet into a lossy, probabilistic "zip file." By default it is sealed — no calculator, no browser, no memory of last week. Steering that file, and breaking it out of the box, is the whole rest of the course.
When you click New Chat, you begin a one-dimensional sequence of tokens. You write some; you hit enter; control passes to the model, which writes its own tokens back until it emits a special "I'm done" token and hands the turn back to you. Together you build out the stream.
That stream is the context window — "kind of like the working memory of this conversation." Anything inside it is directly accessible to the model. Anything outside it might as well not exist.
Pre-training chops all of the internet into tokens and compresses them into the parameters — roughly a 1 TB / 1 trillion-parameter file. It costs tens of millions of dollars and months of compute, so it happens rarely. Post-training then swaps the data for human-written conversations, giving the file the persona of an assistant — "attaching a smiley face to the zip file."
“The zip file is lossy and probabilistic… because we can't possibly represent all of internet in just one terabyte. So we just kind of get the gestalt or the vibes inside this zip file.”
Andrej Karpathy·00:24
Because pre-training is rare and expensive, the model's knowledge is "a little bit out of date." Ask about last week and it simply isn't in the file. Hold onto this — it's the reason half of Part III (tools) exists.
So the default entity is fully self-contained: a file that turns tokens into tokens, carrying knowledge from pre-training and style from post-training. That's it.
“There's no calculator. There's no computer and Python interpreter. There's no worldwide-web browsing. There's none of that. There's no tool use yet… You're talking to a zip file.”
Andrej Karpathy·00:53
The load-bearing points
- New chat = empty token stream = the context window = the model's working memory.
- The model is a ~1 TB, ~1 trillion-parameter lossy, probabilistic compression of the internet.
- Pre-training gives it knowledge; post-training gives it the assistant persona.
- By default it's sealed — no tools — and its knowledge has a cutoff.
Feel the recall
Open a fresh chat and ask a common, stable fact (“how many milligrams of caffeine in a shot of espresso?”). Notice you're reading its compressed memory of the internet, not a lookup.
Show the point
If it answered instantly with no “searching…”, that was pure zip-file recall — vivid because the fact is all over the internet.
Hit the cutoff
Now ask about something from the last two weeks. Watch it hedge, guess, or say it doesn't know.
Show the point
That wall is the knowledge cutoff. In Part III you'll hand it a search tool so it can go get the answer.
It didn't store the internet verbatim — it kept a compressed “gestalt.” Frequently-repeated facts are recalled well; rare ones are fuzzy or invented. Every answer is a statistical best-guess, not a retrieval.
Its knowledge is frozen at pre-training time — the knowledge cutoff. Pre-training is too costly to redo often, so anything after that date isn't in the file unless you add it via a tool or the context window.