Advanced Voice Mode (true audio)
Now the model hears and speaks natively — audio chopped into tokens, no text in between.
Real audio means treating sound the way we treat text: chop the waveform into a spectrogram, quantize little windows into audio tokens (a vocabulary of ~100k chunks), and train the model on them. Now it can hear and speak directly — no text in the middle. This unlocks things fake audio never could: accents, character voices, tone. It's also "very cagey" and refuses a lot.
This is the other voice button. Under the hood, audio is tokenized — a spectrogram sliced into windows, each mapped to one of ~100,000 audio chunks — so the model natively understands and generates sound.
In his demo it explains Rayleigh scattering, then speaks as Yoda, then a pirate, tells a story, counts fast, and does animal sounds — feats impossible for a text model with a TTS bolt-on. But it also refuses often ("I can't actually produce animal sounds") right after doing exactly that.
“I do find that the advanced voice is very, very cagey. It really doesn't like to do stuff — it will refuse a lot. So I do find it sometimes a little bit too cringe.”
Andrej Karpathy·1:27:04
A modality handled inside the model (audio tokens) is far more powerful than one wrapped around it (STT/TTS). You'll see the same distinction for images and video: native / omni-model beats separate model glued on by text.
The load-bearing points
- True audio = audio tokens (spectrogram → quantized chunks) inside the model.
- The model hears and speaks directly — no text intermediary.
- Unlocks voices, accents, tone; the text you see is just a transcript.
- ChatGPT's is cagey/refuses; Grok's voice is far less filtered.
Push the voice
In Advanced Voice, ask it to explain something as a specific character or in a specific accent, then to change tone mid-answer.
Show the point
These only work because audio is native. A TTS wrapper couldn't restyle its own voice on request.
Feel the cageyness
Ask for something playful (an animal sound, an impression). Note where it refuses, then try the same on Grok voice.
Show the point
This is his real-world caveat: capability is there, but guardrails vary a lot by provider.
Convert the sound to a spectrogram, slice it into small time windows, and quantize each into one of ~100k “audio tokens.” The model trains on these just like text tokens.