Tokens
A chatbot doesn't read words. It cuts your text into pieces, like Lego bricks, and predicts the next brick. Those bricks are tokens. They decide how long a chat can be, what it costs, and why it can't count letters.
First, what do you think?
How does a chatbot see the word unbelievable? Guess before you read on.
What is a token?
The whole story in three sentences. A chatbot doesn't read words. It first cuts your text into pieces, called tokens. It predicts the next token, not the next word.
Think of Lego. A sentence is built from bricks. A common word like house is one brick. A long word like unbelievable is several. The tokenizer on this page makes it un, bel, ievable. A space, a comma and an emoji are bricks too.
A token is one of those bricks. It can be a whole word, part of a word, or a single character.
- Common words are one token. The, cat, sandwich. They appear so often that they got their own brick.
- Rare or long words are cut up. Pieces like un or ing show up in lots of words. So they're bricks of their own.
- The model sees numbers. Each brick has a number. The model never sees your letters, only the list of numbers.
A rule of thumb from Google: one token is about four characters. A hundred tokens are about 60 to 80 English words.
The part that cuts the text is the tokenizer. Every model family has its own. So the same sentence can be 9 tokens in one chatbot and 11 in another.
A first small step: type your own name into the token counter below. How many bricks are you?
What the predictor does
The predictor predicts the next brick. One token at a time. Then it adds it, and predicts the next.
Why does that matter to you? Because everything a chatbot does is counted in tokens.
- How long a chat can be. A model can only look at a certain number of tokens at once. That's the next lesson.
- What it costs. If you pay per use, the price depends on the tokens in and out.
- Strange mistakes. Ask how many r's are in strawberry. The model doesn't see letters. It sees bricks like st, raw and berry. Counting letters inside bricks is hard for it.
An example from one maker. On one of its newer tokenizers, a million tokens fit about 555,000 words. Its older models fit about 750,000 words in the same million. Same text, different bricks.
The predictor builds with bricks, not letters. Everything is counted in tokens.
Try it yourself
Type a sentence and see it cut into tokens. Each colour is one token. Try an English sentence, then the same sentence in another language.
Everything runs in your browser. Nothing you type is sent anywhere.
Use the free version of any chatbot. Don't type anything personal.
Ask a chatbot:
How many letters r are in the word strawberry? Then spell the word one letter at a time and count again.What to expect. The first count may be wrong. When it spells the word letter by letter first, it has a better chance.
What to look for. Spelling it out turns each letter into its own brick. Then the predictor can "see" them.
In practice
Three people run into tokens without knowing it.
Case 1The poem with the wrong number of syllables
Zoë (11) asks a chatbot for a haiku. That's three lines with 5, 7 and 5 syllables. The poem is lovely, but the counts are off.
Ask again and again, hoping it gets better.
The predictor sees tokens, not syllables. Counting syllables means looking inside its bricks.
Let it write the poem. Then count the syllables yourself, and fix the lines that are off.
It does what it's good at: words that fit. You do what it's bad at: counting.
Case 2The long document
Ravi (27) pastes a 90-page report into a chatbot and asks for a summary. The chatbot says the text is too long.
Give up.
Every chatbot has a limit in tokens. Ninety pages can be more than fits.
Split the report into parts, and summarise each part. Then ask for a summary of the summaries.
Each part fits within the limit. You still get the whole picture.
Case 3Dutch costs more
Saskia (44) runs a small business and pays per use for an AI tool. She sees her Dutch texts cost more than her colleague's English ones.
Think she's being charged unfairly.
Common pieces get their own brick. In many tokenizers, more English words are common enough for that. Dutch words are more often cut into several. More bricks, higher price.
Try it in the token counter: the same sentence in Dutch and English. Take that into account when she estimates costs.
She knows where the difference comes from, and can plan for it.
By situation
Chatbots are bad at counting letters, syllables and words. For a poem with rules or an essay of exactly 300 words, count yourself. Use the chatbot for ideas and wording.
When you pay per use, the price depends on the tokens in and out. Long prompts, long documents and long answers cost more. Shorter, clearer prompts save money.
You don't pay per token with a free chatbot. But there is a limit on how much you can paste. And on how long you can chat. That limit is counted in tokens.
Myths and mistakes
| Claim | True? | Why |
|---|---|---|
| One token is one word | No | Often it's a word, but also part of a word, a space or a single character. |
| A chatbot reads letters | No | It sees tokens, as numbers. That's why counting letters is hard for it. |
| Every chatbot counts tokens the same way | No | Each model family has its own tokenizer. |
| Every language costs the same | No | Words that are less common for the tokenizer are cut into more pieces. Try it in the counter. |
Related topics
Read first
- Language modelsthe predictor that predicts token by token.
Next
- The context windowhow many tokens fit on the whiteboard.
Related
- Free or paidpaying per token, or a subscription.
Sources
- explainerGoogle for Developers. LLMs: What's a large language model?.A language model predicts the next token, like a very large autocomplete. A token can be a word, part of a word or one character. Training needs a huge amount of text. Predictions often contain mistakes.
- documentationOpenAI. tiktoken.Language models see numbers called tokens, not text. On average one token is about 4 bytes. Common pieces like ing become their own token. Open source (MIT); the tokenizer on this site uses its word list o200k_base.
- documentationGoogle AI for Developers. Understand and count tokens.For Gemini, a token is about 4 characters, and 100 tokens are about 60 to 80 English words. Each model has a maximum number of tokens, and the price depends partly on the tokens in and out.
- documentationAnthropic. Models overview.Each model has a training data cutoff: the date its training text stops. It also lists a reliable knowledge cutoff, and context windows in tokens.
Test
You get 5 questions about the whole page. Get 4 right, and you've mastered the topic.
About the tools in this course. Every tool is described in the same way, in alphabetical order, without a recommendation. Names, prices and features change often: the maker's own site is always right. This site was built with the help of Claude, one of the tools described here.