Tokenization Explained: What AI Tokens Are and Cost

A token is a small piece of text, often about 4 characters, and AI models read and bill you by the token. You pay for the tokens you send and, at a higher rate, for the tokens you get back.

Short answer: Tokenization is the step that cuts your text into tokens before an AI model reads it. Common words are one token, while long or rare words become several. Count your tokens and you can estimate what a prompt costs.

How does AI turn your words into tokens?

AI reads text in pieces, not whole words. Google's Gemini docs say "a token is equivalent to about 4 characters" and "100 tokens is equal to about 60-80 English words." A common rule of thumb is three-quarters of a word per token.

Think of LEGO bricks. A common word is one big ready-made brick. A rare word gets snapped together from small ones.

Here is the round trip:

1. You type some text.
2. A program called a tokenizer cuts it into pieces. OpenAI's is tiktoken, "a fast BPE tokeniser for use with OpenAI's models."
3. Each piece gets an ID number. The model only ever sees those numbers.
4. The model writes its answer as new token IDs.
5. The IDs turn back into words you can read.

These splits come from tiktoken's gpt-4o encoding (o200k_base), as of October 21, 2026.

Even one period matters. "You miss 100% of the shots you don't take" is 11 tokens on gpt-4o's encoding and 12 with a final period.

Why does AI use pieces instead of letters or words?

Pieces are the middle ground between letters and whole words. Spelled out as s-t-r-a-w-b-e-r-r-y, tiktoken counts 10 tokens instead of 3. Letter-sized pieces would make every sentence much longer for the model.

Whole words fail the other way. Every word, name and typo would need its own entry. Byte pair encoding (BPE) settles on a fixed set of pieces and cuts text the same way every time. gpt-4o's encoding holds 200,019 different pieces.

How can you count tokens yourself for free?

You can count tokens in about a minute with a free token counter. All three below need a little code or an API key:

• tiktoken (Python): enc = tiktoken.encoding_for_model("gpt-4o") loads gpt-4o's tokenizer, and len(enc.encode(text)) counts your text.
• Anthropic: Anthropic's token counting endpoint counts before you send. Anthropic says it is "free to use but subject to requests per minute rate limits." The count is an estimate, and "You are not billed for system-added tokens."
• Google: count_tokens, covered in Gemini's token guide, "Returns the total number of tokens in the input only." Thinking tokens aren't in that count; they appear on your usage report after the reply.

Try it now:

1. Run pip install tiktoken.
2. Load the gpt-4o encoding as above.
3. Count "The quick brown fox jumps over the lazy dog." You should get 10.
4. Count "strawberry", then " strawberry" with a leading space. You get 3, then 1.

Why do output tokens cost more? A worked bill

Output tokens cost more than input tokens on the models below. Claude Sonnet 5 is $2 per million input tokens and $10 per million output tokens. Anthropic says that launch price "is now the standard price." A small OpenAI model such as gpt-5.6-luna lists $0.20 input and $1.20 output per million on the Standard tier, on OpenAI's API pricing page.

The formula: tokens × price ÷ 1,000,000. Here is one job, a 2,000-word input with a 1,000-word reply, at three-quarters of a word per token:

On Sonnet 5, the sum is $0.0053 + $0.0133 = $0.0186. The shorter reply costs more than the longer question.

Thinking tokens count too. Gemini's pricing row reads "Output price (including thinking tokens)." Gemini 3.8 Flash and 3.6 Flash list $0.75 input and $3.75 output per million through December 31, 2026. From January 1, 2027 that becomes $1.50 and $7.50 (Standard, paid tier). Prices change, so check the vendor page before you budget.

For bigger jobs, see what an AI agent costs per task or browse our cost estimation tools.

Why do long chats cost more and forget the start?

Every chat turn re-sends the whole conversation as input tokens, so turn 20 pays for turns 1-19 again. Anthropic's docs say each turn's input contains "all previous conversation history plus the current user message."

The context window is the limit. Anthropic defines it as "all the text a language model can reference when generating a response, including the response itself," so input and output share it. Chat apps such as claude.ai can also manage the window "on a rolling "first in, first out" basis," so the oldest text drops out first. Cached input is cheaper: Anthropic prices cache reads at "0.1 times the base input tokens price" for most models.

Do different models count the same text differently?

Yes, each model family cuts text its own way. Anthropic says Claude 4.7 and later models use a newer tokenizer. "The same input text produces approximately 30 percent more tokens than on earlier models," depending on the content.

The same effect shows up across OpenAI encodings. The Gretzky sentence is 11 tokens on o200k_base, 12 on the older cl100k_base and 11 on p50k_base. These counts cover OpenAI's tokenizers only, not Claude's or Gemini's.

Language matters as well. The 2023 paper "Do All Languages Cost the Same?" tested OpenAI's API on 22 languages. Token counts for the same content "vary widely between languages," so some languages cost more. That's one more reason to pick the right model for each task.

Why can't AI count the r's in strawberry?

The model sees pieces, not letters. To the model, "strawberry" is three IDs: st, raw and berry. The letters sit hidden inside them.

The paper "Why Do Large Language Models (LLMs) Struggle to Count Letters?" found "models are capable of recognizing the letters but not counting them." A second paper, "The Strawberry Problem," ties these character-level failures to tokenization. It found such skills "appear suddenly and late in training." More on that in how models learn from training text.

Spelled out with hyphens, the word becomes 10 tokens instead of 3.

Don't confuse tokens with these

Quick check

1. How many tokens is "strawberry" on gpt-4o's encoding? Three: st, raw and berry.
2. On Claude Sonnet 5, which costs more per token? Output, at $10 per million versus $2 for input.

Do thinking tokens cost money? On Gemini, yes: they are billed as output.

Is there one true token count? No. The same text gives different counts on different tokenizers, so count with the one your model uses.