What Are LLM Tokens?

Large language models don't read text as characters or words — they read tokens: fixed pieces of a vocabulary that a tokenizer splits your text into before it ever reaches the model. A token might be a whole word ("cat"), part of a word ("token" + "ization"), a punctuation mark, or a single character, depending on what the tokenizer has seen during training.

Why not just count characters or words?

Every provider bills, and limits context, by token count — not characters or words. A rough rule of thumb for English prose is about 4 characters per token, or roughly 0.75 words per token, but this varies significantly with the content:

• Common English words often tokenize as a single token.
• Rare words, names, and non-English text often split into multiple tokens.
• Code, symbols, and unusual formatting can tokenize less efficiently.

A simple approximation

Our Token Calculator uses this character-based heuristic to give you an instant, client-side estimate:

This is deliberately labeled as an estimate. If you need an exact count for a specific model, use that model's official tokenizer — the vocabulary and splitting rules differ between model families.

Why this matters

Token counts drive three things directly: how much a request costs (see how LLM pricing works), how much of the model's context window a prompt consumes, and whether a request fits inside that window at all.