How Many Tokens Are in X Words? AI Token Estimates Explained

1. Quick Estimate Table

For standard English prose, a widely used rule of thumb is 1 word ≈ 1.33 tokens (or equivalently, 100 tokens ≈ 75 words). Here is a quick reference table:

These estimates assume typical English text without heavy code, URLs, or special formatting. The actual count can swing 10–30% in either direction depending on vocabulary and structure.

2. Why Tokens Are Not the Same as Words

A “token” is the smallest unit of text that an LLM processes. Modern models use subword tokenization algorithms — most commonly Byte Pair Encoding (BPE) — that split text into chunks based on frequency patterns learned from massive training corpora. This means:

• Common short words like “the”, “is”, and “and” are single tokens.
• Longer or less common words get split: “unbelievable” might become “un” + “believ” + “able” (3 tokens).
• Spaces, punctuation, and special characters often consume their own tokens.
• Code, URLs, and technical jargon tokenize far less efficiently than plain English prose.

The key insight: a “word” in English is a human concept defined by spaces. A “token” is a statistical unit defined by the model's vocabulary. They rarely map 1:1.

3. Why Token Counts Vary by Model and Language

Different models use different tokenizers. GPT-4 and GPT-4o use cl100k_base (with a vocabulary of ~100,000 tokens), while GPT-3.5 uses the older p50k_base. Claude uses its own tokenizer, and Gemini uses SentencePiece. The same sentence can produce different token counts on each model because their vocabularies were trained on different data distributions.

Language matters even more. English text averages about 1.3 tokens per word, but:

• Chinese, Japanese, Korean: Each character may become 2–3 tokens, since CJK characters are rarer in the training vocabulary.
• German, Finnish: Long compound words get split into many subwords, inflating token counts.
• Code: Variable names, operators, and syntax symbols create many small tokens. A 100-line Python script may use 3–4x more tokens than an equal-length paragraph of English.

If you work across languages or send code to LLMs, using a word-count estimate will significantly undercount your actual token usage.

4. How Punctuation and Formatting Affect Tokens

Whitespace and formatting are not free. Here are common patterns that inflate token counts beyond what you might expect:

• Markdown formatting. Headers (## Title), bold (**text**), and lists each add tokens for the syntax characters.
• URLs. A single URL like https://example.com/api/v2/users?page=1 can consume 15–20 tokens.
• JSON payloads. Curly braces, colons, commas, and quoted keys all count. A 20-field JSON object can easily exceed 100 tokens.
• Repeated newlines and indentation. Excessive whitespace in prompts wastes tokens. Minify your prompts for production use.

The practical takeaway: when optimizing prompts for cost, strip unnecessary formatting, shorten URLs, and avoid pasting raw data dumps. Summarize the data and describe what you need instead.

5. Why Token Count Matters for Cost and Context Windows

LLM APIs charge separately for input tokens and output tokens. As of mid-2026, GPT-4o charges roughly $2.50 per million input tokens and $10 per million output tokens. Claude 3.5 Sonnet has similar pricing. These numbers seem tiny until you multiply by thousands of requests per day.

Consider a chatbot that includes a 2,000-word system prompt (~2,667 tokens) in every request. At 10,000 requests per day, that system prompt alone costs roughly $0.67/day — $20/month — before the user even types anything. Trimming that system prompt from 2,000 words to 500 words (667 tokens) cuts the cost by 75%.

Context windows impose hard limits. GPT-4o supports 128K tokens, Claude 3.5 supports 200K, and Gemini 1.5 Pro supports up to 2M. These windows include both the system prompt and the conversation history. Once you hit the limit, older messages get dropped or the request fails. Understanding your token budget lets you design conversations that stay within bounds.

For accurate estimates on real text, a heuristic word count is a starting point — but a dedicated token counter gives you the actual number before you commit to an API call.