AI Token Counter

What Are AI Tokens and Why Do They Matter?

When you send text to an AI API — whether it's GPT-4o, Claude, or Gemini — the model never sees raw characters. Instead, it sees a sequence of tokens: compact numeric IDs that represent fragments of text. Tokenization is the first step in every large language model (LLM) inference pipeline, and understanding it is the single most practical skill for controlling API costs and avoiding context window errors in production.

For English text, a rough rule of thumb is 1 token ≈ 4 characters, or about 0.75 words per token (meaning roughly 1.33 tokens per word). A 500-word blog post is around 670 tokens. A 10,000-word document is around 13,000 tokens. But these are averages — token boundaries shift depending on the vocabulary each model was trained with, the language of the text, and how much punctuation or code appears.

How Tokenization Works: BPE Under the Hood

Most frontier LLMs use Byte Pair Encoding (BPE) to split text into tokens. BPE starts with individual characters and iteratively merges the most frequent adjacent pairs until it builds a vocabulary of a fixed size (50,000–200,000+ entries for modern models). The result is a vocabulary where common words and word fragments each have their own token ID, while rare words get split into subword pieces.

Concrete examples from GPT-4's cl100k_base vocabulary:

• “the” → 1 token (extremely common)
• “understanding” → 3 tokens: “under” + “stand” + “ing”
• “GPT-4” → 3 tokens: “GP” + “T” + “-4”
• “Hello, world!” → 4 tokens: “Hello” + “,” + “ world” + “!”

Note that spaces are typically merged with the following wordinto a single token — this is why “ world” (with a leading space) is one token rather than two. This detail is invisible to users but explains why you can't simply count characters and divide by 4 and expect exact results for every input.

Real Use Case: Estimating Cost Before You Build

Suppose you're building a document summarization service. Your users upload PDFs of roughly 20 pages (≈8,000 words ≈ 10,700 tokens). Each request uses a 500-token system prompt and generates a 400-token summary. Total per request: ~11,600 input tokens + 400 output tokens.

Using GPT-4o at $2.50/M input and $10.00/M output:

• Input: (11,600 / 1,000,000) × $2.50 = $0.029 per request
• Output: (400 / 1,000,000) × $10.00 = $0.004 per request
• Total: $0.033 per request — about $33 per 1,000 documents

Switching to GPT-4o Mini at $0.15/M input and $0.60/M output:

• Input: $0.00174 · Output: $0.00024 · Total: $0.002 per request
• That's $2 per 1,000 documents — a 94% cost reduction

This kind of calculation — running token counts through the cost formula before you write a single line of integration code — can save thousands of dollars and prevent architecture mistakes that are painful to unwind after launch.

Context Windows: The Hard Limit

Every model has a context window: the maximum total tokens (input + output) it can process in a single API call. Requests that exceed this limit are rejected with an error — the model does not silently truncate your input. Common context windows in 2025:

• GPT-4o / GPT-4o Mini: 128,000 tokens (~96,000 words)
• o1 / o3: 200,000 tokens (~150,000 words)
• Claude Opus 4 / Sonnet / Haiku: 200,000 tokens
• Gemini 1.5 Pro / 1.5 Flash / 2.0 Flash: 1,000,000 tokens (~750,000 words)

In chat applications, context window pressure builds over time as you include the full conversation history in every request. A 1-hour support chat transcript can easily reach 20,000+ tokens. Strategies to manage this include summarizing older turns into a compact “memory block” and only including recent messages verbatim.

Input vs Output: Where the Cost Actually Lives

One of the most counterintuitive facts about LLM pricing is that output tokens cost 3–5× more than input tokensper million. This is because generating each output token requires one full forward pass through the entire model (autoregressive decoding), while input tokens are processed more efficiently in parallel using attention mechanisms. The practical implication: long, verbose AI responses cost far more than concise ones. Prompting the model to be concise (“Answer in under 100 words”) is a legitimate cost-reduction technique, not just an aesthetic preference.

For most chat applications, input tokens dominate because you include the conversation history, system prompt, and any retrieved documents in every request. As history grows, per-turn input cost grows linearly. Batching, caching, and summarization all attack this growth directly.

Prompt Caching: A Hidden Cost Lever

Both Anthropic and OpenAI now offer prompt caching, which lets you mark a prefix of your prompt as cacheable. If the next API call uses the same prefix, the cached tokens cost dramatically less: Anthropic charges ~10% of normal input price for cache hits; OpenAI charges 50% for cached tokens in most tiers. For applications where a large system prompt or reference document is reused across many requests, prompt caching can cut total API spend by 50–80% without any change to quality.

To use caching effectively, structure your prompts so that the stable, reusable content (system instructions, documents, few-shot examples) comes first, and the dynamic, per-request content (user message, query) comes last. The cache hit applies only to the identical prefix, so even a single character difference in the prefix invalidates it.

Tokens Across Languages and Code

The 4-characters-per-token rule applies primarily to English text. Other languages and content types tokenize differently:

• Non-Latin scripts (Chinese, Japanese, Arabic, Hindi): Often 1–3 characters per token, meaning the same semantic content costs 2–4× more in tokens than English.
• Source code: Typically 2–3 characters per token due to dense punctuation, brackets, and identifiers. A 100-line Python file is roughly 1,000–1,500 tokens.
• Numbers: Each digit is often its own token. “1000000” (7 digits) can be 4–7 tokens.
• URLs: Slashes, dots, and hyphens are separate tokens. A long URL can tokenize to 30+ tokens.

When working with non-English content or code-heavy prompts, add a 20–40% buffer to your token estimates to avoid unexpected context window overruns.