Token Analyzer
Count tokens using multiple tokenisation schemes and compare costs across different AI models.
Token Analyzer
Exact BPE token counts, live truncation highlight, and paragraph-level heatmap.
Loading tokenizer...
How to use the Token Analyzer
Select a model from the dropdown to set the context window and pricing. Paste or type your prompt into the editor. The tool counts exact tokens using the cl100k_base BPE tokenizer (the same vocabulary used by GPT-4, GPT-4o, GPT-4.1, and GPT-5.x), shows estimated input cost, highlights the truncation boundary in red if your text exceeds the model's context limit, and renders a paragraph-level heatmap showing which sections consume the most tokens.
All tokenization runs entirely in your browser via @dqbd/tiktoken, a WebAssembly port of OpenAI's tiktoken library. No text is uploaded to any server. After the initial CDN load (approximately 1 MB), the tool works fully offline.
Check exactly how many tokens your prompt consumes and whether there is sufficient room for the model's response within the context window. The red truncation highlight shows precisely where content would be cut off.
Switch between models to compare input costs. See the exact dollar cost per API call using each provider's published per-million-token pricing, helping teams decide whether to compress prompts or switch model tiers.
The paragraph heatmap visualises token density across your document. Identify which sections are token-heavy and could be condensed or split to fit within a model's context window while preserving the most important information.
Before uploading training examples to OpenAI or other platforms, confirm each example fits within the target model's maximum sequence length. The exact BPE count avoids rejections from exceeding token limits.
Related tools
How the Token Analyzer works
How byte-pair encoding works
Byte-Pair Encoding is a subword tokenization algorithm that starts with individual bytes and iteratively merges the most frequently co-occurring adjacent pairs into a single token until the target vocabulary size is reached. The cl100k_base vocabulary contains 100,256 tokens - a mix of full common words, subword fragments, punctuation sequences, and whitespace patterns. At inference time, the algorithm applies the learned merge rules in priority order to split input text into the fewest possible tokens.
Truncation boundary detection
The tokenizer returns an array of integer token IDs. The tool walks this array until the cumulative count equals the selected model's context limit, then converts the corresponding byte offset back to a JavaScript string character index using a TextEncoder/TextDecoder round-trip. Text before the boundary is rendered normally; text after it is wrapped in a <mark> element with a red background. The overlay is positioned absolutely behind the <textarea> with matching font metrics so the highlight aligns precisely.
Paragraph token heatmap
After tokenizing the full text, the tool splits it into paragraphs (double-newline delimited), estimates each paragraph's token count via the BPE encoder, and maps the count to a blue colour scale where darker blue indicates higher token density. This visualises which sections of a long prompt consume the most context budget, making it easy to identify candidates for condensation or removal.
Cost estimation logic
Input cost is estimated using each model's published input price per million tokens. For example: GPT-5.5 at $5.00/M, GPT-4.1 at $2.00/M, Claude Opus 4.7 at $5.00/M, Gemini 3.1 Pro at $2.00/M. The formula is cost = (tokenCount / 1,000,000) x pricePerMillion. Output tokens, caching discounts, and batch API pricing are not included.
Claude, Gemini and Kimi approximation
Anthropic, Google, and Moonshot have not publicly released their tokenizer vocabularies. This tool applies calibration factors derived from published benchmark comparisons: Claude and Kimi counts are the cl100k_base count multiplied by 1.04, and Gemini counts are multiplied by 0.97. Qwen models use a compatible tokenizer so counts are exact. Accuracy across typical English prompts is within +-3%.
Frequently Asked Questions
Which tokenizer does this tool use - and is it accurate?
The tool uses the cl100k_base BPE vocabulary via @dqbd/tiktoken - a WebAssembly port of OpenAI's tiktoken library - which is the exact tokenizer used by GPT-4, GPT-4o, GPT-4.1, and GPT-5.x. Qwen models use a compatible tokenizer so counts are exact. For Claude, Gemini, and Kimi models, the tool applies calibration factors derived from published benchmark comparisons - results are accurate to within +-3% across typical English prose.
What does "truncation boundary" mean?
The truncation boundary is the character position where the cumulative token count first reaches the selected model's context limit. Text after this boundary is highlighted in red to indicate it would be silently cut off before being passed to the model. The position is calculated by decoding the first N safe token IDs back to bytes and measuring the resulting string length.
How does the heatmap work?
After tokenizing the full text, the tool splits it into paragraphs by double-newline delimiters, counts tokens in each paragraph, and maps the count to a blue colour scale where darker blue means higher token density. This helps identify which sections of a long prompt consume the most context budget.
Why do token counts differ between models?
Different models use different vocabularies and tokenization algorithms. GPT-4, GPT-4o, GPT-4.1, and GPT-5.x share compatible tokenizers (cl100k_base or o200k_base). Claude uses SentencePiece, Gemini uses Google's SentencePiece vocabulary, and Kimi uses a custom tokenizer. This tool applies calibration factors to approximate counts for non-OpenAI models.
Does my text get sent to a server?
No. Tokenization runs entirely in your browser. The @dqbd/tiktoken library is loaded once from a CDN as a WebAssembly module and cached by your browser; after the initial load, the tool works fully offline. Your text never leaves your device.
Can I count tokens for code, not just prose?
Yes. The cl100k_base and o200k_base tokenizers handle source code well - they were trained on large corpora that include GitHub repositories. Code tends to tokenize at a slightly lower character-per-token ratio than English prose because identifiers, operators, and indentation whitespace are encoded efficiently.