Count Tokens for GPT-4o with tiktoken

OpenAI models use the tiktoken library with BPE (Byte Pair Encoding) to tokenize text. GPT-4, GPT-4o, and GPT-3.5-turbo all use the cl100k_base encoding with a vocabulary of ~100,000 tokens. Knowing the exact token count for your prompts is essential for staying within context limits and predicting API costs — the counter shows not just the total but each individual token boundary so you can see exactly how the tokenizer splits your text. A practical rule of thumb for English text: 1 token ≈ 4 characters or ¾ of a word. But this breaks down for code, URLs, numbers, and non-English text. A line of Python code typically uses 1 token per ~3 characters because many identifiers and symbols are not common English words. A base64-encoded string uses roughly 1 token per character because the tokenizer has no learned patterns for random character sequences. For API calls that use the Chat Completions API, add overhead tokens to your count: each message adds 3–4 tokens for role encoding and separators, and the reply priming adds 3 tokens. The tool adds these automatically when you check the "include message overhead" option, giving you the true billable token count.

Count tokens in this technical paragraph:

The OpenAI API uses tiktoken, a fast BPE tokenizer, to convert text into tokens before processing. The cl100k_base encoding used by GPT-4 and GPT-3.5-turbo has a vocabulary of approximately 100,000 tokens. Common English words are usually single tokens, while technical terms, URLs, and code identifiers often split into multiple tokens.