LLM token optimization is a measurement and design problem, not simply a matter of making prompts shorter. This guide shows how to count input and output tokens, estimate API spending with provider-specific rates, remove waste, compress conversation history, use caching where available, and protect answer quality with repeatable evaluation.
Overview
Tokens are the units an LLM processes. A token may represent part of a word, a complete short word, punctuation, or formatting, depending on the model and tokenizer. API billing commonly separates input tokens from output tokens, although the exact billing categories, cache treatment, and rates vary by provider and model. Always use the provider's current documentation and calculator for production estimates.
Token optimization has four objectives:
- Reduce unnecessary input content, such as repeated instructions, stale history, and irrelevant retrieved documents.
- Prevent oversized outputs by defining a useful response format and an appropriate output limit.
- Reuse stable context through caching or application-level storage when the API supports it.
- Preserve quality by testing accuracy, relevance, safety, and consistency before and after each change.
The safest approach is incremental. Establish a baseline, change one part of the request, measure token and quality effects, then keep the change only if it improves the combined cost-quality target. A lower token count is not automatically better if it causes retries, hallucinations, poor retrieval, or extra human review.
For a broader production workflow, see the production-ready AI workflow guide. If your application uses retrieved documents, the RAG tutorial for beginners explains how retrieval choices affect the context sent to a model.
How to estimate
Start with a request-level cost model. Use these variables:
- I: input tokens per request
- O: output tokens per request
- R: requests per day or month
- Pi: provider price for one unit of input tokens
- Po: provider price for one unit of output tokens
- U: the provider's billing unit, such as a documented number of tokens
A basic estimate is:
Cost per request = (I ÷ U × Pi) + (O ÷ U × Po)
Then calculate:
Period cost = cost per request × requests in the period
For a more realistic estimate, split traffic into request types. A support answer, document summarization job, extraction task, and agentic workflow may have very different input lengths, output limits, and retry rates. For each type, record request volume, average input tokens, average output tokens, the selected model, and the percentage of requests that trigger a retry or follow-up call.
Do not estimate token counts by character length alone. Character-to-token ratios vary with language, code, JSON, whitespace, and the selected tokenizer. Use the model's tokenizer or an SDK usage field where available. Log the actual input and output usage returned by the API, while taking care not to store sensitive prompt content unnecessarily. The AI data privacy checklist can help structure safer logging decisions.
For a useful comparison, calculate both absolute savings and percentage savings:
Token reduction % = ((baseline tokens − revised tokens) ÷ baseline tokens) × 100
Cost reduction % = ((baseline cost − revised cost) ÷ baseline cost) × 100
Keep input and output reductions separate. A prompt may reduce input tokens while causing longer answers, or limit output while increasing clarification requests. The combined request cost and task success rate are more informative than either number alone.
Inputs and assumptions
Build a token-budget worksheet with one row for each important request path:
| Field | What to record | Why it matters |
|---|---|---|
| Request type | Chat, extraction, summarization, RAG, or tool call | Different tasks need different budgets |
| Model and tokenizer | Exact model identifier and tokenizer method | Token counts are model-dependent |
| Input tokens | System instructions, history, user text, tools, and retrieved context | Shows where the prompt budget goes |
| Output limit | Configured maximum and observed output length | Prevents uncontrolled responses |
| Quality result | Pass rate, evaluator score, or human review outcome | Detects savings that damage usefulness |
| Retry rate | Requests repeated because of errors or weak answers | Captures hidden token usage |
Separate fixed context from variable context. Fixed context includes role instructions, policy rules, output schemas, and stable examples. Variable context includes the current user request, conversation history, retrieved passages, and tool results. This distinction helps you identify candidates for caching, reuse, summarization, or removal.
When setting an output limit, begin with the smallest limit that can complete the task reliably. Use an explicit format: for example, a specified number of bullets, a compact JSON schema, or a short field-by-field extraction. A limit should be a guardrail, not a substitute for a clear prompt. If responses are frequently cut off, increase the limit or redesign the task into smaller steps.
For RAG systems, measure the retrieved context separately from the instruction text. Sending more passages can increase apparent grounding while also adding noise and cost. Test retrieval count, chunk size, deduplication, and relevance thresholds together. In many applications, a smaller set of highly relevant passages is easier for the model to use than a large undifferentiated context.
Use prompt templates and version control for repeatability. The guide to prompt testing frameworks provides a useful structure for comparing revisions against the same evaluation set.
Worked examples
Assume a support application makes 10,000 requests in a review period. Its baseline averages 2,400 input tokens and 600 output tokens per request. A refactor produces 1,500 input tokens and 450 output tokens while maintaining the same measured task pass rate.
| Measure | Baseline | Revised |
|---|---|---|
| Input tokens per request | 2,400 | 1,500 |
| Output tokens per request | 600 | 450 |
| Total tokens per request | 3,000 | 1,950 |
| Total tokens for 10,000 requests | 30,000,000 | 19,500,000 |
| Total-token reduction | 35% | |
To convert this into a cost estimate, enter the current provider rates for the selected model and billing unit. Do not assume input and output tokens have the same rate. If the application uses multiple models, calculate each model separately before adding the results.
Consider how the reduction was achieved. Removing duplicated system instructions may be low risk. Replacing a long conversation with a structured summary may save tokens, but the summary can omit a detail that matters later. Reducing retrieved passages can lower cost, but only if retrieval quality remains acceptable. Changing to a smaller model may have a larger cost effect than prompt editing, but it requires a controlled quality comparison.
A practical optimization sequence is:
- Measure actual usage by request type.
- Remove duplicated or obsolete instructions.
- Trim conversation history using a clear retention rule.
- Deduplicate and rank retrieved context.
- Define the smallest reliable output format and limit.
- Cache stable prefixes or reusable context when supported.
- Compare models using the same evaluation set.
- Monitor retries, latency, quality, and cost after release.
For implementation ideas, review prompt templates for common AI developer workflows and the AI output evaluation metrics guide.
When to recalculate
Recalculate your token budget whenever a pricing input, model, tokenizer, or traffic pattern changes. Provider pricing, context limits, cache rules, and SDK usage fields can change, so treat them as configuration inputs rather than permanent facts. Verify current details before making a purchasing or architecture decision.
Also revisit the estimate after:
- A prompt refactor, new few-shot examples, or a changed output schema
- A conversation-memory or summarization change
- A RAG index, chunking, retrieval, or reranking change
- A model upgrade, fallback-model change, or provider migration
- A sudden increase in latency, retries, or truncated responses
- A major change in request volume or user behavior
- A monthly cloud-cost review or new internal budget target
Keep a small dashboard with input tokens, output tokens, total cost, latency, retry rate, and quality results by request type. Review averages alongside percentiles so a small number of oversized requests does not disappear in the mean. When a change appears beneficial, test it on a fixed evaluation set and a representative traffic sample before making it the default.
The goal of LLM cost optimization is not the shortest possible prompt. It is a predictable system that spends tokens where they improve the result. Measure first, optimize the largest sources of waste, and recalculate whenever the underlying model, pricing, workload, or quality target changes.
