Token Efficiency: How to Reduce LLM Costs Without Losing Quality

Every token your large language model (LLM) application sends or generates shows up on the bill, and token efficiency decides how much correct work you get for it. Output and reasoning tokens often cost more than input tokens, and a tighter answer format or a cached system prompt can cut your LLM token cost while you track answer quality.

This guide covers token pricing, a weighted efficiency metric, and six techniques you can use to cut consumption, including model routing, prompt caching, agentic loop bounds, and deterministic preprocessing.

What Token Efficiency Measures

Token efficiency is the amount of correct, evaluated work your system completes per token consumed, counting fresh input, cached input, and output. A system that resolves a support ticket in 900 tokens is more efficient than one that spends 4,000 on the same ticket, provided both answers are right. Only an evaluator can confirm both answers were right, which is why efficiency tracking belongs inside your LLM observability stack.

A large context window sets the upper limit on what the model accepts, and it says little about how efficiently you use that space. In retrieval-augmented generation (RAG) question-answering benchmarks, material that matches the topic without carrying the answer lowers accuracy as more of it reaches the model. You improve efficiency by putting the highest-signal tokens first and keeping the rest out.

Why Token Costs Are More Uneven than They Look

Input, cached, and output tokens have separate prices, and the spread varies by model. Claude Opus 5.5 and GPT-6 Astra charge more for output than input and discount cached input. Anthropic bills thinking as output, and OpenAI does the same with reasoning tokens. Your agent can incur output charges for reasoning before a tool call, even when you never see that work. GitHub folded these differences into one weighted metric called Effective Tokens (ET):

Effective Tokens = model multiplier × (1.0 × input + 0.1 × cache + 4.0 × output)

The formula weights fresh input, cache reads, and output by their relative cost, and GitHub's model multiplier normalizes across Haiku, Sonnet, and Opus tiers. For example, a request using 10,000 cached, 2,000 fresh input, and 500 output tokens with a model multiplier of one works out to 1,000 + 2,000 + 2,000 = 5,000 effective tokens. Fixed weights only approximate one model's price sheet, and a drop in ET won't always match the same dollar drop. In GitHub Agentic Workflows (gh-aw), GitHub has since replaced ET with dollar-denominated AI Credits. You can adjust the default weights to reflect your model's cache-read and output prices.

Where Token Waste Comes From

Token waste comes from prompts that grew after launch and from outputs nobody constrained. A single request rarely shows the waste. Daily call volume across chat features, retrieval pipelines, and agent workflows then multiplies it.

System Prompts That Only Grow

System prompts collect edge-case patches and almost never lose any. You can audit a production classification prompt for instructions that duplicate earlier patches. Running the shorter version against your test inputs shows whether the cuts change quality.

Full Documents Where Excerpts Would Do

Raw transcripts, whole documents, and every retrieved chunk end up in prompts because sending everything is easier than choosing. A full sales-call transcript lands in every call that asks about it, though a single question rarely needs more than a few paragraphs. The rest raises the input bill and dilutes the model's attention.

Output Formats Nobody Asked For

Preambles, restated questions, markdown tables, and reasoning fields all consume output tokens, which often cost more than input tokens. You can drop the reasoning fields from your JavaScript Object Notation (JSON) response when the task doesn't need them. Compare output length and latency before and after that change to see whether it helps your workload.

Agents That Re-Send Their History

Agent loops that re-send growing histories accumulate billed input faster than the history itself grows. Duplicate tool calls add billed input on top of the re-sent history. In a GitHub workflow that only scans local file changes, one run made 342 calls to search_repositories, 58 percent of all its tool calls. Tool-call traces, the core of agentic AI observability, expose a looping agent within one run.

Repeated Prefixes Without Caching

Identical system prompts, tool definitions, and reference blocks re-sent at full price are the content that provider caching discounts. Provider caches match on exact prefixes, so a changing timestamp or user identifier near the top of the prompt can prevent reuse of the content that follows. Freezing the run timestamp once per run instead of per step keeps the top of your prefix stable.

How to Measure Token Efficiency

Token efficiency equals completed tasks, or an evaluator score from your AI evaluation metrics, divided by effective tokens for each task type. Two tasks that each send 800 input tokens and return the same correct answer show why weighting is necessary. Task A responds in 200 output tokens and Task B in 800. The weighted totals are 1,600 effective tokens for A and 4,000 for B, so B uses 2.5 times the effective tokens for identical work. Producing that comparison from production traffic depends on how you instrument calls:

• Attribution tags: Each call carries session, agent, and task-type identifiers as OpenTelemetry (OTel) span attributes, so cost rolls up to the workflow that caused it.
• Weighted counts: Raw totals treat a cache read and an output token as equal, which hides where spend goes, while ET weighting breaks spend out by token type.
• Output-to-input ratio: A rising ratio is an early signal of verbosity drift, and tracking it per workflow keeps one global average from masking input-heavy and output-heavy patterns.
• Per-task baselines: Per-session numbers penalize tasks that are long by design.

Per-task baselines also protect you from false alarms, because average ET can rise when your workload shifts toward larger pull requests even if your token-reduction changes still work. Comparing ET within the same task type and input size before and after a change separates a workload shift from a regression. Those comparisons run on per-call attribution, and Coralogix, a cross-stack observability platform, collects it in AI Center. The token tracking there logs every AI transaction, tracks token usage by session and agent, and correlates cost with AI performance.

Practical Techniques to Reduce Token Consumption

Model routing, provider-side caching, ranked retrieval, output limits, loop bounds, and code-side pre-processing each cut token consumption. Routing and output limits often change through configuration alone, while caching, retrieval budgets, and loop bounds need design work. The techniques stack, so one request can combine savings from routing, caching, and a length cap.

Route Each Task to the Smallest Model That Passes Evals

Extraction, classification, and routing are high-volume tasks that smaller models often handle, and your evaluation results show whether they do. Haiku 5.5 costs less per token than Opus 5.5. A per-class eval, such as exact match for classification and field-level checks for extraction, tells you which tier each workload needs before you switch.

Cache Stable Prefixes on the Provider Side

Anthropic's cache_control parameter marks cacheable content blocks, or goes once at the top level for automatic caching, with a ttl that sets the cache lifetime. Cache writes cost more than ordinary input, while discounted reads can reduce the cost of repeatedly sending long prefixes. OpenAI caches automatically, with model-specific lifetime controls through prompt_cache_retention or prompt_cache_options.ttl. Your savings depend on the write price, retention period, and how many reads occur before the entry expires.

Send Ranked Evidence Under a Hard Token Budget

Retrieving broadly, reranking, and keeping only the top chunks under a fixed per-request token cap is the core of context engineering. The cap limits how many chunks reach the model, and compression handles any chunk that still exceeds its share. In long-document RAG benchmarks, answer quality rises and then falls as retrieved chunks accumulate, and the peak sits at a different context size for each model. Reranking plus query-aware compression can reduce the tokens you send, with cost savings that depend on your model's input pricing. Progressive retrieval goes further by loading summaries or metadata first and fetching full text only when the model asks for it.

Constrain Output Length and Shape

max_tokens on Anthropic and max_output_tokens on OpenAI's Responses API set hard ceilings, and on OpenAI reasoning models that ceiling includes hidden reasoning tokens. A cap set too tight cuts the answer off mid-thought and still bills the generated tokens, so size it to the task. Naming keys in a JSON schema leaves little room for preambles, and lower effort settings (reasoning.effort on OpenAI, output_config.effort on Anthropic) trim thinking on simple tasks.

Bound Agentic Loops and Load Tools on Demand

Each agent loop needs an explicit iteration cap and exit condition. Three frameworks expose the cap as a configuration setting:

• LangGraph: The recursion_limit setting caps super-steps (rounds of node executions), and you can set that ceiling for each run.
• CrewAI: max_iter bounds iterations.
• LangChain: Two middleware, ModelCallLimitMiddleware and ToolCallLimitMiddleware, bound model and tool calls.

Iteration caps bound how long an agent runs, while Anthropic's deferred tool loading, controlled by the defer_loading flag, keeps tool definitions out of the context window until tool search discovers them. When an agent goes off track, starting a fresh context with a sharper prompt can save you from paying to reread a thread full of corrections on every turn.

Pre-Process Data with Code Before It Reaches the Model

Filtering, projecting, and summarizing with deterministic code avoids model calls for that preprocessing. Three approaches shrink the prompt before the model sees it:

• Structured Query Language (SQL) views: A SQL view returns only the columns your task needs.
• Field filters: Filtering with jq keeps the fields your prompt uses.
• Structural search: An ast-grep query matches code by syntax tree, so the agent reads matching constructs instead of every line a text grep returns.

Each approach moves selection work out of the model's context. Measuring ET before and after the change shows how much the preprocessing saved.

Monitoring Token Efficiency in Production

The per-task metrics above need alerts attached in your LLM monitoring tools, because efficiency gains erode as prompts pick up instructions, models change, and agents gain tools. Start with thresholds that flag sustained changes in the output-to-input ratio and cost per resolved task relative to trailing averages, with evaluation windows and tolerances set for each workload. The generative AI (GenAI) semantic conventions define token attributes such as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. OpenTelemetry for AI traces then carry the session and agent identifiers that turn a spike into a named culprit.

Code agents are worth a separate dashboard, since input tokens can dominate their cost even with caching. Their telemetry includes:

• Claude Code: claude_code.token.usage and claude_code.cost.usage carries enough attributes for per-developer attribution through OTel export alone.
• Codex: The codex.sse_event OpenTelemetry event carries token counts when a response completes.
• Gemini CLI: The CLI emits the gemini_cli.token.usage metric.

Each tool reports token counts by type, which ET weighting needs.

Token Efficiency Is a Cost Architecture Decision

Model selection, caching, retrieval, output constraints, and loop design interact, and cutting 38.4 percent of tool-output tokens raised billed cost by 6.8 percent in one measured agent workload. A raw token count alone can mislead. The ET formula gives your team a common unit for comparing these choices, provided its weights reflect the models you use. Computing ET per completed task for your highest-volume workflow makes a good first step, and you can rerun it after every prompt, model, or tool change. You can send your agent and LLM traces to Coralogix and read token cost on the same traces that show whether a workflow produced a correct output.

Start a free 14-day Coralogix trial with an eight-unit quota and no credit card. The trial lets you compare token cost across your own workflows.

Frequently Asked Questions About Token Efficiency

What causes token efficiency to drop?

A declining token efficiency score usually points to prompt bloat, verbose output defaults, or an agent chaining more steps than the task needs. The score equals completed tasks (or a quality score) divided by ET, with weights for the model and each token type, so any decline means fewer correct tasks per weighted token.

What is the difference between prompt caching and response caching?

Prompt caching is a provider feature that reuses the computed state of an unchanged prefix while still generating a fresh answer, and you control it through Anthropic's cache_control or OpenAI's prompt_cache_options and prompt_cache_retention. Response caching lives in your application and returns a stored answer for an identical or semantically similar query without calling the model. Semantic response caches add a risk that prompt caching doesn't, because a near-match can return the answer to a different question.

How much does 1,000 tokens cost in AI?

Cost depends on the model and the token type. On Claude Opus 5.5, 1,000 input tokens cost $0.004, 1,000 output tokens cost $0.02, and 1,000 cache-read tokens cost $0.0002. GPT-6 Astra charges $0.01, $0.05, and $0.001 for the same counts. Both providers change prices by model, so confirm current rates before budgeting.

Which programming language is most token-efficient?

One analysis of RosettaCode programs using a GPT-4 tokenizer found a 2.6 times difference between Clojure, the most efficient language in its main comparison, and C, the least efficient. Token counts depend on both the programming language and the tokenizer used to encode it. For code agents, filtering which files and functions enter context is a lever you control regardless of language.

Does improving token efficiency reduce output quality?

Not necessarily, and a quality score paired with ET shows whether a cut saved money or pushed the cost into retries. Concise chain-of-thought prompting cut response length by nearly half with little overall effect, though the penalty reached 27.69 percent on GPT-3.5 math problems.