LLM Context Compression Techniques: From Summarization to LLMLingua

Compressing context is the cheapest way to make a model both smarter and roughly 70% cheaper, and almost nobody does it deliberately. Teams keep paying for 32k-token prompts that contain 4k tokens of actual signal, then complain that latency is bad and bills are high. The fix isn't a bigger model. It's compression that you control, measure, and version.

This piece walks through the four compression families that actually ship in production: extractive summarization, abstractive summarization, LLMLingua-style prompt compression, and structured distillation (bundles, semantic indexes). We'll cover where each one wins, what it costs in quality, and the measurement discipline that separates real compression from "I deleted some sentences and called it a day."

Why Raw Context Windows Are A Tax

A 1M-token context window sounds like an unlock. In practice it's a footgun. Three reasons most engineers ignore until the bill hits:

• Linear cost. Input tokens are billed linearly. A 200k-token prompt at $3/M input tokens costs $0.60 per call. Run that 50k times a day and you're at $30k/month before the model says a single word.
• Sublinear attention quality. Recall degrades with position. The "lost in the middle" problem is real and reproducible across Claude, GPT-4o, and Gemini. Padding context with low-signal tokens actively hurts answers.
• Latency tax. Time-to-first-token scales with prompt length. A 200k prompt adds 4-8 seconds of prefill latency on most frontier APIs. Users notice.

We covered the bill side of this in context compression and cost savings, and the recall side in the context rot problem. The punchline of both posts: every token you don't send is a token that can't hurt you.

The token-to-signal ratio nobody measures

Pull any production prompt from your logs. Count tokens. Now highlight the spans the model actually needed to answer correctly. The ratio is usually somewhere between 8% and 22%. Everything else is preamble, repetition, dead retrieval hits, JSON noise, and tool definitions that aren't called this turn. That gap is your compression headroom.

Extractive vs Abstractive Compression

Compression splits cleanly into two philosophies before you ever pick a tool:

Extractive: BM25, embedding similarity, sentence ranking

Extractive methods score each sentence (or chunk) against the question, then keep the top-k. Classical BM25 still beats dense embeddings on lexical-heavy domains (legal, medical, code). Dense retrieval wins on paraphrased questions. Hybrid (RRF fusion of BM25 + embeddings) wins almost everywhere.

The trap: extractive compression preserves wording but destroys structure. If your source has nested arguments where sentence 12 depends on sentence 3, ranking each in isolation will keep the conclusion and drop the premise. Use sliding windows or sentence-pair scoring to fix this.

Abstractive: cheap to think about, expensive to run

A small model (Haiku, Mini, Flash) at ~$0.25/M input tokens can compress a 100k-token document into a 5k-token summary for roughly $0.025. That sounds cheap until you multiply by every document, every refresh cycle, every user. Cache aggressively. Re-summarize only when source changes.

The hidden cost is drift: abstractive summaries paraphrase, and paraphrase loses precision. If your downstream model needs the exact phrase "Section 4.2(b)" or the exact number "€1,827.43," abstractive compression will silently round these into "section four" and "about 1,800 euros." That's a bug, not a feature.

LLMLingua and Prompt Compression In Production

LLMLingua (Microsoft Research, 2023) and its successors LongLLMLingua and LLMLingua-2 attack the problem differently. Instead of summarizing, they use a small language model (typically LLaMA-7B or a distilled variant) to score the perplexity of every token in the prompt, then drop the tokens the target model "would have predicted anyway."

The intuition: if GPT-4 can predict a token from context with high confidence, that token carries low information and can be dropped. The result reads like garbled English to humans but compresses 4x-20x with surprisingly small quality drops on QA, summarization, and code tasks.

Numbers from real deployments

Benchmarks we've seen replicate reliably:

• LongBench (multi-doc QA): 5x compression, 1-3 F1 point loss vs uncompressed
• GSM8K (math reasoning): 4x compression, near-zero accuracy loss
• HotpotQA: 8x compression, 4-7 F1 point loss (multi-hop hurts more)
• Code completion: Don't. Compression murders code accuracy.

When LLMLingua earns its keep

The sweet spot is RAG pipelines with 10-50 retrieved chunks where you can't predict in advance which chunks matter. Run retrieval, then LLMLingua-2 on the concatenated chunks before sending to the answer model. Cost of the compression pass is negligible (small model, batched), savings on the answer call are 3x-10x.

We've seen production setups push a 24k-token RAG prompt down to 4k tokens with no measurable quality regression on a 500-question eval set. At $15/M input on Claude Sonnet, that's $0.30 saved per call. At 10k calls/day, $3,000/day.

What LLMLingua does badly

Three failure modes worth knowing:

1. Structured outputs. If the model needs to copy exact JSON keys, exact identifiers, or exact numbers out of context, perplexity-based dropping will mangle them.
2. Instruction following. System prompts and few-shot examples compress poorly because the target model uses their exact phrasing as a template.
3. Long chains of reasoning. Multi-hop questions where the answer requires fragments from 5+ chunks tend to lose the connective tissue.

Structured Distillation: Bundles And Semantic Indexes

Extractive and abstractive methods compress prose. Structured distillation does something better: it replaces prose with schema. Instead of asking the model to re-read a meandering 8k-token Slack thread, you send it a 400-token bundle with the decision, the rationale, the dissenting view, and the open questions.

We've written the case for this approach in detail in why bundles beat threads and context window at scale. The short version: when you control the upstream context shape, compression ratios of 20x-30x are routine and quality goes up, not down, because you're stripping ambiguity rather than tokens.

The bundle shape that actually compresses

A good distillation bundle has six slots:

1. Decision: The conclusion in one sentence
2. Owner: Who is accountable, named
3. Rationale: 3-5 bullets, why this and not alternatives
4. Counterpoint: The strongest objection, captured honestly
5. Open questions: What's unresolved
6. References: Source URLs, doc IDs, dates

A 12k-token discussion compresses into ~500 tokens of bundle. The downstream model loses nothing it needed and gains the structure it can navigate with attention pointers rather than full re-reading.

Semantic indexes for codebases

For code, the equivalent of bundles is a semantic index: a per-file map of "what this exports, what it imports, what it's responsible for, what it depends on." A 4MB repo compresses into a 30k-token index. Agent decides which files to actually read in full based on the index. Token spend per agent task drops 60-80%.

Measuring Quality Loss From Compression

The cardinal sin in compression work: shipping ratios without quality measurement. "I got 8x compression" is not a result. "I got 8x compression with 0.3 F1 point loss on a 1,200-question eval" is.

The minimum viable eval

Before turning on any compression in production, you need:

• A frozen eval set of 200+ representative inputs with reference outputs (or LLM-as-judge rubrics)
• A baseline from the uncompressed pipeline scored against that eval set
• A delta tolerance agreed before you run the experiment ("we accept up to 2% accuracy loss in exchange for >5x cost reduction")

Metrics that catch real regressions

The 7-day production canary

Offline eval is necessary but not sufficient. Production traffic surfaces edge cases the eval missed. Run compression on 5% of traffic for 7 days. Compare downstream conversion, escalation rate, regeneration rate, and human override rate against the uncompressed control. If any of those move >1.5% adverse, roll back and investigate before scaling.

When NOT To Compress

Compression has real costs. Some of the time, the right answer is to send the full thing.

1. When the prompt is already small

A 2k-token prompt isn't worth compressing. The savings are pennies and you're adding a failure mode (the compressor itself). Floor: don't bother under ~4k tokens unless volume is extreme.

2. When precision matters more than volume

Legal review, medical decision support, financial calculations, anything regulated. The risk of a paraphrased "approximately" replacing an exact figure is not worth the cost saved.

3. When the model already caches it

Anthropic prompt caching, OpenAI context caching, and Gemini's implicit caching can drop the input cost of repeated context by 90% with zero quality loss. If your system prompt is stable and reused, cache it instead of compressing it. We covered the mechanics in our prompt-caching writeup.

4. When few-shot examples drive quality

Few-shot examples earn their tokens. Compressing them with LLMLingua tends to break the pattern-matching the model uses to imitate them. Leave them alone or rewrite them tighter by hand.

5. When you haven't measured the baseline

Compressing without a baseline isn't engineering, it's vibes. You'll ship a regression and only notice when the support queue lights up.

A Stack That Works In Production

Here's a layered approach we've seen ship at scale. Each layer addresses a different waste source:

1. Structured distillation for anything you produce yourself (system prompts, decisions, historical context). Highest ratio, lowest risk.
2. Prompt caching for stable prefixes. Free 90% discount on repeated tokens.
3. Hybrid retrieval (BM25 + embeddings + RRF) to keep retrieved context small in the first place. Don't retrieve 50 chunks and then compress; retrieve 8 good ones.
4. LLMLingua-2 on the final concatenated prompt as a safety net. 3x-5x on top of everything above.
5. Eval gate on every change. No compression ships without a number next to it.

Stacked, these techniques move a typical 24k-token RAG prompt into the 2-3k range. The same answer model. Same quality bar. Roughly one-tenth the bill.

Key Takeaways

• Context windows are a tax. Every unused token costs money and hurts recall.
• Extractive compression is safe and cheap; abstractive is powerful but drifts.
• LLMLingua-2 buys 4x-20x compression with low quality loss on QA and reasoning tasks.
• Structured distillation (bundles, semantic indexes) beats everything when you control the source.
• Measure before and after with a frozen eval set. No measurement, no shipping.
• Don't compress small prompts, regulated outputs, or few-shot examples without strong evidence.

FAQ

Does compression hurt latency or help it?

Helps it, usually. Shorter prompts mean shorter prefill. The compression step itself adds 50-300ms but saves 2-8 seconds of prefill on long prompts. Net win on anything >10k tokens.

Can I just use a smaller model instead of compressing?

Sometimes. Smaller models are cheaper but worse at reasoning. Compression keeps you on the better model while paying less. Use both: compress the prompt and route easy tasks to a cheaper model.

How does compression interact with prompt caching?

They're complementary. Cache the stable prefix (system prompt, tool defs). Compress the volatile tail (retrieved docs, user history). Don't compress cached prefixes, you'll bust the cache and pay full price.

What's the easiest compression win for a team that has done nothing?

Audit your system prompt. Most production system prompts are 1.5k-4k tokens of accumulated cruft. A one-hour rewrite by hand typically cuts 40-60% with zero quality loss. Free money, no models required.