Semantic data preprocessing for LLM token optimization
Your AI is reading your documents the way a bored intern reads a contract. Here's what to do about it.
Pick up any book… I’ll wait.
Got it? Good, now open to a random page and read only the first and last word of each line, top to bottom.
You got somewhere between “vaguely aware of the topic” and “completely lost.” That’s what happens when you strip a signal to its edges and hope the meaning survives. Sometimes it does. Mostly it doesn’t.
Here’s the uncomfortable truth: when you send a long document to an LLM without preprocessing it, you’re not doing much better. You’re feeding it every word — the meaningful ones, the filler ones, the boilerplate ones, the ones that exist purely because someone wrote a template in 2019 and nobody deleted it. The model reads all of it. You pay for all of it. Every single token.
For a one-off query, that’s fine. At SME scale AI-assisted analytics, customer service, document review, compliance checking, knowledge retrieval across thousands of requests a month well… that’s a line item on your P&L that grows quietly until someone notices it in a board report.
This article maps the full landscape of semantic compression tools, explains how to choose between them, and gives you a framework to make the decision.
When “cleaning your data” stops being enough
Most teams start their token-optimization journey in the right place. Strip HTML tags. Remove boilerplate headers and footers. Deduplicate. Normalise whitespace. Pack tabular data into CSV instead of verbose JSON.
These are good first steps. For structured data like databases, spreadsheets, API responses they often solve the problem entirely. A well-formatted CSV is one of the most token-efficient ways to pass tabular information to a model.
But if your prompts still exceed cost limits, latency targets, or context window boundaries after all that tidying, you’ve hit the operational ceiling of structural cleanup. The problem is no longer the formatting. The problem is the content itself.
Unstructured text like emails, reports, contracts, meeting transcripts, support tickets, product documentation carries enormous amounts of redundant, low-signal information baked into the way humans write. We repeat ourselves. We use five words where one would do. We write introductory paragraphs that say nothing. This is normal human communication. It is catastrophically inefficient for token billing.
Cleaning doesn’t fix this. You need something fundamentally different: semantic compression.
What semantic compression actually is
The distinction that matters:
Structural optimization changes how information is packaged. Same content, different container. Smaller label, same data. CSV instead of JSON.
Semantic compression changes how much information is conveyed, while preserving which information survives. It reconstructs meaning using fewer tokens. It is a transformation of the signal, not the packaging.
The closest analogy comes directly from the research: JPEG compression for images. When you save a photograph as a JPEG, you’re not resizing it you’re discarding the parts of the image your eye is least likely to notice, while preserving the parts that carry the most perceptual value. The result is a smaller file that looks like the same image, because the meaningful signal is intact.
Semantic compression does this for text. Done well, it is not lossy in any way that matters operationally. Done badly, you’re back to reading the edges of pages.
Compression ratios of 10× to 20× are achievable for some workloads with performance that is often indistinguishable from the uncompressed input, and in some cases, actually better, because you’ve removed the noise that was confusing the model.
At an SME spending several thousand pounds per month on API costs, a 5× compression on document-heavy workloads is a meaningful budget conversation.
The compression landscape at a glance
Before going deeper into any one area, here’s the full map. Think of these tools as sitting on a spectrum between two poles: heavyweight and model-driven (high fidelity, higher cost, slower) and lightweight and heuristic (faster, cheaper, no model dependency, less nuanced). The table below is your reference throughout:
Comparison of compression approaches
Approach | Compression | Fidelity | Compute cost | Best for |
|---|---|---|---|---|
LLMLingua series | Medium–high | Very high | Medium | RAG, reasoning, enterprise |
Keyword-based compression | Medium–high | Medium | Zero | Analytics, summaries, edge |
BoilNet / Boil-the-Text | Medium–high | Medium | Zero | Analytics, summaries, edge |
LLM-powered summarisers | Low–medium | High | Medium | Variable input, clear task |
Encoder pruning (SCONET, SC) | Low–medium | High | Low–medium | Domain-specific pipelines |
Embedding-based compressors | Medium–high | High | Medium | Redundant long-form docs |
Sparse attention / context pruning | Low–medium | Very high | High | Self-hosted, long-context |
Rule-based filters (RAKE, TextRank) | Low | Medium–high | Low | Domain-tuned, high-volume |
Industry built-in compressors | Low–medium | High | Medium | Existing LangChain/API stacks |
Hybrid and custom methods | Variable | Tunable | Variable | Specific domain at scale |
The tools in depth
LLMLingua series — state of the art in model-driven compression
LLMLingua comes out of Microsoft Research. The core idea is elegant: train a small, fast encoder model to predict which tokens in a prompt are critical for the downstream task. Then remove the ones that aren’t. You don’t need a large model to make this judgment — a small, specialised model can identify token importance with high accuracy, and its computational cost is easily justified by the savings on the expensive model downstream.
There are three generations:
LLMLingua v1 is the foundation. Up to 20× compression with minimal performance loss on reasoning tasks. For most knowledge-retrieval and summarisation workloads, the compressed output performs within a few percentage points of the full input. Start here to prove the concept.
LongLLMLingua addresses the “lost in the middle” problem — research has repeatedly shown that LLMs attend better to information at the beginning and end of a context window than to content buried in the middle. LongLLMLingua reorganises and compresses long documents to counteract this, improving downstream performance by approximately 17% at 4× compression. If you’re running any RAG pipeline, this is the variant to know.
LLMLingua-2 is the production-ready successor: 3–6× faster than v1, more robust across varied input types, more consistent. If you’re building something that needs to run reliably at scale across multiple domains, reach for this one.
Best for: Any workload where output quality is non-negotiable — compliance analysis, legal document review, technical support triage, financial report summarisation.
Keyword-based semantic compression — the lean approach
No model. No training data. No GPU. No dependencies. These tools run directly in a browser, on an edge device, on a laptop with no internet connection. The philosophy is closer to Lean manufacturing than machine learning: identify the minimum information required to preserve meaning, remove everything else.
For some workloads, this approach achieves 10–20× compression by extracting semantically dense keywords and their immediate context, discarding filler, repetition, and low-frequency connective tissue. The tradeoff is fidelity, fine-grained reasoning tasks where exact phrasing, legal nuance, or causal chains need to survive compression intact are not candidates here.
Think of it as giving your team commander’s intent: the essential information they need to act, without the full operational briefing. Appropriate for many decisions. Not appropriate for the ones where the detail is the mission.
Best for: Internal analytics summaries, first-pass document triage, privacy-sensitive workloads where data cannot leave your environment, reporting pipelines where “good enough” is genuinely good enough.
BoilNet / Boil-the-Text — statistically grounded lightweight compression
BoilNet draws on TF-IDF; a statistical method for estimating how important a word is within a document relative to a larger corpus to distinguish high-signal terms from background noise. It removes low-information tokens, retains high-value keywords with short surrounding phrases that preserve meaning, and optionally reweights text segments by aggregate importance.
Compared to ad-hoc keyword-based approaches, BoilNet is more formalised and grounded in well-understood statistical principles, aaand more predictable, producing consistent results across documents. It retains all the operational advantages of lightweight tools: model-free, language-agnostic, suitable for edge devices and offline pipelines. It sits in the middle ground between raw heuristic filtering and full ML-based NLP: more disciplined than the former, simpler than the latter.
Best for: Analytics and BI pipelines, document classification, preprocessing stages before more expensive downstream systems, large-scale text filtering where cost and latency matter more than nuance.
Limitations: Scores words, not meaning. Can miss nuance, tone, or logical relationships. Context preservation is shallow compared to model-based approaches.
LLM-powered summarisers — using one model to make another cheaper
Sometimes the most practical move is to use a fast, low-cost LLM with tightly scoped instructions (“extract only facts relevant to X”) as a preprocessing step before your primary model. It may sound circular; it’s highly effective when your definition of “relevant” is stable and well-specified. The output can be structured JSON, bullet points, key-value pairs, rather than just shorter prose. You’re compressing through controlled interpretation.
Best for: Multi-step AI pipelines, domain-specific extraction (contracts, support tickets, reports), situations where meaning matters more than exact phrasing.
Limitations: Additional API call adds latency and cost. Risk of hallucination or omission if prompts are poorly defined. Less deterministic than statistical methods.
Encoder pruning (SCONET, Selective Context) — a learned relevance filter
Tools like SCONET and Selective Context use smaller, efficient BERT-family encoder models to score token importance and drop low-value content. Each token receives an importance score; low-scoring tokens are removed; the remainder is stitched into a shorter sequence. More semantically aware than keyword or TF-IDF methods, still relatively lightweight compared to full LLMs, and deterministic once trained. Think of it as a trained relevance filter, not a reasoning system.
Tabular data has the same problem: too many columns, many of them redundant. Cutting them before modelling is covered in The Power of Unsupervised Feature Selection.
Best for: Domain-specific pipelines where you control the data, preprocessing before classification or retrieval, situations requiring repeatability with some semantic awareness.
Limitations: Requires model inference, though cheaper than LLMs. Needs tuning or fine-tuning for best results. Less flexible than prompt-based LLM compression.
Embedding-based compressors — removing redundancy, not rewriting
These methods operate in vector space: convert chunks of text into embeddings, measure similarity between chunks, remove or merge semantically redundant segments. Common implementations include FAISS-based clustering, centroid summarisation, and models like ColBERT-XR. Excellent at removing duplicate or near-duplicate content; scales well to large corpora; particularly effective in retrieval systems. The key distinction from summarisation: this approach selects and filters rather than rewrites it doesn’t alter the surviving text.
Best for: RAG pipelines with large document stores, deduplication before indexing, knowledge base optimization.
Limitations: Requires embedding computation (local or API-based). Can lose narrative flow or logical ordering. Works best when redundancy, not complexity, is the primary problem.
Sparse attention and context pruning — architectural optimization
Rather than shrinking the input, these approaches change how models process it. MInference and Recurrent Compression Transformers optimise attention patterns at the model level, enabling efficient handling of much longer sequences without preprocessing or altering the original text at all. The tradeoff: requires custom or self-hosted models, not available in most commercial APIs, and significantly more complex to implement and tune.
Best for: Long-context LLM deployments, self-hosted or open-source model stacks, high-throughput inference systems.
Limitations: Not accessible via standard commercial APIs. High implementation overhead. Overkill for most SME contexts unless you’re already running self-hosted infrastructure.
Rule-based semantic filters (RAKE, TextRank) — the underrated workhorse
RAKE and TextRank are classical NLP algorithms that identify important keywords and sentences using graph-based ranking. Fast, cheap, interpretable and when tuned for a domain with predictable vocabulary (legal, financial, medical, logistics), surprisingly effective. A well-tuned heuristic filter often outperforms a general-purpose model. Compression: 1.5–5×. This is where many SMEs should start: no dependencies, no model management, and a meaningful reduction in token spend from day one.
Best for: High-volume domain-specific pipelines, situations where interpretability matters, teams that need something working this week rather than next quarter.
Industry built-in compressors — zero-integration-overhead options
Several commercial platforms now ship compression utilities directly. LangChain includes LLMChainExtractor, EmbeddingFilter, DocumentReducer, and ContextualCompressionRetriever — worth evaluating if you’re already in that ecosystem. Anthropic’s Claude supports contextual compression via system prompt instruction, where you instruct the model to compress documents before reasoning over them; not the most token-efficient approach since you still pay for the input, but there’s zero integration overhead. OpenAI supports similar patterns via embedding clustering and reranking.
Best for: Teams already committed to a specific platform who want compression without adding a new dependency to the stack.
Hybrid and custom methods — when the off-the-shelf wheel needs new spokes
For domain-specific workloads at scale let say legal documents, medical records, financial filings, security logs, many teams end up building pipelines that combine keyword extraction, clustering, relevance scoring, and domain-specific rules. When your data has predictable structure and your task is well-defined, a custom pipeline built from these components will outperform any off-the-shelf tool. This isn’t a recommendation to reinvent the wheel; it’s recognition that the wheel sometimes needs new spokes for specific terrain. Expect this to be a phase-two conversation once you’ve validated compression with a simpler approach first.
Deep focus: embeddings-based pruning
I want to focus on this separately becuse embeddings-based pruning offers the highest compression-to-fidelity ratio currently available, and is the approach most likely to be underestimated in an SME context.
Here’s the core idea without requiring a machine learning background:
Imagine a twenty-page policy document. Somewhere in it, three paragraphs say essentially the same thing in three different ways because three people drafted different sections in 2021, 2022, and 2023 and nobody ever consolidated them. The model reads all three. You pay for all three. Semantically, they contribute one unit of information.
Embeddings-based pruning catches this:
Convert each sentence or paragraph into a mathematical vector representing its meaning in geometric space.
Measure similarity between vectors. Sentences that are close in this space carry similar meaning.
Remove sentences that are close enough to others to be redundant, keeping only the most representative member of each cluster.
Pass the distilled result, covering all the conceptual ground of the original, without the repetition to the primary model.
Compression ratios of 80–95% are achievable in documents with high semantic redundancy. Policy documents, regulatory filings, corporate communications, and lengthy email threads almost universally exhibit this property.
The tradeoffs: requires an embeddings model (small, efficient options like Sentence-Transformers run comfortably on a standard CPU); not fully deterministic; slower than keyword filtering, though still faster than the primary LLM call in most pipeline architectures.
This is the approach to reach for when you need the highest compression-to-fidelity ratio and the document has been written by more than one person over more than six months. Those two conditions reliably predict high semantic redundancy.
How to choose: a decision framework
The fundamental question is not “which tool is best?” It’s “what are the tradeoffs I’m willing to accept, and what does my workload actually require?” Four axes matter:
Axis 1: Fidelity requirements
“If compression removes something important, what happens?” If a human catches it in review lightweight tools are sufficient. If it triggers a wrong decision or compliance risk you need model-driven compression with measurable fidelity guarantees.
Axis 2: Data sensitivity
“Can this data leave our environment?” If documents contain PII or regulated data, any cloud-based compression tool introduces a new data-processing relationship. Model-free approaches running locally are preferable.
Axis 3: Compute budget
“What does my team have capacity to run and maintain?” Model-driven tools require deployment, version management, and retuning. For a lean team without dedicated ML engineering, heuristic approaches carry no such overhead.
Axis 4: Workload economics
“What do the numbers actually say?” Take your monthly API spend on document-heavy workloads. Apply a conservative compression ratio. Subtract implementation and compute costs. If payback is under three months, the decision is obvious.
The projects that fail are not the ones that chose the wrong tool. They’re the ones that implemented a heavyweight solution for a lightweight problem or a lightweight solution for a problem that required precision.
Semantic compression as a production discipline
The way most organisations think about AI cost management today reminds me of early Lean manufacturing. Teams focused on the obvious waste: off-spec products, delivery delays, visible inventory. It took time to recognise that the subtler wastes overprocessing, unnecessary motion, excess information were often larger in aggregate.
Token waste is the information equivalent of overprocessing. You’re running a sophisticated, expensive process on inputs that haven’t been optimised for it. Every redundant sentence in your prompt is the equivalent of machining a part to a tighter tolerance than the design requires. You paid for precision you didn’t need.
Semantic compression is the application of Lean thinking to AI pipelines. What is the minimum information required to achieve the required output? Deliver exactly that — nothing more.
This is not a one-time optimization. As your AI usage scales, your preprocessing pipeline needs to scale with it. The organisations that treat this as a production discipline from the beginning will have a meaningful structural cost advantage over those that treat it as an afterthought.
The tools exist. The techniques are mature. The economics are straightforward. The only variable is when you start.
Key takeaways
Structural cleanup (stripping HTML, deduplication, CSV formatting) has a ceiling. When you hit it, the problem is content, not packaging.
Semantic compression can achieve 10–20× token reduction with minimal loss of operationally relevant information sometimes with improved model performance.
The right tool depends on four factors: fidelity requirements, data sensitivity, compute budget, and workload economics. Match the tool to those constraints, not to what sounds most sophisticated.
For most SMEs, the right starting point is either a rule-based filter (RAKE/TextRank) or keyword-based compression zero dependencies, immediate returns, no model management overhead.
For workloads with high semantic redundancy and non-negotiable fidelity, embeddings-based pruning or LLMLingua-2 are worth the additional implementation effort.
Treat this as a production discipline, not a one-time fix. Build preprocessing into the pipeline from the start.
Want help mapping this to your specific workload?
Every SME AI pipeline is different. The right compression approach depends on your documents, your risk tolerance, your team’s capacity, and your cost curve. If you’d like a structured conversation about where the leverage actually is in your stack, I’m happy to talk.
Still paying for tokens your model didn't need? Caught between spiralling API costs and the pressure to scale AI across your business? Watching your monthly model spend grow while your team struggles to identify where the waste actually lives in the prompts, the pipeline, the documents, or all three?
It's time to stop feeding your AI documents it doesn't need to read!
I help SMEs audit their AI pipelines, identify the highest-leverage compression opportunities, and implement the right preprocessing strategy; whether that’s a lightweight rule-based filter you can deploy this week, an embeddings-based pruning layer for your redundant long-form documents, or a full LLMLingua integration for compliance-grade workloads. All tailored to your data sensitivity requirements, your team’s capacity, and your actual cost curve not a generic blueprint.
I deliver lean, measurable results: reduced token spend, faster inference, and pipelines built to scale. Not just a report. Not just a recommendation. A working implementation with clear before-and-after economics.
Share your current AI workload and cost challenges, and I’ll explain exactly where the compression leverage is in your stack, which techniques apply, what compression ratios are realistic for your document types, and what the payback period looks like. I’ll outline the implementation path, detail my hands-on approach from pipeline audit to production deployment, and introduce the tools and partners I bring to every engagement.
Your initial discovery consultation is completely free. Don’t wait until your AI bill becomes a board conversation.
Contact me today to start your journey from reactive firefighting to proactive, data-driven dominance.