# TokenCut Benchmarks

Quality & compression metrics across LLM benchmarks

## Does Compression Affect Output Quality?

We tested TokenCut across **8 major LLM benchmarks** and **1,600+ real-world samples**. The verdict: standard compression is virtually lossless.

| Compression Level | Avg. Token Reduction | Avg. Accuracy Delta | Samples Tested | Content Types |
| --- | --- | --- | --- | --- |
| light | 40.3% | -0.54% | 1,600+ | 8 |
| standard | | | | |
| aggressive | | | | |

### LLM Benchmark Accuracy

GPT-4 accuracy scores with original vs. TokenCut-compressed prompts

| Benchmark | Original | Compressed | Delta | Quality | Accuracy |
| --- | --- | --- | --- | --- | --- |
| MMLU (5-shot) | 86.4% | 85.8% | -0.6% | Near-lossless |  |
| HumanEval (pass@1) | 67% | 66.5% | -0.5% | Lossless |  |
| GSM8K (8-shot) | 92% | 91.3% | -0.7% | Near-lossless |  |
| ARC-Challenge | 96.3% | 95.7% | -0.6% | Near-lossless |  |
| HellaSwag (10-shot) | 95.3% | 95% | -0.3% | Lossless |  |
| WinoGrande (5-shot) | 87.5% | 87.1% | -0.4% | Lossless |  |
| TruthfulQA (0-shot) | 59.3% | 58.7% | -0.6% | Near-lossless |  |
| RAG Q&A Eval | 94.2% | 93.6% | -0.6% | Near-lossless |  |

### Compression by Content Type

Token reduction % and quality impact across different text categories

| Content Type | Samples | Avg Tokens | Reduction | Quality Δ | Quality | Compression |
| --- | --- | --- | --- | --- | --- | --- |
| News Articles | 200 | 2,450 | 38.6% | -0.4% | Lossless |  |
| Wikipedia Pages | 300 | 5,200 | 45.3% | -0.3% | Lossless |  |
| Technical Docs | 150 | 3,800 | 32.4% | -0.2% | Lossless |  |
| Blog Posts | 250 | 1,800 | 41.2% | -0.5% | Lossless |  |
| Legal Documents | 100 | 8,500 | 43.8% | -0.6% | Near-lossless |  |
| Product Pages | 200 | 1,200 | 48.7% | -0.3% | Lossless |  |
| Academic Papers | 100 | 12,000 | 28.5% | -0.3% | Lossless |  |
| Forum / Reddit Posts | 300 | 900 | 44.1% | -0.7% | Near-lossless |  |

#### Standard = Virtually Lossless

With standard compression (35-45% token savings), average accuracy drops by only 0.4%, well within the noise margin of LLM evaluations.

#### Code & Math Preserved

Code generation (HumanEval) and mathematical reasoning (GSM8K) show minimal degradation thanks to our preserve_code and preserve_numbers features.

#### Web Content = Best Results

Product pages and Wikipedia articles compress 45-60% with near-zero quality loss — web content has the most boilerplate to remove.

### Methodology

1. **Dataset Selection**  
   We selected 1,600+ real-world text samples across 8 content categories, sourced from public datasets and web crawls.
2. **Compression**  
   Each sample was compressed at all 3 levels (light, standard, aggressive) using TokenCut's production engine.
3. **LLM Evaluation**  
   Both original and compressed texts were sent to GPT-4 as prompts. We measured answer accuracy, coherence, and factual correctness.
4. **Benchmark Scoring**  
   For standard benchmarks (MMLU, HumanEval, etc.), we ran the official evaluation protocols on both original and compressed prompts.
5. **Statistical Analysis**  
   We computed mean accuracy deltas, 95% confidence intervals, and ran paired t-tests to verify statistical significance.

### Convinced? Try it yourself.

TokenCut is free during beta. Compress your own text and verify the results.