So you've switched to TOON format and cut your tokens in half—nice! But what if I told you we can squeeze even more out of it? With minification, you can push savings to 60-70%. If you're running a production LLM app where every token counts (literally), these techniques can make a real dent in your costs.
Token Savings Comparison
What is TOON Minification?
Minification removes unnecessary whitespace, optimizes field names, and applies compression techniques while maintaining data integrity. It's similar to JavaScript minification but optimized for LLM tokenization.
✅ What Gets Removed
- • Extra whitespace and newlines
- • Comments and documentation
- • Unnecessary indentation
- • Optional punctuation
✅ What Stays Intact
- • All data values
- • Data structure and hierarchy
- • Field relationships
- • Semantic meaning
Technique 1: Whitespace and Indentation Removal
Standard TOON uses 2-space indentation for readability. Minification removes all non-essential whitespace.
📄 Standard TOON (23 tokens)
# User profile with nested address user: id: 12345 name: "Sarah Mitchell" email: [email protected] address: street: "123 Main St" city: Boston zip: "02101" active: true
⚡ Minified TOON (16 tokens - 30% reduction)
user:id:12345;name:"Sarah Mitchell";email:[email protected];address:street:"123 Main St";city:Boston;zip:"02101";active:true
Whitespace Optimization Rules:
- • Remove all indentation (2-4 tokens per object)
- • Replace newlines with semicolons for nesting
- • Keep single space after colons only when needed for parsing
- • Remove blank lines (1 token each)
Technique 2: Field Name Abbreviation
Long descriptive field names improve readability but consume tokens. For production use, abbreviate fields while maintaining meaning.
📄 Descriptive Field Names (45 tokens)
customers[3]{customer_id,first_name,last_name,email_address,phone_number,subscription_status}:
1001,Sarah,Mitchell,[email protected],555-0101,active
1002,Michael,Chen,[email protected],555-0102,trial
1003,Jennifer,Kumar,[email protected],555-0103,active⚡ Abbreviated Field Names (32 tokens - 29% reduction)
cust[3]{id,fn,ln,em,ph,stat}:
1001,Sarah,Mitchell,[email protected],555-0101,active
1002,Michael,Chen,[email protected],555-0102,trial
1003,Jennifer,Kumar,[email protected],555-0103,activeField Abbreviation Best Practices:
- • Use common abbreviations:
id,nm,addr,qty,amt - • Maintain consistency across dataset
- • Document abbreviations in system comments (not in prompt)
- • LLMs understand abbreviations with context
Technique 3: Optimal Delimiter Selection
Default comma delimiter is readable but not always most efficient. Choose delimiters based on your data patterns.
Delimiter Token Costs
| Delimiter | Token Cost | Best For |
|---|---|---|
| , | 1 token | Simple data without commas |
| | | 1 token | Data containing commas (addresses) |
| \t (tab) | 1 token | Copy-paste from spreadsheets |
| ; | 1 token | Nested inline structures |
Example: Using Pipe for Address Data
# Pipe delimiter avoids quote wrapping
addr[2]{st,city,zip}|:
123 Main St, Suite 100|Boston|02101
456 Oak Ave, Apt 5B|Seattle|98101Technique 4: Remove Comments and Documentation
Comments are helpful during development but should be stripped in production prompts.
📄 With Comments (35 tokens)
# Product catalog for Q4 2024
# Updated: 2024-12-15
# Total products: 3
products[3]{id,name,price}:
# Electronics category
1,Laptop,999.99
2,Mouse,29.99
# Accessories
3,Keyboard,79.99⚡ Without Comments (18 tokens - 49% reduction)
products[3]{id,name,price}:
1,Laptop,999.99
2,Mouse,29.99
3,Keyboard,79.99Rule: Remove all lines starting with # before sending to LLM. Keep metadata in your application code, not in the prompt.
Technique 5: Inline Small Nested Objects
Objects with 2-3 fields can be inlined to save tokens on newlines and indentation.
📄 Nested Format (27 tokens)
order:
id: 12345
customer:
name: "Sarah"
email: [email protected]
total: 149.99
status: shipped⚡ Inlined Format (19 tokens - 30% reduction)
order:id:12345;cust:{nm:"Sarah";em:[email protected]};tot:149.99;stat:shippedWhen to Inline:
- • Objects with 2-4 simple fields
- • No further nesting inside
- • When readability isn't critical (production prompts)
- • Use semicolons to separate inline fields
Technique 6: Array Structure Optimization
Structured arrays with field definitions are efficient, but simple arrays can be even more compact for primitive values.
📄 Structured Array (18 tokens)
tags[5]{name}:
javascript
react
typescript
nextjs
tailwind⚡ Simple Array (12 tokens - 33% reduction)
tags[5]:javascript,react,typescript,nextjs,tailwind
Array Format Decision:
- • Use simple arrays: For primitives (strings, numbers, booleans)
- • Use structured arrays: For objects with multiple fields
- • Simple arrays save ~5-8 tokens per array declaration
Real-World Example: E-commerce Order
📄 Standard TOON (78 tokens)
# Order details for processing order_id: ORD-2024-001 customer: customer_id: 12345 name: "Sarah Mitchell" email: [email protected] phone: "555-0101" items[3]{product_id,product_name,quantity,unit_price,subtotal}: 101,Laptop,1,999.99,999.99 102,Mouse,2,29.99,59.98 103,Keyboard,1,79.99,79.99 shipping_address: street: "123 Main Street" city: Boston state: MA zip: "02101" order_total: 1139.96 order_status: processing
⚡ Minified TOON (34 tokens - 56% reduction)
oid:ORD-2024-001;cust:{id:12345;nm:"Sarah Mitchell";em:[email protected];ph:"555-0101"};itm[3]{pid,nm,qty,pr,sub}:101,Laptop,1,999.99,999.99;102,Mouse,2,29.99,59.98;103,Keyboard,1,79.99,79.99;ship:{st:"123 Main Street";city:Boston;st:MA;zip:"02101"};tot:1139.96;stat:processingMinification Breakdown:
Cost Impact Analysis
Token reduction directly impacts API costs. Here's the financial impact across different usage volumes:
| Monthly Volume | JSON Cost | Standard TOON | Minified TOON | Savings |
|---|---|---|---|---|
| 100K requests | $150 | $75 | $50 | $100/mo |
| 500K requests | $750 | $375 | $250 | $500/mo |
| 1M requests | $1,500 | $750 | $500 | $1,000/mo |
| 5M requests | $7,500 | $3,750 | $2,500 | $5,000/mo |
Based on GPT-4 Turbo pricing ($0.01 per 1K input tokens) with average 150 tokens per request.
Automated Minification Tools
Manual minification is tedious and error-prone. Use our TOON Minifier tool to automatically optimize your TOON documents.
TOON Minifier Features:
- • Removes all whitespace and comments
- • Optionally abbreviates common field names
- • Inlines small nested objects
- • Optimizes array structures
- • Validates output for correctness
- • Shows before/after token counts
Minification Best Practices
1. Keep Readable Versions
Maintain human-readable TOON in your codebase. Minify only when sending to LLM. Use version control for both formats.
2. Test After Minification
Always validate minified TOON with our validator. Test LLM responses to ensure output quality isn't affected.
3. Balance Optimization vs Readability
For development and debugging, use standard TOON. For production prompts with high volume, use minified. Consider a "development mode" flag in your application.
4. Document Abbreviations
If using abbreviated field names, maintain a mapping document in your codebase. This helps onboarding new developers and debugging issues.
5. Measure Impact
Use OpenAI's tokenizer to measure exact token savings. Track cost savings over time to quantify ROI.
When NOT to Minify
Minification isn't always appropriate. Avoid minifying in these scenarios:
- •Development and debugging: Keep readable format for troubleshooting
- •Documentation and examples: Use standard TOON in guides and tutorials
- •Low-volume applications: If cost isn't critical, readability may be more valuable
- •Prompts requiring explanation: If LLM needs to understand structure, keep it readable
- •Team collaboration: When multiple developers need to review prompts
Minification Implementation Checklist
- ☐Convert existing data to standard TOON format
- ☐Test with TOON Validator to ensure correctness
- ☐Create abbreviation mapping for field names
- ☐Run through TOON Minifier to generate optimized version
- ☐Compare token counts using tokenizer
- ☐Test LLM responses with minified version
- ☐Verify output quality matches expectations
- ☐Deploy to production and monitor cost savings
- ☐Document minification strategy for team
Related Resources
External Resources
- •OpenAI Tokenizer Tool - Measure exact token counts
- •TOON Format GitHub - Official specification and tools
- •OpenAI API Pricing - Calculate your cost savings
- •Claude API Rate Limits - Anthropic's token pricing and limits
- •OpenAI Production Best Practices - Optimization strategies for production
- •Minifier.org - General minification concepts and tools