TOON format provides approximately 50% token reduction compared to JSON. Minification techniques can increase this savings to 60-70%. For production LLM applications processing substantial token volumes, these additional optimizations significantly impact operational costs.
Token Savings Comparison
What is TOON Minification?
Minification removes unnecessary whitespace, optimizes field names, and applies compression techniques while maintaining data integrity. It's similar to JavaScript minification but optimized for LLM tokenization.
Elements Removed
- • Extra whitespace and newlines
- • Comments and documentation
- • Unnecessary indentation
- • Optional punctuation
Data Preserved
- • All data values
- • Data structure and hierarchy
- • Field relationships
- • Semantic meaning
Technique 1: Whitespace and Indentation Removal
Standard TOON uses 2-space indentation for readability. Minification removes all non-essential whitespace.
📄 Standard TOON (23 tokens)
# User profile with nested address user: id: 12345 name: "Sarah Mitchell" email: [email protected] address: street: "123 Main St" city: Boston zip: "02101" active: true
Minified TOON (16 tokens - 30% reduction)
user:id:12345;name:"Sarah Mitchell";email:[email protected];address:street:"123 Main St";city:Boston;zip:"02101";active:true
Whitespace Optimization Rules:
- • Remove all indentation (2-4 tokens per object)
- • Replace newlines with semicolons for nesting
- • Keep single space after colons only when needed for parsing
- • Remove blank lines (1 token each)
Technique 2: Field Name Abbreviation
Long descriptive field names improve readability but consume tokens. For production use, abbreviate fields while maintaining meaning.
📄 Descriptive Field Names (45 tokens)
customers[3]{customer_id,first_name,last_name,email_address,phone_number,subscription_status}:
1001,Sarah,Mitchell,[email protected],555-0101,active
1002,Michael,Chen,[email protected],555-0102,trial
1003,Jennifer,Kumar,[email protected],555-0103,activeAbbreviated Field Names (32 tokens - 29% reduction)
cust[3]{id,fn,ln,em,ph,stat}:
1001,Sarah,Mitchell,[email protected],555-0101,active
1002,Michael,Chen,[email protected],555-0102,trial
1003,Jennifer,Kumar,[email protected],555-0103,activeField Abbreviation Best Practices:
- • Use common abbreviations:
id,nm,addr,qty,amt - • Maintain consistency across dataset
- • Document abbreviations in system comments (not in prompt)
- • LLMs understand abbreviations with context
Technique 3: Optimal Delimiter Selection
Default comma delimiter is readable but not always most efficient. Choose delimiters based on your data patterns.
Delimiter Token Costs
| Delimiter | Token Cost | Best For |
|---|---|---|
| , | 1 token | Simple data without commas |
| | | 1 token | Data containing commas (addresses) |
| \t (tab) | 1 token | Copy-paste from spreadsheets |
| ; | 1 token | Nested inline structures |
Example: Using Pipe for Address Data
# Pipe delimiter avoids quote wrapping
addr[2]{st,city,zip}|:
123 Main St, Suite 100|Boston|02101
456 Oak Ave, Apt 5B|Seattle|98101Technique 4: Remove Comments and Documentation
Comments are helpful during development but should be stripped in production prompts.
📄 With Comments (35 tokens)
# Product catalog for Q4 2024
# Updated: 2024-12-15
# Total products: 3
products[3]{id,name,price}:
# Electronics category
1,Laptop,999.99
2,Mouse,29.99
# Accessories
3,Keyboard,79.99Without Comments (18 tokens - 49% reduction)
products[3]{id,name,price}:
1,Laptop,999.99
2,Mouse,29.99
3,Keyboard,79.99Rule: Remove all lines starting with # before sending to LLM. Keep metadata in your application code, not in the prompt.
Technique 5: Inline Small Nested Objects
Objects with 2-3 fields can be inlined to save tokens on newlines and indentation.
📄 Nested Format (27 tokens)
order:
id: 12345
customer:
name: "Sarah"
email: [email protected]
total: 149.99
status: shippedInlined Format (19 tokens - 30% reduction)
order:id:12345;cust:{nm:"Sarah";em:[email protected]};tot:149.99;stat:shippedWhen to Inline:
- • Objects with 2-4 simple fields
- • No further nesting inside
- • When readability isn't critical (production prompts)
- • Use semicolons to separate inline fields
Technique 6: Array Structure Optimization
Structured arrays with field definitions are efficient, but simple arrays can be even more compact for primitive values.
📄 Structured Array (18 tokens)
tags[5]{name}:
javascript
react
typescript
nextjs
tailwindSimple Array (12 tokens - 33% reduction)
tags[5]:javascript,react,typescript,nextjs,tailwind
Array Format Decision:
- • Use simple arrays: For primitives (strings, numbers, booleans)
- • Use structured arrays: For objects with multiple fields
- • Simple arrays save ~5-8 tokens per array declaration
Real-World Example: E-commerce Order
📄 Standard TOON (78 tokens)
# Order details for processing order_id: ORD-2024-001 customer: customer_id: 12345 name: "Sarah Mitchell" email: [email protected] phone: "555-0101" items[3]{product_id,product_name,quantity,unit_price,subtotal}: 101,Laptop,1,999.99,999.99 102,Mouse,2,29.99,59.98 103,Keyboard,1,79.99,79.99 shipping_address: street: "123 Main Street" city: Boston state: MA zip: "02101" order_total: 1139.96 order_status: processing
⚡ Minified TOON (34 tokens - 56% reduction)
oid:ORD-2024-001;cust:{id:12345;nm:"Sarah Mitchell";em:[email protected];ph:"555-0101"};itm[3]{pid,nm,qty,pr,sub}:101,Laptop,1,999.99,999.99;102,Mouse,2,29.99,59.98;103,Keyboard,1,79.99,79.99;ship:{st:"123 Main Street";city:Boston;st:MA;zip:"02101"};tot:1139.96;stat:processingMinification Breakdown:
Cost Impact Analysis
Token reduction directly impacts API costs. Here's the financial impact across different usage volumes:
| Monthly Volume | JSON Cost | Standard TOON | Minified TOON | Savings |
|---|---|---|---|---|
| 100K requests | $150 | $75 | $50 | $100/mo |
| 500K requests | $750 | $375 | $250 | $500/mo |
| 1M requests | $1,500 | $750 | $500 | $1,000/mo |
| 5M requests | $7,500 | $3,750 | $2,500 | $5,000/mo |
Based on GPT-4 Turbo pricing ($0.01 per 1K input tokens) with average 150 tokens per request.
Automated Minification Tools
Manual minification is tedious and error-prone. Use our TOON Minifier tool to automatically optimize your TOON documents.
TOON Minifier Features:
- • Removes all whitespace and comments
- • Optionally abbreviates common field names
- • Inlines small nested objects
- • Optimizes array structures
- • Validates output for correctness
- • Shows before/after token counts
Minification Best Practices
1. Keep Readable Versions
Maintain human-readable TOON in your codebase. Minify only when sending to LLM. Use version control for both formats.
2. Test After Minification
Always validate minified TOON with our validator. Test LLM responses to ensure output quality isn't affected.
3. Balance Optimization vs Readability
For development and debugging, use standard TOON. For production prompts with high volume, use minified. Consider a "development mode" flag in your application.
4. Document Abbreviations
If using abbreviated field names, maintain a mapping document in your codebase. This helps onboarding new developers and debugging issues.
5. Measure Impact
Use OpenAI's tokenizer to measure exact token savings. Track cost savings over time to quantify ROI.
When NOT to Minify
Minification isn't always appropriate. Avoid minifying in these scenarios:
- •Development and debugging: Keep readable format for troubleshooting
- •Documentation and examples: Use standard TOON in guides and tutorials
- •Low-volume applications: If cost isn't critical, readability may be more valuable
- •Prompts requiring explanation: If LLM needs to understand structure, keep it readable
- •Team collaboration: When multiple developers need to review prompts
Related Resources
External Resources
- •OpenAI Tokenizer Tool - Measure exact token counts
- •TOON Format GitHub - Official specification and tools
- •OpenAI API Pricing - Calculate your cost savings
- •Claude API Rate Limits - Anthropic's token pricing and limits
- •OpenAI Production Best Practices - Optimization strategies for production
- •Minifier.org - General minification concepts and tools