LLM Cost Optimization at Scale — Prompt Caching, Model Routing, and Batch Inference in Production

LLM API costs follow a simple formula: tokens consumed × price per token. At low volume, this is negligible. At production scale, it becomes a significant line item. A system processing 1 million requests per day at an average of 1,000 tokens per request (input + output) on GPT-4o costs roughly $7,500/day — $225,000/month. Even GPT-4o-mini at the same volume runs $450/day.

Most teams discover this problem after launch, when the invoices arrive. The good news is that there are systematic, engineering-driven approaches to reduce LLM costs by 50-80% without degrading output quality. This post covers the major strategies: caching, model routing, batch inference, prompt optimization, token budgeting, self-hosting economics, and the evaluation framework needed to ensure cost cuts do not break things.

The Cost Anatomy of an LLM Request

Before optimizing, understand where the money goes:

Total cost = (input_tokens × input_price) + (output_tokens × output_price)

For GPT-4o (as of early 2026):

• ✓Input: $2.50 per 1M tokens
• ✓Output: $10.00 per 1M tokens

For GPT-4o-mini:

• ✓Input: $0.15 per 1M tokens
• ✓Output: $0.60 per 1M tokens

Output tokens are 4× more expensive than input tokens on GPT-4o. This means reducing output length has a disproportionate impact on cost. A verbose 500-token response costs 4× more than a concise 125-token response on the output side alone.

Strategy 1: Prompt Caching

Many LLM applications see the same or very similar questions repeatedly. Customer support systems, FAQ bots, documentation assistants — the query distribution follows a power law where a small number of queries account for a large share of traffic.

Exact-Match Caching

The simplest form: hash the full prompt (system + user message) and cache the response. If the same prompt appears again, return the cached response without calling the LLM.

import hashlib
import json
from redis import Redis

redis = Redis(host="localhost", port=6379, db=0)
CACHE_TTL = 3600  # 1 hour

def cached_llm_call(messages: list[dict], model: str, **kwargs) -> str:
    cache_key = hashlib.sha256(
        json.dumps({"messages": messages, "model": model}, sort_keys=True).encode()
    ).hexdigest()

    cached = redis.get(cache_key)
    if cached:
        return json.loads(cached)

    response = openai_client.chat.completions.create(
        model=model, messages=messages, **kwargs
    )
    result = response.choices[0].message.content

    redis.setex(cache_key, CACHE_TTL, json.dumps(result))
    return result

Exact-match caching has a low hit rate for conversational applications (every conversation is unique) but works well for structured queries — the same product lookup, the same policy question phrased identically.

Semantic Caching

Two users asking "What's the return policy?" and "How do I return an item?" should get the same cached response. Semantic caching uses embedding similarity instead of exact matching:

1. ✓Embed the incoming query
2. ✓Search for similar queries in the cache (cosine similarity > threshold)
3. ✓If a match is found, return the cached response
4. ✓If not, call the LLM, cache the response with the query embedding

import numpy as np
from openai import OpenAI

client = OpenAI()

class SemanticCache:
    def __init__(self, similarity_threshold: float = 0.92):
        self.threshold = similarity_threshold
        self.cache: list[dict] = []  # in production, use a vector DB

    def _embed(self, text: str) -> list[float]:
        response = client.embeddings.create(
            model="text-embedding-3-small", input=text
        )
        return response.data[0].embedding

    def _cosine_similarity(self, a: list[float], b: list[float]) -> float:
        a, b = np.array(a), np.array(b)
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

    def get(self, query: str) -> str | None:
        query_embedding = self._embed(query)
        best_match = None
        best_score = 0.0

        for entry in self.cache:
            score = self._cosine_similarity(query_embedding, entry["embedding"])
            if score > best_score:
                best_score = score
                best_match = entry

        if best_match and best_score >= self.threshold:
            return best_match["response"]
        return None

    def put(self, query: str, response: str):
        embedding = self._embed(query)
        self.cache.append({
            "query": query,
            "embedding": embedding,
            "response": response,
        })

The similarity threshold is critical. Too low (0.85) and you return irrelevant cached responses. Too high (0.98) and the hit rate drops to near-zero. Start at 0.92 and tune based on quality evaluations.

Cache Invalidation

Cached responses go stale when the underlying data changes. Strategies:

• ✓TTL-based: Simple, predictable. Set TTL based on how frequently your data changes.
• ✓Event-based: Invalidate cache entries when relevant documents are updated. Requires tracking which source documents informed each cached response.
• ✓Versioned: Include a data version in the cache key. When data updates, increment the version and old entries naturally expire.

Strategy 2: Model Routing

Not every request requires GPT-4o. A simple greeting ("Hi, how can I help?") does not need the same model as a complex multi-step reasoning task. Model routing sends each request to the most cost-effective model that can handle it.

Classification-Based Routing

Train a lightweight classifier (or use a small LLM) to categorize incoming requests by complexity:

from enum import Enum

class Complexity(Enum):
    SIMPLE = "simple"      # greetings, FAQs, simple lookups
    MODERATE = "moderate"  # multi-step reasoning, summarization
    COMPLEX = "complex"    # analysis, code generation, nuanced decisions

MODEL_MAP = {
    Complexity.SIMPLE: "gpt-4o-mini",
    Complexity.MODERATE: "gpt-4o-mini",
    Complexity.COMPLEX: "gpt-4o",
}

PRICE_MAP = {
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},  # per 1K tokens
    "gpt-4o": {"input": 0.0025, "output": 0.01},
}

def classify_complexity(query: str) -> Complexity:
    # Use a fine-tuned classifier or a cheap LLM call
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": (
                "Classify the complexity of this customer support query. "
                "Respond with exactly one word: simple, moderate, or complex.\n"
                "simple: greetings, FAQs, status checks\n"
                "moderate: how-to questions, comparisons, multi-step requests\n"
                "complex: complaints needing investigation, technical debugging, "
                "policy exceptions"
            )},
            {"role": "user", "content": query},
        ],
        max_tokens=10,
    )
    label = response.choices[0].message.content.strip().lower()
    return Complexity(label) if label in Complexity._value2member_map_ else Complexity.COMPLEX

def route_request(query: str, messages: list[dict]) -> str:
    complexity = classify_complexity(query)
    model = MODEL_MAP[complexity]
    response = openai_client.chat.completions.create(
        model=model, messages=messages
    )
    return response.choices[0].message.content

The router itself costs tokens (the classification call), so it must be cheap. GPT-4o-mini with max_tokens=10 costs a fraction of a cent per classification.

Confidence-Based Fallback

A more sophisticated approach: always try the cheap model first. If its confidence is low (measured by logprobs or a self-assessment), escalate to the expensive model.

def route_with_fallback(messages: list[dict]) -> str:
    # Try cheap model first
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages + [
            {"role": "system", "content": (
                "After your response, rate your confidence on a scale of 1-5 "
                "where 5 means you are certain your answer is correct and complete. "
                "Format: [CONFIDENCE: N]"
            )}
        ],
    )
    content = response.choices[0].message.content

    # Extract confidence
    confidence = extract_confidence(content)  # parse [CONFIDENCE: N]

    if confidence >= 4:
        return strip_confidence_tag(content)

    # Low confidence — escalate to expensive model
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
    )
    return response.choices[0].message.content

This approach is self-correcting: the expensive model only runs when needed. In practice, 60-80% of requests can be handled by the cheap model.

Strategy 3: Batch Inference

If your application can tolerate latency (email processing, nightly report generation, bulk classification), batch inference offers significant savings.

OpenAI Batch API

OpenAI offers a 50% discount for batch requests with 24-hour turnaround:

import json

# Prepare batch file
requests = []
for i, ticket in enumerate(tickets):
    requests.append({
        "custom_id": f"ticket-{ticket.id}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "system", "content": CLASSIFICATION_PROMPT},
                {"role": "user", "content": ticket.text},
            ],
            "max_tokens": 100,
        },
    })

# Write JSONL file
with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

# Upload and submit
batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch_job = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

Async Processing Queues

For requests that need faster turnaround but can still be batched, use a queue-based architecture:

# Producer: enqueue requests
import redis

r = redis.Redis()

def enqueue_llm_request(request_id: str, messages: list[dict]):
    r.lpush("llm_queue", json.dumps({
        "id": request_id,
        "messages": messages,
        "enqueued_at": time.time(),
    }))

# Consumer: process in batches
def process_batch(batch_size: int = 20, max_wait_seconds: int = 5):
    batch = []
    deadline = time.time() + max_wait_seconds

    while len(batch) < batch_size and time.time() < deadline:
        item = r.brpop("llm_queue", timeout=1)
        if item:
            batch.append(json.loads(item[1]))

    if not batch:
        return

    # Process batch concurrently with asyncio
    results = asyncio.run(process_concurrent(batch))
    for request_id, result in results:
        r.set(f"llm_result:{request_id}", json.dumps(result), ex=3600)

Batching amortizes overhead and allows you to use rate limits more efficiently.

Strategy 4: Prompt Optimization

Shorter prompts cost less. This is obvious but underappreciated. Many production prompts contain redundant instructions, excessive examples, and verbose formatting that can be reduced without affecting quality.

Reduce Few-Shot Examples

Few-shot examples are expensive — each example consumes input tokens on every request. Reduce the number of examples to the minimum needed for consistent output:

# Before: 5 examples (≈ 500 tokens of examples)
PROMPT_V1 = """Classify the sentiment of this review.

Example 1: "Great product!" → positive
Example 2: "Terrible experience." → negative
Example 3: "It's okay." → neutral
Example 4: "Absolutely love it!" → positive
Example 5: "Would not recommend." → negative

Review: {review}
Sentiment:"""

# After: 1 example per class (≈ 200 tokens)
PROMPT_V2 = """Classify the sentiment as positive, negative, or neutral.

Examples:
"Great product!" → positive
"Terrible experience." → negative
"It's okay." → neutral

Review: {review}
Sentiment:"""

Run an evaluation to verify that reducing examples does not degrade accuracy. Often, 1-2 examples per class is sufficient for well-defined tasks.

Use Structured Output

Instead of asking the model to generate free-form text and then parsing it, request structured JSON output. This reduces output tokens and eliminates parsing errors:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    response_format={"type": "json_object"},
    max_tokens=150,  # JSON is typically more concise
)

Instruction Compression

Review your system prompts for redundancy. LLMs do not need verbose, human-friendly instructions:

# Before (87 tokens):
"""You are a helpful customer support assistant for our company.
When a customer asks a question, you should look at the provided context
and answer their question based on that context. If you cannot find
the answer in the context, please let the customer know that you
don't have that information available."""

# After (42 tokens):
"""Answer the customer's question using only the provided context.
If the context lacks the answer, say you don't have that information."""

Both produce equivalent behavior. The second saves 45 tokens per request — at 1M requests/day on GPT-4o, that is $112/day in input token savings alone.

Strategy 5: Token Budgeting

max_tokens

Always set max_tokens to a reasonable limit for your use case. Without it, the model might generate a 2,000-token response when you only need 100 tokens.

# Classification: max 10 tokens
response = client.chat.completions.create(
    model="gpt-4o-mini", messages=messages, max_tokens=10
)

# Short answer: max 150 tokens
response = client.chat.completions.create(
    model="gpt-4o-mini", messages=messages, max_tokens=150
)

# Detailed explanation: max 500 tokens
response = client.chat.completions.create(
    model="gpt-4o", messages=messages, max_tokens=500
)

Stop Sequences

Use stop sequences to terminate generation early when the model has produced the needed output:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    stop=["\n\n", "---"],  # stop at paragraph break or separator
)

Strategy 6: Fine-Tuning vs Prompting

A fine-tuned GPT-4o-mini can often match GPT-4o quality on a specific task at a fraction of the cost. The economics:

Fine-tuning costs: ~$25 for a small training set (500 examples), one-time. If you are making more than 100K requests/month on a well-defined task, fine-tuning almost always pays for itself within the first month.

The catch: fine-tuning is only effective for well-defined, consistent tasks. It does not help for open-ended reasoning or novel queries.

Strategy 7: Self-Hosted Models

Running open-source models on your own infrastructure eliminates per-token costs entirely. The question is whether the infrastructure cost is lower than the API cost.

Cost Breakeven Analysis

Running Llama 3.1 70B on an NVIDIA A100 GPU:

• ✓Cloud GPU cost: ~$2.50/hour (AWS p4d.24xlarge, amortized)
• ✓Throughput: ~30 requests/second with vLLM
• ✓Monthly cost: ~$1,800/month
• ✓Equivalent API cost: 30 req/s × 86,400 s/day × 30 days × $0.0003/req = $23,328/month

At 30 requests/second sustained throughput, self-hosting is roughly 13× cheaper. But the breakeven depends on your actual utilization. If you only process 1 request/second, the GPU still costs $1,800/month while the API would cost only $777/month.

vLLM Deployment

# Start vLLM server
# vllm serve meta-llama/Llama-3.1-70B-Instruct \
#   --tensor-parallel-size 4 \
#   --max-model-len 8192 \
#   --gpu-memory-utilization 0.9

# Use OpenAI-compatible API
from openai import OpenAI

local_client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)

response = local_client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=messages,
)

Self-hosting adds operational complexity: GPU procurement, model updates, monitoring, failover. The decision should be based on a realistic assessment of your team's infrastructure capabilities.

Monitoring and Cost Allocation

You cannot optimize what you do not measure. Track token usage at multiple levels:

from dataclasses import dataclass
from datetime import datetime

@dataclass
class LLMUsageRecord:
    timestamp: datetime
    model: str
    feature: str        # which product feature triggered this call
    team: str           # which team owns this feature
    input_tokens: int
    output_tokens: int
    cached: bool
    cost_usd: float
    latency_ms: float

def log_usage(response, feature: str, team: str, cached: bool = False):
    usage = response.usage
    model = response.model
    cost = calculate_cost(model, usage.prompt_tokens, usage.completion_tokens)

    record = LLMUsageRecord(
        timestamp=datetime.utcnow(),
        model=model,
        feature=feature,
        team=team,
        input_tokens=usage.prompt_tokens,
        output_tokens=usage.completion_tokens,
        cached=cached,
        cost_usd=cost,
        latency_ms=response._response_ms,
    )
    metrics_backend.emit(record)

Build dashboards that show:

• ✓Daily/weekly/monthly spend by feature and team
• ✓Cost per request by model and feature
• ✓Cache hit rates
• ✓Model routing distribution
• ✓Token usage trends

Evaluation: Ensuring Quality Survives Cost Cuts

Every cost optimization carries the risk of degrading quality. You must measure quality before and after each change.

A/B Testing Framework

Route a percentage of traffic to the optimized path and compare quality metrics:

import random

def handle_request(messages: list[dict], request_id: str) -> str:
    if random.random() < 0.1:  # 10% to control group
        response = call_llm(messages, model="gpt-4o")  # original path
        log_experiment(request_id, group="control", response=response)
    else:
        response = optimized_route(messages)  # optimized path
        log_experiment(request_id, group="treatment", response=response)
    return response

Quality Metrics

For each optimization, define measurable quality criteria:

• ✓Accuracy: Does the response correctly answer the question? (Evaluated by LLM-as-judge or human review on a sample.)
• ✓Completeness: Does the response cover all aspects of the question?
• ✓Relevance: Is the response focused on the question without unnecessary information?
• ✓Format compliance: Does the response follow the expected structure?

Run these evaluations on a held-out set of 200+ request-response pairs. Compare scores between the original and optimized paths. Only deploy optimizations that maintain quality scores within 5% of the baseline.

Case Study: Customer Support Automation

A SaaS company operating a customer support automation platform processed 50,000 tickets per day using GPT-4o for classification, response generation, and escalation decisions. Monthly LLM spend had reached $38,000 and was projected to grow 40% quarter-over-quarter as ticket volume increased.

Stripe Systems was engaged to reduce costs without degrading customer satisfaction scores (CSAT) or resolution accuracy.

Baseline Analysis

The team started by instrumenting every LLM call to understand the cost distribution:

Optimization 1: Semantic Caching

Many support tickets are near-duplicates. "How do I reset my password?" appears dozens of times daily with slight variations. The team implemented semantic caching with a similarity threshold of 0.93 on the response generation pipeline.

Implementation details:

• ✓Cache store: Redis with vector search (RediSearch module)
• ✓Embedding model: text-embedding-3-small (cheap, fast)
• ✓Cache TTL: 24 hours (knowledge base updates daily)
• ✓Scope: applied to response generation and knowledge base search only (classification and escalation need per-ticket precision)

Results:

• ✓Cache hit rate: 34% on response generation, 41% on knowledge base search
• ✓Monthly savings: $12,100
• ✓Quality impact: CSAT scores unchanged (cached responses are identical to original responses for semantically equivalent queries)

Optimization 2: Model Routing

Not every ticket needs GPT-4o. Password resets, account status inquiries, and simple how-to questions are well within GPT-4o-mini's capabilities.

Router implementation:

• ✓A fine-tuned GPT-4o-mini classifier categorizes tickets into simple/moderate/complex
• ✓Simple and moderate tickets (70% of volume) route to GPT-4o-mini
• ✓Complex tickets (30% of volume) route to GPT-4o
• ✓Router cost: ~$45/month (negligible)

# Router training data: 2,000 labeled tickets
# Features: ticket text, category, customer tier
# Labels: simple, moderate, complex

# Routing rules:
#   simple → gpt-4o-mini (password resets, status checks, FAQ)
#   moderate → gpt-4o-mini (how-to, feature questions, billing)
#   complex → gpt-4o (complaints, bugs, multi-issue, escalations)

Results:

• ✓70% of response generation shifted to GPT-4o-mini
• ✓Monthly savings: $8,800
• ✓Quality impact: CSAT for simple/moderate tickets dropped 0.3 points (from 4.4 to 4.1 on a 5-point scale) — within the acceptable 5% threshold

Optimization 3: Prompt Optimization

The existing prompts were verbose, with redundant instructions and excessive few-shot examples. The team systematically shortened them:

• ✓Ticket classification prompt: 580 tokens → 340 tokens
• ✓Response generation prompt: 920 tokens → 580 tokens
• ✓Escalation decision prompt: 410 tokens → 260 tokens
• ✓System prompts consolidated, redundant safety instructions deduplicated

Results:

• ✓Average tokens per request: 340 → 210 (across all features)
• ✓Monthly savings: $4,100
• ✓Quality impact: no measurable change in any metric

Combined Results

The total monthly savings of $25,000 came with a minor quality tradeoff: a 0.2 point CSAT decrease and a 1.5 percentage point resolution accuracy decrease, both within the pre-agreed acceptable thresholds. Response latency actually improved because cached responses are instant and GPT-4o-mini is faster than GPT-4o.

Projected Savings at Scale

With ticket volume projected to grow 40% per quarter, the cost optimization infrastructure scales linearly. Without optimization, the projected monthly spend at 100K tickets/day would have been $76,000. With the optimizations in place, it is projected at $22,000 — a $54,000/month difference.

The engineering effort to build and validate these optimizations took 6 weeks. The return on investment was measured in days, not months. The key lesson: LLM cost optimization is not about using cheaper models — it is about using the right model for each request, eliminating redundant computation, and measuring quality to ensure you are not trading accuracy for savings blindly.