Lite Models for Agentic Summarization

When an agent does a lot of thinking and backend processing before it answers, the user stares at a blank screen. The obvious fix — stream the model's raw chain of thought — is a trap: the reasoning is verbose, technical, and often exposes internals (SQL, table names, tool calls) you don't want a business user to see.

The pattern that worked for us: let the big model think; let a small "Lite" model narrate. A cheap, fast model watches the main model's internal reasoning and, off the critical path, rewrites each chunk into one friendly line of progress.

---

The Challenge

Our agents run heavy multi-step work per turn: interpreting an ambiguous question, choosing tools, calling a Conversational Analytics pipeline (schema resolution → SQL → execution → chart), then composing an answer. A single turn can take tens of seconds. Two conflicting requirements:

1. Show live progress so the turn feels alive, not hung.
2. Never show raw reasoning — it's long, technical, and leaks implementation detail. And never make the summary itself slow the answer down.

The raw material is already there. Modern models (Gemini 3.x here) emit thought parts when a thinking planner is attached — the model's own internal reasoning, interleaved with its output. The problem is purely one of presentation: raw thoughts are unshippable, and asking the main model to also produce a polished user-facing summary is expensive and adds latency to the very path you're trying to keep fast.

---

The Pattern: a Lite summariser as a side-car

Split the two jobs across two models of very different sizes:

flowchart LR
    U["User question"] --> M["MAIN model\nheavy reasoning + tools\n(produces the answer)"]
    M -->|raw thought chunks| Q{{"async queue\n(off critical path)"}}
    Q --> L["LITE model\none-shot summarise\n≤12 words, ~40 tokens"]
    L -->|friendly line| P["paced FIFO\nmin 2.5s dwell"]
    P --> GE["'Show thinking' pane\n(live progress)"]
    M ==>|answer, never blocked| GE

The double arrow is the important one: the answer path never waits for the Lite model. Summaries are decoration; if one is slow or fails, the answer still ships.

---

Why a "Lite" model specifically

The instinct is to reuse the main model for summarisation ("it already has the context"). Resist it. The Lite model wins on every axis that matters here:

• Latency floor. Summarisation runs many times per turn (once per reasoning chunk). A lite model returns in well under a second; the big model would add seconds of wall-clock per chunk. Even fired async, the big model's throughput and quota get eaten by chatter.
• Cost. Capped input (~800 chars), capped output (~40 tokens), minimal thinking, temperature=0. It is the cheapest possible call that still reads as human.
• Separation of concerns. The summariser has one job and a tiny prompt. It never inherits the main agent's tools, planner, or thinking config — so it can't wander, call a tool, or emit thoughts of its own.

It is a direct one-shot model call, not an agent — no Runner, session, sub-agent, planner, or tools. That keeps it predictable and disposable.

---

Two voices, one summariser

Reasoning comes from two different sources, and they read differently, so the summariser takes a kind that swaps the system prompt:

• analysis — the data reasoning from the analytics pipeline ("what is this query working out"). Rule: never mention SQL, tables, columns, joins, schemas.
• orchestration — the main agent's own planning reasoning ("what am I about to do next"). Rule: never mention tools, function names, or internals.

Both collapse to one short British-English sentence, max ~12 words, no markdown.

class ThoughtSummariser:
    async def summarise(self, text: str, *, kind: str = "analysis") -> str | None:
        """One friendly line, or None on timeout/error/empty. Never raises."""
        ...
        request = LlmRequest(
            model=self._model,
            contents=[Content(role="user", parts=[Part(text=prompt + text[:800])])],
            config=GenerateContentConfig(
                temperature=0,
                max_output_tokens=40,
                thinking_config=ThinkingConfig(thinking_level=ThinkingLevel.MINIMAL),
            ),
        )

---

The three rules that make it safe

A summariser that can stall the turn, crash it, or leak raw reasoning is worse than no summariser. Three invariants keep it honest:

1. Best-effort — it never raises, and it never stalls

summarise() wraps the call in a 4-second timeout and swallows every error, returning None. The caller then decides the fallback:

• analysis → fall back to a static catalog label (a canned, safe line). A thought is never lost.
• orchestration → drop it. There is no safe static fallback, because the raw reasoning is exactly the technical detail we're stripping. Better to show nothing than to leak.

2. Off the critical path — bounded, fire-and-forget

Each summary runs as its own asyncio task, gated by:

• a concurrency semaphore (max 3 in flight), and
• a per-turn cap (max 12 summaries).

So a burst of reasoning can't spawn unbounded calls, and the answer stream is never blocked waiting for a summary. At end of turn, in-flight summaries are cancelled — a summary that resolves after the answer must never appear beneath it.

3. Paced, so it's readable

Reasoning arrives in bunches, and the UI renders each line the instant it lands — so a burst flickers past unreadably. Summaries queue FIFO and release no faster than one per 2.5s (tuned live; 1.5s read too fast). None are skipped mid-turn; the only drop is the end-of-turn cancel above.

sequenceDiagram
    participant Main as Main model
    participant Orch as Orchestrator
    participant Lite as Lite model
    participant UI as Show thinking

    Main->>Orch: raw thought chunk
    Orch->>Lite: summarise(chunk)  [async task, sem≤3, cap≤12]
    Note over Main,UI: main loop keeps running — never blocks
    Lite-->>Orch: "Checking recent call volumes…" (≤4s, or None)
    alt success
        Orch->>UI: friendly line (paced, ≥2.5s dwell)
    else timeout / error
        Orch->>UI: static fallback (analysis) OR drop (orchestration)
    end
    Main->>UI: final answer (immediately, unpaced)
    Note over Orch,Lite: end of turn → cancel any in-flight summaries

---

Deployment notes

• cloudpickle-safe. In our stack the agent is serialised for deployment. The summariser holds only a model-name string; the model client is built lazily at call time and never stored, so nothing unpicklable (httpx/gRPC locks) ends up on the instance.

• Everything is env-flagged and defaults off, so the feature is opt-in per environment and pure passthrough when disabled:

  Env var	Effect
  ENABLE_LIVE_THOUGHTS	Master switch for live progress
  ENABLE_LIVE_THOUGHT_SUMMARY	Summarise the analysis reasoning with the Lite model
  SUMMARISE_NATIVE_THOUGHTS	Strip the main model's raw thoughts, show Lite orchestration summaries instead
  THOUGHT_SUMMARY_MODEL	Which Lite model to use (default gemini-3.1-flash-lite)

---

The Results

• Perceived latency collapsed. The turn now shows a friendly, paced trail of what it's doing within seconds, instead of a blank pane for 20–40s.
• Zero technical leakage. Business users never see SQL, table names, or tool calls — only plain-English lines. Raw model thoughts are stripped before they reach the UI.
• Negligible cost and no added answer latency. Summaries are tiny lite-model calls, capped in count and concurrency, and run entirely off the answer path.
• Fails invisibly. On any timeout or error the system falls back to a safe canned line or drops silently — the answer is never affected.

---

When to reach for this pattern

Use a Lite-model side-car summariser when all of these hold:

• Your agent has meaningful "dead air" — multi-step reasoning or slow tool calls.
• The raw reasoning exists but is not shippable (verbose, technical, sensitive).
• You want progress UX without taxing the main model's latency, cost, or quota.

Skip it if turns are already fast, or if the model's raw thoughts are clean enough to show directly. The whole value is in the asymmetry: a big model to be right, a tiny model to be readable.