Why LLM Outputs Must Be Parsed, Not Trusted

The Problem

Every software system has a boundary where untrusted input becomes trusted domain data. For web applications, that boundary is between the HTTP request and the application model — validated, typed, rejected if malformed. For AI-augmented systems, an equivalent boundary exists between the LLM response and the system that acts on it.

Most teams do not treat it that way.

Instead, LLM output flows into application logic as raw strings or loosely-typed JSON dictionaries. Parsing happens deep inside the system — sometimes implicitly, sometimes never. The result is the same class of bugs that input validation was invented to prevent: runtime type errors, missing fields treated as empty, malformed structure silently misinterpreted.

The problem is not that LLMs produce bad output. The problem is that the output is untyped until the moment something breaks.

The principle is parse-don't-validate: at the system boundary, convert unstructured input into a typed value or reject it explicitly. Nothing that has not been parsed should cross the boundary. This is standard practice for external data sources — APIs, databases, user input. LLMs are external data sources. The same rule applies.

An LLM response that successfully produces a commitment-shaped blob of text is not a commitment. It is raw material for parsing. The difference matters because:

A string blob cannot be reasoned about by the type system
A string blob can contain anything — including fields that look right but mean the wrong thing
A string blob has no attribution, no authorization, no versioning

A parsed, typed `CoordinationOutput` can be passed through a pipeline where each step assumes its input is valid. The invariant is enforced once, at the boundary, and the rest of the system benefits from it everywhere.

Current Options

Untyped string handling

LLM output flows as strings or raw JSON dicts through the application. Parsing happens ad hoc where needed.

+ Fast to prototype

+ No schema design required upfront

+ Works with any LLM that produces text

− Runtime errors appear far from the boundary where they originated

− No compile-time guarantee that consumers receive valid data

− Schema drift between what the LLM produces and what the code expects is invisible until runtime

− Cannot audit what was received vs. what was processed

JSON Schema validation middleware

LLM output is validated against a JSON Schema after receipt. Invalid responses are rejected.

+ Catches schema violations at the boundary

+ Schema is explicit and versionable

+ Works across languages without rewriting validators

− JSON Schema does not produce typed values — validation succeeds but the result is still a dict

− The gap between JSON Schema and the language type system must be bridged manually

− Nested type errors require careful schema authoring to catch

Typed structured output contracts

LLM is instructed to produce output matching a typed schema. Output is parsed into a domain type immediately on receipt. Nothing untyped crosses the boundary.

+ The type system enforces the contract from the parse point forward

+ Schema is derived from or aligned with the domain type — one source of truth

+ Parse failures are explicit errors, not silent misinterpretations

+ The LLM prompt schema and the Rust type are kept in sync by tooling, not convention

− Requires schema design before prompt design — output-first thinking

− Complex nested schemas constrain LLM expressiveness

− Schema evolution requires versioning strategy

Why Structured Outputs

+ Type-system guarantees from the boundary inward

Once parsed, CoordinationOutput is a typed Rust value. Every function that receives it gets compile-time guarantees about its structure. No defensive checking inside business logic.

+ Failures are explicit and early

A malformed LLM response fails at the parse step with a structured error. It does not surface as a null pointer exception three call frames into the commitment pipeline.

+ Observable conformance

Parse success/failure rates are a metric. A model that drifts away from the schema shows up in monitoring before it corrupts business data.

+ Prompt and type stay in sync

When the domain type changes, the prompt schema must change — the coupling is explicit and enforced. With untyped handling, schema drift is silent.

Trade-offs

− Output-first design discipline required

You must define what the LLM should return before writing the prompt. Teams used to iterative prompt engineering find this constraining initially.

− Schema complexity limits model flexibility

A highly nested schema with strict constraints reduces the model's ability to express nuance. The schema must be designed to capture what matters, not to over-specify.

− Versioning overhead

When the domain evolves, both the Rust type and the prompt schema need updating. Without discipline, they drift.

Our Motivation

The rule in Converge is simple: nothing that has not been parsed crosses the LLM boundary. The LLM is external infrastructure. Its output is untrusted input. The parse step is the point where we assert what we received and convert it to something the rest of the system can rely on.

In practice, this means:

1. The domain type is defined first. What does the coordination layer need to produce for the commitment layer to form a commitment? That is the schema. 2. The prompt is written to produce output matching the schema. The schema is embedded in the system prompt as structured instructions. 3. The LLM response is parsed using `serde_json::from_str::<CoordinationOutput>`. If parsing fails, the error is logged, the coordination cycle is retried or surfaced as a coordination failure. The failure does not propagate. 4. The parsed `CoordinationOutput` moves through the commitment pipeline as a typed value. No JSON, no strings, no untyped maps.

The benefit is not just safety. It is testability. A function that takes a `CoordinationOutput` can be unit tested with constructed values — no LLM required. The parsing step can be tested with representative LLM responses. The two concerns are separated cleanly.

This pattern also enables a useful property: the LLM can be swapped. The parse contract defines what any model must produce to participate in the coordination layer. A model that cannot reliably produce conforming output is disqualified — not by performance benchmarks, but by the type system.

In Practice

coordination/output.rs

use serde::{Deserialize, Serialize};

/// The typed output of the coordination phase.
/// This is what the LLM must produce. Nothing else crosses the boundary.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CoordinationOutput {
    /// Summary of the shared model derived from evidence
    pub shared_model_summary: String,

    /// Explicit assumptions the coordination phase identified
    pub assumptions: Vec<Assumption>,

    /// Options surfaced for commitment consideration
    pub options: Vec<CommitmentOption>,

    /// Confidence in the shared model (0.0–1.0)
    pub model_confidence: f32,

    /// Contested claims that require human resolution
    pub contested: Vec<ContestedClaim>,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Assumption {
    pub statement: String,
    pub load_bearing: bool,   // if false, commitment may still hold
    pub source: String,       // which evidence this came from
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommitmentOption {
    pub description: String,
    pub trade_offs: Vec<String>,
    pub stopping_criteria: Vec<String>,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ContestedClaim {
    pub claim: String,
    pub positions: Vec<String>,
}

The LLM must produce JSON that deserializes into this struct. If it cannot, the response is rejected at the boundary — not silently degraded inside business logic.

coordination/parser.rs

use crate::coordination::CoordinationOutput;
use crate::errors::CoordinationError;

/// Parse the raw LLM response into a typed CoordinationOutput.
/// This is the boundary function. Nothing beyond this point is untyped.
pub fn parse_coordination_response(
    raw: &str,
) -> Result<CoordinationOutput, CoordinationError> {
    serde_json::from_str::<CoordinationOutput>(raw)
        .map_err(|e| CoordinationError::ParseFailure {
            reason: e.to_string(),
            raw_response: raw.to_string(),
        })
}

/// The commitment layer only accepts a typed CoordinationOutput.
/// It cannot receive a string. The type system enforces this.
pub fn form_commitment(
    coordination: CoordinationOutput,
    authorized_by: UserId,
) -> Result<CommitmentEvent, CommitmentError> {
    // confidence gate — low-confidence coordination does not proceed
    if coordination.model_confidence < 0.7 {
        return Err(CommitmentError::InsufficientConfidence {
            actual: coordination.model_confidence,
            required: 0.7,
        });
    }

    // contested claims block commitment — require resolution first
    if !coordination.contested.is_empty() {
        return Err(CommitmentError::UnresolvedContention {
            claims: coordination.contested,
        });
    }

    Ok(CommitmentEvent {
        id: CommitmentId::new(),
        coordination_summary: coordination.shared_model_summary,
        assumptions: coordination.assumptions,
        authorized_by,
        formed_at: Utc::now(),
    })
}

form_commitment() takes a CoordinationOutput, not a String or Value. The compiler enforces that nothing untyped reaches this function. The confidence and contention gates are part of the commitment logic, not defensive parsing.

Looking Forward

Structured output support is improving across all major LLM providers — JSON mode, function calling, constrained decoding. The direction is toward schema-first prompting as a first-class feature, not an afterthought.

The trend matters for commitment systems specifically. As structured output becomes more reliable, the boundary between coordination and commitment becomes sharper. The model produces a `CoordinationOutput`. The commitment layer consumes it. The two are coupled by a schema version, not by convention.

The remaining challenge is schema evolution. As the domain grows, `CoordinationOutput` changes. Old responses may not parse against new schemas. This requires explicit versioning — each response carries the schema version it was produced against, and the system knows how to handle schema migrations. This is the same problem event sourcing solves for stored events. The solutions are similar.

Recommendation

Define the output type first. Write the Rust struct (or equivalent) that represents what the coordination phase must produce before you write the prompt. Let the type drive the schema, and the schema drive the prompt.

Parse at the boundary. Use a single function — the boundary function — that converts raw LLM output into a typed value or fails with an explicit error. Instrument this function. The parse failure rate is a meaningful signal about model behavior.

Never let raw LLM output deeper than the parse function. If a caller of your coordination layer ever needs to handle a String or Value, the boundary is in the wrong place.