Actually, Loop Detection Shouldn't Wait for the Provider — Designing Client-Side Repetition Guards for Agent Harnesses

TL;DR: Most current repetition detection asks the provider to detect it and return a truncation signal. We take a different approach: treat loop-detection as a spectrum — near-exact verbatim loops accumulate toward abort, lexical sentence similarity only steers. That distinction lets you intervene earlier and more richly than any provider-side finish_reason can offer.

Every agent harness eventually faces the same failure mode: degenerate repetition. The model starts strong, then circles — rewording the same paragraph three times with minor variation. Or worse, a tight token loop that chokes the context window until generation stalls entirely.

Most current approaches delegate this to the provider. Providers eventually signal truncation only after the tokens have already been generated and billed — typically via finish_reason: "length" (OpenAI) or stop_reason: "max_tokens" (Anthropic). By that point, the damage is done: wasted compute, a polluted context window, and an agent stuck in recovery mode.

We took a different approach. Instead of treating repetition as a binary signal from the provider, we ask: what if the harness itself could distinguish between a model rephrasing for clarity and a model stuck in a token loop? That distinction changes everything — it lets you steer instead of abort, and it works across any model family.

This post documents our design decisions, what we learned from studying how other systems handle this problem, and why catching loops mid-stream changes what's possible.

The fundamental tradeoff

There are two philosophies for handling degenerate output:

The provider-only path is simpler to implement. The model knows its own weights and can detect token-path pathologies more precisely than any external observer. vLLM, for example, explicitly exposes n-gram repetition detection that can terminate generation early, plus repetition, frequency, and presence penalties[1].

But provider-side detection has structural limitations:

1. Latency of detection: The provider must generate the full repetitive sequence, recognize it internally, then signal truncation. All those tokens are billed and transmitted before you learn anything went wrong.
2. Context window pollution: Degenerate text enters your conversation history before being truncated at the API boundary. Subsequent turns see the loop residue in context — even though the user never saw it, the model does.
3. No steering possible: A provider can only abort. It cannot inject a corrective signal like "you're repeating yourself — try approaching from a different angle." That's something an agent harness has to handle itself if it wants early intervention without termination.
4. Portability: A harness may use a hosted frontier model today, a local Qwen model tomorrow, a speculative decoder afterward, different providers for planning and execution, fallback models during outages. Provider-native repetition behavior is part of an unstable dependency surface.

The client-side approach trades implementation complexity for provider-independent observation and intervention. A streaming monitor sits between you and the provider, watching tokens as they arrive. If repetition crosses a threshold, it can either inject a steering message (nudge the model back on track) or abort entirely before the damage compounds.

The precise architectural claim is not "client detects early, provider detects late" — the token has necessarily crossed the network boundary before a client-side detector can inspect it. The accurate claim is:

Detection should not wait for the provider's terminal judgment, because the harness can observe the stream continuously and intervene as soon as the evidence is sufficient.

Provider-side controls and harness-side controls are complementary layers, not competing philosophies. The provider can detect token-path pathologies (e.g., vLLM's n-gram repetition detection and penalties[1]); the harness owns task-level state and can therefore make richer intervention decisions. That distinction survives future provider improvements.

What we built

Our dedup crate (MIT licensed) is available at https://git.sly.so/kade/dedup — free for any harness developer to pull in[2].

It runs client-side with WASM bindings for JavaScript consumption and operates at three detection levels — each catching a different failure mode:

Near-exact detection (n-gram pass)

A sliding window over tokens, hashed with a rolling hash function (Rabin–Karp). Consecutive windows are compared for similarity using Jaccard coefficients on the tokenized output, plus character-level n-gram fallback. This catches verbatim repetition and near-duplicates where words have been slightly shuffled but the structure is identical.

The threshold here is strict (default 0.95) — only very similar matches trigger escalation toward an abort decision. The goal: catch loops that are clearly degenerate, not just rephrasing.

// High-level shape of the n-gram pass
pub struct NgramState {
    window_size: usize,
    step_size: usize,
    hashes: Vec<u64>,
    last_end: usize,
}

impl NgramState {
    /// Feed a streaming chunk and return any detected repetition sections.
    pub fn feed_chunk(&mut self, chunk: &str) -> RepetitionCheck { ... }
}

Sentence-level lexical similarity (sentence pass)

Beyond exact matches, some loops are conceptual rather than textual — the model describes the same concept using different words. We split output into sentences and compare them by token-overlap similarity (Jaccard over stop-word-filtered tokens). This is lexical sentence similarity, not semantic similarity — it measures shared vocabulary, not shared meaning.

This level can steer but never aborts on its own; a sentence-level match needs corroboration from the n-gram pass or multiple overlapping matches before triggering corrective action.

This asymmetry is intentional: legitimate rephrasing ("the r documentation" vs "r's docs") should only get a nudge, not termination. Verbatim loops die fast; conceptual circling gets redirected.

Important: Earlier versions of this design labeled this "semantic detection." That was imprecise. Jaccard over tokens measures lexical overlap, not semantic equivalence. Two sentences can have identical meaning with near-zero token overlap ("The process exhausted available memory" vs "It crashed after running out of RAM"), and conversely, high token overlap can mask opposite meaning ("The deployment succeeded because tests passed" vs "The deployment failed because tests did not pass"). A genuine semantic layer would require embeddings or a meaning-aware representation — we reserve that for future work. This distinction is supported by the SemEval STS literature, which defines semantic textual similarity as measuring semantic equivalence, not lexical overlap, and treats Jaccard n-gram as a lexical feature[3]. Recent work extends Jaccard with semantic alignment (synonyms, embedding proximity) to bridge this gap[4].

The streaming monitor: gated re-analysis, not incremental

All detection runs via a StreamingState that consumes chunks as they arrive. No polling timers — gating logic lives inside the state machine itself:

pub fn feed_chunk(&mut self, chunk: &str) -> RepetitionCheck {
    // Gate 1: warmup suppression (first ~3s, native builds only)
    if self.warmup_gate_active() { return RepetitionCheck::empty(); }

    // Gate 2: minimum content length
    if self.content.len() < self.settings.min_repeat_length * 8 {
        return RepetitionCheck::empty();
    }

    // Gate 3: skip if chunk adds no new characters
    if chunk.is_empty() || chunk == self.slice_at(last_check) {
        return RepetitionCheck::empty();
    }

    // Accumulate & re-analyze from scratch on new content
    self.content.push_str(chunk);
    let check = check_repetition(&self.content, &self.settings);

    if check.repeated && has_ngram_sections(&check) {
        self.repetition_count += 1;
    }
    check
}

pub fn determine_action(&self, check: &RepetitionCheck) -> StreamAction {
    if !check.repeated { return StreamAction::None; }

    // Sentence-level only → steer (never abort)
    let ngram_sections = filter_ngram(check);
    if ngram_sections.is_empty() { return StreamAction::Steer; }

    // Accumulated n-gram matches beyond threshold → abort
    if self.repetition_count >= max_repetitions {
        return StreamAction::Abort;
    }

    // Two very strong n-gram confirmations → abort even early
    let strong = ngram_sections.iter().filter(|s| s.similarity > 0.9).count();
    if strong >= 2 { return StreamAction::Abort; }

    StreamAction::Steer
}

When determine_action returns Steer, the harness appends a corrective system message to context before the next provider call. In our implementation it reads:

The assistant output appears to be circling the same point. Proceed directly to implementation without restating the problem.

The exact wording is tunable per workload and evolves as we learn which prompts nudge effectively without becoming part of a loop attractor (see threat model below). What matters structurally: the steer message has clearly separated provenance, so the model can't confuse it with its own reasoning.

Precision on "incremental": The current implementation is stream-triggered batch analysis, not algorithmically incremental detection. It accumulates the full output and re-runs check_repetition over the accumulated text whenever gates pass. The gating (warmup, content-length, chunk-delta) keeps steady-state cost low — most chunks skip analysis entirely — but the worst-case complexity on a long repetitive stream is quadratic in accumulated length. A recent performance fix (fix super-quadratic check_repetition on repetitive input) addressed the most pathological case by changing hash matching from all-pairs-per-bucket to first-occurrence-only pairing, but a genuinely incremental architecture (rolling-hash state + indexed historical signatures) would be stronger. We distinguish "streaming detection with gated re-analysis" from "incremental detection" going forward.

The asymmetric policy

The key insight from this work: not all repetition deserves the same response. Near-exact verbatim loops accumulate toward abort because they're almost certainly degenerate. Lexical sentence similarity alone can steer but never kill — the model might be circling conceptually while making real progress in different words.

This prevents false-positive aborts on legitimate rephrasing while letting verbatim loops die fast enough that the user barely notices anything happened.

Presets

The relaxed preset is not monotonically more or less aggressive — it moves sensitivity along different axes simultaneously. We intend to rename presets by intended workload (chat, reasoning, agent, code) with benchmarked false-positive characteristics rather than an implied scalar sensitivity.

Progress-aware trajectory detection

The loopy_value signal is arguably the seed of the most interesting system here: a continuous measure from productive debugging iterations to stuck behavior, using error→fix cycles, tool alternation, and progress markers.

That is a fundamentally different abstraction. A loop is not necessarily a simple fixed point:

It may instead be a cycle across multiple states:

Or at the agent-action level, a cycle that produces no progress:

The surface text may be entirely different on every iteration. A pure repetition detector can miss this indefinitely.

The real quantity of interest is closer to:

where:

• \(R_t\) = recurrence of states/actions
• \(\Delta P_t\) = measurable progress
• \(C_t\) = cyclic structure
• \(N_t\) = novelty or state-space exploration

A system becomes suspicious not merely when \(R_t\) is high, but when recurrence is high and progress approaches zero.

That distinction prevents an obvious failure mode:

This is repetitive. It is also productive.

By contrast:

This is approaching a fixed point. The second case is where intervention should intensify.

This is the conceptual center of the architecture. The current dedup crate provides the textual recurrence signals (\(R_t\)); the loopy_value detector (in the r harness) adds progress signals (\(\Delta P_t\)), cyclic structure (\(C_t\)), and novelty (\(N_t\)). Together they form a progress-aware trajectory detector rather than a text filter.

Research on circular reasoning in reasoning models supports this direction — Duan et al. show that semantic recurrence precedes textual repetition in reasoning loops, and use sequential change detection to predict loop onset[5]. The neural text degeneration problem has long been observed as a decoding strategy issue; Holtzman et al. showed that decoding strategy substantially affects repetitive degeneration, motivating nucleus sampling[6].

Loopy value in practice

Within our harness (~/src/r/) we layer detection across four tiers:

Tier 1 wraps the streaming monitor — one instance per assistant turn, feeds every chunk via feed_chunk(), acts on recommendations (none/steer/abort). Tier 2 uses the same crate's batch API for post-turn analysis of complete messages. Tier 3 compares turn signatures across a sliding window — if two consecutive turns produce identical tool-call sequences and similar text, that's a loop worth investigating. Tier 4 operates at the reasoning level, computing loopy_value from error→fix cycles, tool alternation patterns, and progress markers (file modifications, passing tests, new code). This feeds into inference budget decisions — productive loops get more context allowance; stuck ones trigger earlier abort thresholds.

Related work

Loop-detection approaches fall into two non-overlapping categories: generation-time repetition mitigation (applied during token sampling) and post-hoc agent-loop detection (applied after tool calls or turns complete). dedup sits in neither — it operates at the harness level, streaming text as a byproduct of either mode.

Generation-time approaches attack the failure from inside the decoder. Holtzman et al.'s nucleus sampling showed that decoding strategy substantially affects repetitive degeneration[6]. DRY sampling[7] adds an explicit penalty for already-seen n-gram suffixes at sampling time, cutting verbatim loops at their source. vLLM exposes internal n-gram repetition detection plus frequency/presence penalties that can terminate early — but only if the provider implements them[1]. These approaches work on a model's own weights — in llama.cpp, vLLM, SGLang, and similar local runtimes — but they still cannot intervene across providers when the model is hosted remotely.

Agent-loop detection approaches look at tool-call structure rather than text. The agent-loop-detector project hashes tool calls and detects identical repetitions, cyclic call patterns, and idle status-check loops[8]. Pipis et al.'s work on hidden-state circularity in reasoning models goes further — they identify that semantic recurrence precedes textual repetition by roughly 40 sentences on average, using a V-shaped attention mechanism to detect the cycle before it manifests as token degeneration[5]. Their CUSUM-based early warning gives an actionable window before any n-gram detector would fire.

Where this work sits. No published system combines streaming text detection with asymmetric steering (steer vs abort depending on what kind of repetition) and progress-aware trajectory analysis. Existing approaches operate at one layer: provider-side penalties, post-hoc n-gram guards, or tool-call cycle detection. dedup fills the gap by observing tokens as they stream across the network boundary — not to beat hidden-state predictors (that would require model access), but to be the layer that decides what to do once evidence of looping is sufficient.

On Duan et al.'s early warning. Their V-shaped attention detection runs on hidden states, which means it only works for models you run yourself (llama.cpp, vLLM, SGLang) — not when the model is a black-box API. That's a real limitation: our client-side n-gram detector catches loops after the model has entered an attractor, whereas their approach predicts onset before textual repetition begins. The right answer is likely both: hidden-state predictors flag early warning when running locally; harness-level text detectors decide intervention policy across any provider. The harness doesn't need to be first — it needs to own the decision layer.

Why client-side matters in practice

We cover the four structural limitations of provider-only detection in detail above; what's worth adding here is why each one translates to real engineering cost at scale.

1. Latency of detection: A 200-token degenerate loop isn't a minor delay — those tokens have already been transmitted, billed, and (if untruncated) committed to context window memory before the provider's finish_reason: "length" arrives.
2. Context window pollution: Degenerate text enters conversation history before being truncated at the API boundary. Even with a speculative buffer (see below), every degenerate token occupies space that subsequent turns see in context — and there is no way to retroactively delete it from a model's attention window.

The detector becomes part of a transactional generation boundary.

1. No steering possible: A provider can only abort. Client-side can steer when lexical sentence recurrence suggests rephrasing would help more than termination.

The cross-provider argument is where this becomes infrastructure rather than an implementation note:

Provider-native repetition behavior doesn't transfer between models. We hit this directly when swapping from a hosted frontier model to running Qwen3.6-27B locally mid-session — the frequency penalties baked into the provider's inference engine didn't carry over, and what had been contained under one decoder became pathological under another.

Client-side detection abstracts that away. The detector does not need to know why the model looped; it only needs to expose sufficiently reliable evidence to the harness. Provider-level prevention and harness-level detection solve different layers of the problem.

The steering mechanism needs a threat model

"Inject a corrective message" sounds straightforward, but it has nontrivial failure modes:

Context contamination

Every steering intervention becomes part of the model's context. A pathological model might then produce:

I am repeating myself.
I should stop repeating myself.
I am repeating myself.

The guard can accidentally become part of the loop attractor.

Intervention habituation

If every mild similarity event produces "You are repeating yourself. Try another approach," the model may learn:

Prompt injection interaction

If the model operates over untrusted tool output, a steering message must have clearly separated provenance and priority. Otherwise a malicious document could attempt to exploit the intervention mechanism or imitate its syntax.

Distributed race conditions

Streaming cancellation is not instantaneous. By the time the client decides to abort, there may already be:

The harness must define whether late tokens are discarded, remain visible in logs, enter context, or contribute to detector state.

We model intervention as a protocol, not just message injection:

The quarantine step is important.

What's next

We're prioritizing these in order of leverage:

1. Adversarial benchmarks (highest priority). Without evaluated benchmarks — covering legitimate repetition vs lexical degeneration vs productive agent iteration vs pathological loops — threshold tuning is guesswork. We need a dataset that other harness builders can compare against.
2. Genuinely incremental streaming. The quadratic re-analysis on long repetitive streams is a real liability at scale. Replacing full re-analysis with rolling-hash state + indexed historical signatures would give O(|new chunk| + k) steady-state cost.
3. Cross-model calibration. Thresholds are currently tuned for Qwen3.6-27B; systematic evaluation across model families (and decoding temperatures) is needed to find universal defaults versus per-model settings.
4. Multilingual sentence splitting. Our current splitter uses English-optimized regex. CJK and RTL text needs proper segmentation — likely a dedicated tokenizer or language-aware rules.
5. Recurrence topology (speculative). Track whether repeated regions have stable periodic spacing even when each unit differs slightly — a third layer between n-gram matching and loopy_value.
6. Progress-aware trajectory. This is the deepest idea in the architecture but depends on benchmarks being done first, since we can't calibrate progress signals without ground truth about productive vs pathological loops.

*This article was updated on 2026-08-29 following peer review that identified several imprecise technical claims, and again on 2026-09-01 with a second round addressing related-work depth, empirical framing, and cross-document consistency. Key corrections across both rounds: (1) fixed author attribution for Duan et al. (arXiv 2601.05693); (2) corrected sensitive preset n-gram threshold in README to match the actual code; (3) replaced fabricated finish_reason: "repetition_truncation" claim with accurate provider strings (length, max_tokens); (4) restructured opening to lead with asymmetric policy rather than provider critique; (5) expanded related work into generation-time vs agent-loop-detection categories, citing DRY sampling and positioning dedup at the harness intervention layer; (6) added concrete steer prompt example; (7) ranked "What's next" by leverage instead of listing flatly; (8) reconciled dedup README performance table with post's quadratic-worst-case characterization.

References

[1] vLLM Team. "vLLM Documentation: Repetition Detection and Penalties". 2025.

[2] Kade. "dedup: Text Repetition Detection Engine for Streaming LLM Output". 2026.

[3] Agirre et al.. "SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-lingual Evaluation". in *Proceedings of SemEval*. 2016.

[4] Chen, H. and others. "A graph-based model for semantic textual similarity measurement". 2025.

[5] Duan et al.. "Circular Reasoning: Understanding Self-Reinforcing Loops in Large Reasoning Models". 2026.

[6] Holtzman et al.. "The Curious Case of Neural Text Degeneration". *International Conference on Learning Representations (ICLR)*. 2020.

[7] Anonymous. "Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time". *arXiv preprint arXiv:2608.22761*. 2026.

[8] Princeu3. "agent-loop-detector: Detect and Break Repetitive Loops in Tool-Using AI Agents". 2025.