Actually, That Was The Easy Part: Async Pipelines, Per-Context Isolation, and Five Days of ATSInfer Engineering

⚠️ This is an engineering log, not a benchmark post. Every Phase A/B/C/D table row is "⬜ Not run" or "TBD." The async pipeline, MoE hot cache, and adaptive MTP changes described here have not been empirically validated end-to-end. The one measured number (71.46 t/s aggregate decode) is the static-placement baseline without any new async/hot-cache/adaptive work enabled. The benchmark post comes next. This post documents the 48 commits across two branches that made the infrastructure possible.

The previous post ended on a high note: +38% decode speed on partial offload, MTP speculative decoding working at 28-34% acceptance, and a DSPark-inspired adaptive draft tuner on the horizon. The algorithm was elegant. The knapsack solver was clean. The paper’s vision felt close.

Then we spent ten days in the trenches on merge/laguna-pr25165 (July 19–28): the async pipeline, per-context MoE isolation, crash fixes, MTP position guards, GDN classification, metrics plumbing. Then we spent two days on apply-pr-26323 (August 5–6) on the MoE hot cache — a scheduler-boundary war that taught us more about op_params[0] bit layout than we ever wanted to know. (The hot cache has a long tail: initial WIP June 3, routing observation wired July 27, consolidation August 5.)

This is the story of what happens after the algorithm works: the async pipeline that makes it actually fast, the per-context refactor that makes it actually safe, the crash fixes that make it actually boot, the metrics plumbing that makes it actually observable, and the hot cache that almost worked until the scheduler ate it. Forty-eight commits across two branches. The algorithm was maybe 10% of the work.

The async pipeline: making the paper's 3.29x real

The ATSInfer paper [1] claims 3.29x decode throughput from "asynchronous CPU-GPU coordination." The idea: overlap weight transfers (H2D over PCIe) with compute execution using two CUDA streams. While the GPU computes split N, the transfer engine is already copying weights for split N+1.

The infrastructure was already in ggml. The transfer stream (stream 1) and compute stream (stream 0) existed in ggml-cuda.cu. The cpy_tensor_async function could route copies to the transfer stream. Event-based synchronization (copy_event + transfer_pending) ensured compute never read stale data. The split graph lookahead was there too — starts N+1 copies while N computes.

What was missing: ATSInfer's placement decisions were not connected to any of it. Weight transfers happened synchronously. The GPU sat idle during every H2D copy. SM utilization hovered at 25-30% instead of the paper's 60-70%.

The fix: atsinfer_schedule_h2d_transfers()

Commit d9592cf2f[2] wires ATSInfer into the existing async pipeline:

// src/atsinfer.cpp — new function
void atsinfer_schedule_h2d_transfers(struct atsinfer_ctx * ctx) {
    if (!ctx || !ctx->async_enabled) return;

    // For each tensor the solver placed on GPU but currently lives on CPU:
    // launch async H2D copy on the transfer stream
    for (auto & transfer : ctx->pending_transfers) {
        ggml_backend_tensor_copy_async(
            transfer.src,      // CPU buffer
            transfer.dst,      // GPU buffer
            transfer.stream    // routes to CUDA stream 1
        );
    }
}

The call site in llama_context::graph_compute() (src/llama-context.cpp:2798):

void llama_context::graph_compute(ggml_cgraph * graph, bool block) {
    // ... ATSInfer observation lifecycle ...

    // NEW: schedule async transfers BEFORE compute
    if (atsinfer_enabled && atsinfer) {
        atsinfer_schedule_h2d_transfers(atsinfer);
    }

    // Existing: async compute (transfers overlap with prior split)
    ggml_backend_sched_graph_compute_async(sched.get(), graph);

    // ... MoE perf collection, reschedule check ...
}

The promotion path (atsinfer_promote_weights()) was also converted from synchronous ggml_backend_tensor_set to ggml_backend_tensor_copy_async, routing through the transfer stream. The old synchronous fallback in atsinfer-promote.cpp was removed entirely.

What we expect (not yet measured)

The paper reports 70% GPU SM utilization improvement from async coordination. We have not yet profiled with Nsight to verify this on our hardware. The validation plan requires our GPU node (RTX 4090, 24GB) and nsys profile:

nsys profile -t cuda,nvtx \
  -o /tmp/atsinfer-async-profile \
  ~/llama.cpp/build/bin/llama-server \
  --model /path/to/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \
  --ctx-size 210000 --gpu-layers 48 \
  --spec-type draft-mtp --n-gpu-layers-draft 999 \
  --atsinfer 1 --atsinfer-dynamic 1 \
  --port 11434

These are placeholders. Empirical validation is planned for our GPU node — see "Testing methodology" below.

Per-context MoE isolation: killing the globals

The MoE hot cache performance counters were a process-global static:

// src/llama-moe-hot-cache-perf.cpp:225 (BEFORE)
static llama_moe_layer_perf_state g_llama_moe_layer_perf;
static std::atomic<int> g_llama_moe_layer_perf_mode{-1};

This meant: - Cross-context metric bleeding: Tenant A's expert activation data leaked into Tenant B's results - Process-wide mutex contention: A single std::mutex serialized all MoE callbacks across all contexts - llama_moe_layer_perf_json(ctx) ignored its ctx parameter: Returned global aggregate, not per-context data - ~105 references in one file, all accessing the global

Commit 1d73f1343[3] removed both globals entirely. All MoE perf state now lives in llama_context::moe_perf:

// src/llama-context.h (AFTER)
struct llama_context {
    // ... existing members ...
    llama_moe_layer_perf_state moe_perf;  // per-context, no sharing
};

The migration pattern for all 105 references:

// BEFORE:
g_llama_moe_layer_perf.layers[i].calls++;

// AFTER:
ctx->moe_perf.layers[i].calls++;

For internal functions that don't receive ctx* directly, a thread-local bridge provides access:

// Thread-local bridge for callbacks that lack ctx parameter
static thread_local llama_moe_layer_perf_state * tls_perf_ptr = nullptr;

static inline llama_moe_layer_perf_state & tls_perf() {
    GGML_ASSERT(tls_perf_ptr != nullptr);
    return *tls_perf_ptr;
}

The bridge is set in llama_moe_layer_perf_graph_compute_begin(ctx) and cleared in graph_compute_end(ctx). This is defense-in-depth: the ggml eval callback already passes ctx* via user_data (line 867), but not all internal paths propagate it.

Result: 14/14 CTests pass. Zero process globals remain. Multi-tenant server deployments no longer corrupt each other's expert routing data.

Observation timing: the JIT compilation trap

The first graph_compute after a placement change includes JIT compilation and memory allocation overhead. This skews profiling data — the observation pass measures compilation time, not inference time, and the solver makes garbage decisions.

The original warmup skip (warmup_skip = 3) handled initial startup. But it did NOT handle re-schedule events. Every time atsinfer_apply_placement() or atsinfer_check_reschedule() moved tensors, the next observation pass was poisoned.

Commit b0656a020[4] adds two flags to atsinfer_ctx:

// src/atsinfer.h
struct atsinfer_ctx {
    // ... existing fields ...
    bool placement_changed = false;   // set by apply_placement() and check_reschedule()
    bool skip_next_obs_pass = false;  // set by start_observation() when placement_changed
};

The lifecycle:

void atsinfer_apply_placement(struct atsinfer_ctx * ctx) {
    // ... apply tensor placements ...
    ctx->placement_changed = true;  // mark: next obs will be poisoned
}

void atsinfer_start_observation(struct atsinfer_ctx * ctx) {
    if (ctx->placement_changed) {
        ctx->skip_next_obs_pass = true;
        ctx->placement_changed = false;
        return;  // skip this pass entirely
    }
    // ... normal observation start ...
}

void atsinfer_finish_observation(struct atsinfer_ctx * ctx) {
    if (ctx->skip_next_obs_pass) {
        ctx->skip_next_obs_pass = false;
        return;  // discard poisoned data
    }
    // ... normal observation finish, feed solver ...
}

Lifecycle: Computing → apply_placement()/check_reschedule() → PlacementChanged → start_observation() sees placement_changed → SkipNextObs (discard poisoned pass) → Computing. If no placement change: Computing → Observing → Solving → Computing.

Note: JIT compilation + memory alloc poisons timing data. The SkipNextObs pass discards this and waits for stable compute.

The Gemma-4 MTP segfault: four sites, one bug

Gemma-4 with --spec-type draft-mtp and --kv-unified shares a single llama_memory object between target and draft contexts (is_mem_shared == true). The server's prompt_clear() called common_context_seq_rm() on both contexts unconditionally. The second call corrupted the internal sequence linked list.

segfault at 10 ip 00007f... error 4 in libllama.so.0.0.10506

Offset 0x10 — a NULL pointer member access. Classic double-free pattern on a linked list.

The fix required guarding four seq_rm call sites, not just the one in prompt_clear():

Commits d64716f8c (initial fix) and 63e0e288d[5] (secondary fix, all four sites):

// Guard pattern applied to all four sites:
if (!spec->draft.is_mem_shared) {
    common_context_seq_rm(ctx_dft, seq_id, start, end);
}
// When is_mem_shared, the target's seq_rm already handled both

Additionally, ATSInfer itself was disabled for internal MTP draft contexts (they don't need tensor placement — they inherit the target model's layout).

Metrics and observability: making it visible

Five commits added the metrics infrastructure that makes ATSInfer debuggable in production:

Speculative draft metrics (commit 68e42dd63[6])

Three new Prometheus metrics on /metrics:

llama_cpp:speculative_draft_tokens_total 4523
llama_cpp:speculative_draft_tokens_accepted_total 1287
llama_cpp:speculative_draft_acceptance_rate 0.2845

Wired through server_metrics.n_draft_total / n_draft_accepted, accumulated in on_decoded(), exposed in all_metrics_def.

TPOT feedback loop (commits 4ace88687[7], 182652f16, dbf33a412)

The rescheduler (Algorithm 3) needs real TPOT data to rate-limit re-scheduling. Previously it used a hardcoded estimate. Now:

// include/llama.h:1728 — public API
LLAMA_API void llama_atsinfer_update_tpot(struct llama_context * ctx, float tpot_ms);

// common/speculative.cpp:~1907 — wired after TPOT EMA update in accept()
llama_atsinfer_update_tpot(params.ctx_tgt, tpot_ema_ms);

// src/atsinfer-reschedule.cpp:~77 — consumed by rate limiter
uint64_t get_min_interval_us() const {
    return (uint64_t)(5.0f * measured_tpot_ms * 1000.0f);
}

The rescheduler now waits at least 5x the measured TPOT between re-schedule checks, preventing thrashing during fast decode.

NVML compute utilization (commit d88815ef0[8])

The adaptive draft tuner used "GPU memory %" as a saturation proxy. This was misleading — VRAM usage doesn't correlate with compute saturation. A model can use 95% VRAM but only 30% SM utilization.

The fix: a runtime dlopen wrapper for NVML (common/nvml_wrapper.{h,cpp}) that requires zero build-time CUDA toolkit dependency:

// common/nvml_wrapper.cpp
bool nvml_wrapper::init() {
    handle = dlopen("libnvidia-ml.so.1", RTLD_LAZY);
    if (!handle) return false;  // graceful fallback on non-NVIDIA

    auto init_fn = (nvmlReturn_t(*)()) dlsym(handle, "nvmlInit_v2");
    // ... resolve symbols ...
}

float nvml_wrapper::get_compute_utilization() {
    nvmlUtilization_t util;
    nvmlDeviceGetUtilizationRates(device, &util);
    return util.gpu;  // 0-100% SM utilization
}

In compute_adaptive_n_max(), this becomes a Priority 2 signal (above latency spike, below acceptance rate):

// GPU saturated >90% → preemptively cap draft length
if (nvml_compute_load > 0.90f) {
    return std::max(adapt.min_n_max, adapt.current_n_max - 1);
}

MoE performance summary (commit 6f2900a6a[9])

A lightweight alternative to the expensive llama_moe_layer_perf_json() (full JSON serialization). Called from /metrics on every scrape:

// include/llama.h:1716
LLAMA_API void llama_moe_layer_perf_summary(
    struct llama_context * ctx,
    uint64_t * total_calls,
    uint64_t * total_expert_activations,
    float * avg_topk_fraction
);

The 3-tier GDN classification

Qwen3.6's GatedDeltaNet layers caused a subtle segfault: ATSInfer's is_gdn_node() matched tensor names too broadly, catching transient activation nodes that happened to share a name prefix with GDN state buffers. Promoting these transients to GPU while their compute stayed on CPU triggered fused-op backend alignment failures.

Commit 783cb3987[10] implements 3-tier classification:

The key fix: is_gdn_node() now only matches the tensor's own name, not its source tensors. A budget sanity check fails early if forced GPU tensors exceed budget by >20%, instead of producing an impossible ~80GB placement.

The MTP position offset saga

Five commits (37acd5871, 26fc729f9, 8883d4664, 6effb5166, 15c2e99a1) fixed a cascade of MTP position bugs:

1. Dummy tokens for never-drafting sequences (37acd5871): In M-RoPE mode, sequences that never received drafts still got dummy position entries, corrupting the position table.

2. Stale draft KV (26fc729f9, 8883d4664): Draft KV entries from previous requests persisted into new requests. Fixed by clearing stale entries in process() for all MTP modes, and clearing before first decode.

3. Position offset calculation (6effb5166): For kv-unified + Standard RoPE models (Gemma-4), the draft token placement used the wrong offset. The fix corrects n_past initialization.

4. M-RoPE position conflict (15c2e99a1): In non-chain_heads draft rebuild, the M-RoPE position table was overwritten by sequential positions. Fixed by preserving the multi-dimensional position encoding.

Each fix was validated with the ATSInfer recurrent regression CTests (tests/test-batch-decode-consistency.cpp).

Testing methodology: the atsinfer_launcher (shell edition)

All testing now uses a purpose-built shell harness (~1500 lines across 8 shell scripts + 2 shared libraries). It provides reproducible A/B comparisons between baseline and ATSInfer-enabled server configurations.

The old harness (nuked from orbit)

We previously maintained a Python async harness (~5000 lines across 8 modules: cli.py, process.py, config.py, investigation.py, report.py, models.py, exceptions.py, logging.py). It worked but had accumulated complexity:

Why the rewrite? The Python harness was over-engineered for what we needed: sequential A/B server comparisons with Prometheus metrics snapshots. The shell version is faster to modify, easier to audit, and has zero external dependencies beyond what's already on the GPU node. The Python harness is archived locally.

Architecture

Key design decisions

Sequential execution: Configurations run one at a time with a 15-second GPU cleanup delay between them. VRAM is not released instantly after process termination — nvidia-smi polling confirms memory is free before the next run. This is the same pattern the old Python harness used (await asyncio.sleep(15) after kill_existing_servers).

Process group isolation: Each server spawns in its own process group (os.setsid in Python, preexec_fn=os.setsid in subprocess; in shell we track PID and use os.killpg equivalent via tracked PID). Termination uses graceful SIGTERM → 10s wait → SIGKILL escalation. This prevents orphaned CUDA contexts from holding VRAM.

Crash detection: The log monitor watches for "Segmentation fault" and "core dumped" in real-time. In the shell version, this happens in a background tail process with grep. In Python it was an async task reading stdout line-by-line. Both raise/capture crash timestamps for precise crash-after-start measurements.

Standardized prompts: Three prompts of increasing complexity (short fix ~50 tokens, medium refactor ~500 tokens, long coding task ~2000+ tokens) ensure consistent workload across runs. Temperature is fixed at 0.7 for realistic variance (vs 0.0 for deterministic in the old harness).

Real-time throughput measurement: The collect_realtime_throughput function in lib/metrics.sh measures wall-clock latency via date +%s%N around a curl POST to /v1/chat/completions, then extracts eval_count and eval_duration_ms from the response usage field for accurate tokens/sec calculation. This avoids the pitfall of measuring prompt-processing time as generation time.

Configuration matrix

# configs/p0-prod-baseline.sh — Stock production config (ATSInfer disabled by budget failure)
# configs/p1-budget-override.sh — ATSInfer 21.5 GB budget (component-priority strategy)
# configs/p2-kv-q5.sh — q5_1/q5_0 KV cache
# configs/p3-ctx-131k.sh — 131K context
# configs/perf-collect.sh — MoE perf JSON for hot cache
# configs/hotcache-128.sh — 128 MiB hot cache
# configs/hotcache-512.sh — 512 MiB hot cache

Each config is a sourced shell script declaring SERVER_MODEL, SERVER_PORT, SERVER_CTX_SIZE, SERVER_CACHE_TYPE_K/V, SERVER_GPU_LAYERS, SERVER_ATSINFER, SERVER_CPU_MOE, EXTRA_FLAGS array, etc. The launch.sh sources the config, builds the command line, and starts the server.

Contrast with llama-bench

The upstream llama-bench (~/src/llama.cpp/build/bin/llama-bench) is a synthetic throughput benchmark — it measures raw token generation speed on a single context with controlled batch sizes, no server overhead, no concurrent slots, no KV cache sharing, and no real prompts. Our harness measures end-to-end server performance with:

llama-bench is excellent for microbenchmarking kernel changes (e.g., "did my new GEMM kernel speed up prefill by 5%?"). Our harness is for macrobenchmarking the full ATSInfer pipeline in a production-like server deployment ("does async H2D + dynamic reschedule actually improve decode throughput on real coding workloads with 8 concurrent slots?").

Deployment

The launcher runs locally on our GPU node. No SCP/SSH deployment needed — the model files, binary, and scripts are all on the GPU node. The launch.sh binds to the node's IP so remote clients (your laptop, CI) can hit the /metrics and /v1/chat/completions endpoints.

# On GPU node:
cd atsinfer_Launcher
./launch.sh p1-budget-override --background --tail

# From remote client:
curl http://<GPU_NODE_IP>:11434/v1/metrics

# Run benchmarks:
./benchmark.sh --config p1-budget-override --runs 5

# Compare runs:
./compare.sh runs/2026-08-06_120000-p1-budget-override runs/2026-08-06_130000-p2-kv-q5

Experimental Results (RTX 4090 24GB)

Config: --atsinfer-budget 21500 (component-priority strategy), Ornith-1.0-35B IQ4_NL, 256K ctx, 8 parallel, q8_0 KV, full offload.

Aggregate decode: 71.46 t/s (wall 54.32s, 3882 tokens, 2 seats × 5 turns)
Per-turn decode: 36-39 t/s steady, burst to 85 t/s on short final turn
TTFT: ~0.35-0.55s

This is the production baseline — ATSInfer static placement + component budget strategy working at full offload. No hot cache, no async pipeline measurement yet.

p0-prod-baseline: OOM at default budget

Default ATSInfer budget (15.6 GB) fails: Critical component 'base_model_weights' exceeds available VRAM budget. Server still boots (falls back to scheduler), but ATSInfer placement is disabled. VRAM after load: 22.8 GB / 24.5 GB.

hotcache-128: crash reproduced, then fixed

Config: p1 + --moe-hot-cache-max-mib 128 --moe-hot-cache-path /tmp/moe_perf_ornith.json --moe-hot-cache-auto-reserve-mib 64

Hot cache init: 127.32 MiB CUDA0 buffer, 48 experts selected (layers 0-39, 2 per layer 0-7, 1 per layer 8-39).

Before negative-id fix: Crash at ggml_compute_forward_mul_mat_id:1631 (expert id -1).

After fix (pending rebuild): Expected to pass warmup. First real decode untested.

Phase A: Baseline replication

Phase B: MoE stress test

Phase C: Nsight profiling

Phase D: 140k context (Qwen3.6-35B-A3B)

The documentation audit

While writing this post, we ran a full cross-reference audit of the project docs, journals, and todo files against the actual codebase. Findings:

Critical corrections applied

Medium corrections

• Stale line number in atsinfer-algorithm2.md (607→641 for atsinfer_apply_placement())
• Missing README.md diagrams in DIAGRAMS_INDEX.md (8 uncataloged)
• Missing 10th session-history diagram in index
• graph TD → flowchart TD type mismatch in index

The audit verified 18 items as correct, including all commit hashes, class locations, and the Algorithm 3 reschedule sequence.

Commit graph: See git.sly.so/kade/llama.cpp/graph?branch=refs/heads/apply-pr-26323 for the full history.

Commit breakdown

merge/laguna-pr25165 (July 19–28, ~31 commits): - Async H2D pipeline (Phases 1–4): d036399d9, 5a99ef390, 903d410eb, 6504fc738, 5d496df0d, 0f2ec8587, d9592cf2f - Per-context MoE isolation: 1ebdc96fa, 1d73f1343, b0656a020 - Observation timing fix: b0656a020 (July 28) - MTP position fixes (Gemma-4 is_mem_shared): 0b8fa58de, e0d7226e2, 7cb56ee6e, add83f0d9 - 3-tier GDN classification: 783cb3987 - Draft metrics + NVML + TPOT feedback: 9c549a4bc, d88815ef0, 68e42dd63 - Phase C adaptive batch splitting: a15739723, ef12bd047, 2fa91b00a, 4364c2e45 - Refactors: caaf064a9 (per-model preload state), f36fd1b63 (upstream merge fixes)

apply-pr-26323 (August 1–6, ~17 commits on top of e5da798cf): - Aug 1: e5da798cf async CUDA stream coordination (split-graph boundary, op_offload callback, lookahead scheduling) - Aug 5: d4622035e file consolidation (12 build fixes, MoE hot cache + ATSInfer into subdirs) - Aug 5: 6d8c8dfda debug flag + error message fix - Aug 6 (today): 30+ commits debugging worklist type (I32 vs F32), CUDA host buffers, pinning, negative-id CPU kernel fix, warmup bypass

The MoE Hot Cache Saga: A Scheduler-Boundary War (June 3–August 6)

The MoE hot cache has a long tail. Initial WIP commit c999ae674 (June 3) added runtime and graph infrastructure. 54e796da9 (July 27) wired live MoE routing observation into the hot cache lifecycle. f4830d41a (July 23) added activation-weighted placement + hot cache JSON export.

The bring-up on apply-pr-26323 branch happened in two bursts:

The goal

Copy the most-frequently-routed experts (observed via --moe-perf-json-output) into a compact CUDA0 buffer. At graph time, split each MoE FFN into a hot lane (cached experts, CUDA0) and a cold lane (remaining experts, CPU during decode), merged back per token. The hot cache uses a flat weighting — every observed expert gets equal priority — with a dummy padding slot for alignment.

The file consolidation disaster (and recovery)

The codebase had a confusing duplicate layout:

src/
├── llama-moe-hot-cache.cpp          # 1506 lines — REAL impl
├── llama-moe-hot-cache-graph.cpp    # 1506 lines — REAL impl
├── moe-hot-cache/
│   ├── llama-moe-hot-cache.cpp      # 3 lines — STUB
│   └── ...stub files...

Twelve build fixes later (commit d4622035e): everything consolidated into src/moe-hot-cache/ and src/atsinfer/ subdirectories. Clean build on GPU node (RTX 4090, 24GB). The stub-vs-real trap cost half a day.

The crash that wasn't a null pointer

Server starts, hot cache initializes cleanly (127.32 MiB CUDA buffer, 48/10,240 experts selected). First llama_decode() warmup:

llama-server: ggml/src/ggml-cpu/ggml-cpu.c:1631: ggml_compute_forward_mul_mat_id: Assertion `i02 >= 0 && i02 < n_as' failed.

Not a null pointer. An expert ID of -1 in the cold lane's mul_mat_id ids tensor. The CPU backend's row-grouping loop (thread 0, pre-barrier) asserts that every id is in [0, n_expert).

The missing consumer: ALLOW_NEGATIVE_IDS flag with zero readers

The hot cache graph builder writes a flag into op_params[0] of every mul_mat_id node:

// src/moe-hot-cache/llama-moe-hot-cache-graph.cpp:330-334
uint32_t flags = LLAMA_MOE_HOT_CACHE_MUL_MAT_ID_FLAG_ALLOW_NEGATIVE_IDS | ...;
memcpy(t->op_params, &flags, sizeof(flags));

The flag uses bits 4–6 of op_params[0] (bit 0 is GGML_PREC_F32 = 10, occupying bits 1+3). Deliberate placement — precision and flags share the same 32-bit word.

Runtime consumers of this flag across the entire tree: ZERO.

The flag contract was designed for kernel behavior that was never implemented. The cold lane pads with -1.0f (no dummy slot exists for cold). The ALLOW_NEGATIVE_IDS flag was supposed to tell the kernel: "skip negative ids, zero their output rows unless SKIP_NEGATIVE_ID_OUTPUT_ZERO is also set." No kernel reads it.

The fix: CPU kernel patch (commit on apply-pr-26323)

ggml/src/ggml-cpu/ggml-cpu.c:1627-1640 — the grouping loop (ith==0 only, race-free):

const int32_t i02 = *(const int32_t *)((const char *)ids->data + iid1*ids->nb[1] + id*ids->nb[0]);
if (i02 < 0) {
    if (!(flags & GGML_MUL_MAT_ID_FLAG_ALLOW_NEGATIVE_IDS)) {
        assert(0 && "negative id without ALLOW_NEGATIVE_IDS flag");
    }
    if (!(flags & GGML_MUL_MAT_ID_FLAG_SKIP_NEGATIVE_ID_OUTPUT_ZERO)) {
        memset(dst_row, 0, ne01 * nb0);  // zero the output row
    }
    continue;  // skip this id
}
assert(i02 < n_as);

The negative-id flag constants are defined locally in llama-moe-hot-cache-graph.cpp — upstream ggml.h does not expose them, so the hot cache owns its own constant definitions and static asserts internally.

The bisection table: budget pressure, not expert count

The correlation is ATSInfer placement pressure — higher budget → more experts promoted → more cold-lane -1 padding ids survive into the scheduler's unresolved parallel region → CPU backend sees them.

The warmup bypass decision (strategic retreat)

Every crash to date occurred in warmup (throwaway kernel warm / CUDA-graph capture). The decode path (real inference) was never reached. The code already special-cases cparams.warmup everywhere — the unpinned warmup branch graph was never sound.

Next cut (decided, not yet implemented): Gate the hot-cache FFN replacement with !cparams.warmup in src/models/qwen35moe.cpp:505. Fall through to vanilla MoE FFN for warmup. Then test the FIRST REAL DECODE — the cpu_decode_routing shortcut, worklist-from-logits, cold lane pinned backend_cpu, cold_first_row_input MAP_CUSTOM2 with pinned dst.

CUDA latent gap

ggml_cuda_mul_mat_id (ggml-cuda.cu:1896) has no negative-id support either. Currently no negatives are produced (CPU cold lane handles all decode), but if warmup ever uses the CUDA path, it'll assert. Follow-up item.

What's left

Immediate (requires GPU node)

1. Rebuild with negative-id fix on apply-pr-26323 branch
2. Implement warmup bypass in qwen35moe.cpp:505 (!cparams.warmup gate)
3. Test hotcache-128 through first real decode — the cpu_decode_routing path is the next untested surface
4. Run Phase A baseline replication on merge/laguna-pr25165 (confirm +38% still holds)
5. Run Phase C Nsight profiling (validate async overlap, SM utilization target 60-70%)

Known regressions

Medium-term

• Unified scheduling framework (ATSInfer + MoE hot cache + adaptive draft in one decision loop)
• Per-sequence adaptive n_max (independent tuning per request)
• Upstream preparation: split into logical PR-sized chunks
• Async pipeline empirical validation (Nsight on GPU node)

Lessons learned (part 2)

5. The algorithm is the easy part

The knapsack solver is ~200 lines. The async pipeline wiring, per-context isolation, crash guards, metrics plumbing, position offset fixes, file consolidation, build fixes, negative-id contract, CPU kernel patch — ~3,500 lines. The paper describes the algorithm. Production requires everything else.

6. Globals are technical debt with interest

The MoE perf globals "worked" for single-context testing. They became a correctness bug the moment we ran multi-tenant. The 105-reference migration took a full day. Every global is a future refactor with compound interest.

7. Crash fixes multiply

One segfault (shared KV cache) required four site guards. One position bug (M-RoPE) required five commits. The hot cache crash required: file consolidation (12 fixes), debug flag, error message fix, flag contract audit, CPU kernel patch, warmup bypass decision. Crashes are never "just one fix" — they're symptom cascades. Budget time for the cascade.

8. Document the audit trail

Thirty-plus todo files and ninety-five journal entries made this post possible. Every fix has a rationale. Every commit has a story. Future foxes can trace any claim to its source.

9. Flags in op_params[0] are a shared namespace

GGML_PREC_F32 = 10 (bits 1+3) + mul_mat_id flags (bits 4-6) = one 32-bit word, two independent consumers. The precision bits are read by fused-moe and custom ops. The flag bits should be read by mul_mat_id kernels. They weren't. When you pack flags into a shared word, verify every consumer. The static asserts in llama-moe-hot-cache-graph.cpp now enforce this.

10. Warmup is throwaway — don't fight the scheduler for it

The hot cache warmup graph was fundamentally unpinned: cold lane on nullptr backend → scheduler picks CUDA → -1 padding ids hit CUDA mul_mat_id → OOB expert reads → inf/NaN. Three iterations of pinning ops (worklist, logits, weights, set_rows inputs) each exposed the next unpinned boundary. The code already had cparams.warmup guards everywhere. The correct fix: skip the hot-cache replacement entirely during warmup. The decode path is where correctness matters.

Empirical data in this post reflects GPU node runs through 2026-08-06. The async pipeline and hot cache decode path remain unvalidated by Nsight. The engineering is done. The benchmarks are next. 🦊

See also: [Previous post][11] · [llama.cpp fork][12]

References

[1] Feng, S. and others. "Automated Tensor Scheduling for Hybrid {CPU-GPU} {LLM} Inference on Consumer Devices". 2025.

[2] Kade. "Commit d9592cf2f: Phase 3 async H2D transfer pipeline". 2025.

[3] Kade. "Commit 1d73f1343: per-context MoE isolation". 2025.

[4] Kade. "Commit b0656a020: observation timing fix". 2025.

[5] Kade. "Commit 63e0e288d: seq_rm 4-site guard". 2025.

[6] Kade. "Commit 68e42dd63: speculative draft metrics". 2025.

[7] Kade. "Commit 4ace88687: TPOT public API". 2025.

[8] Kade. "Commit d88815ef0: NVML compute utilization". 2025.

[9] Kade. "Commit 6f2900a6a: MoE summary API". 2025.

[10] Kade. "Commit 783cb3987: 3-tier GDN classification". 2025.

[11] Kade. "Actually, That's Really Fast: Hybrid CPU-GPU Scheduling for LLM Inference". 2026.

[12] Kade. "llama.cpp fork with ATSInfer branches". 2026.