Actually, That's Really Fast: Hybrid CPU-GPU Scheduling for LLM Inference — The ATSInfer Journey

The release of Qwen3.6-27B in April 2026 sparked a flurry of experiments in the llama.cpp community. While full GPU offload remains the default for high-end inference, practitioners with partial offload hardware (single RTX 4090, consumer GPUs, laptops with iGPUs) have pushed for smarter tensor placement strategies. The paper "Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices" [1] promised a general solution. But does it actually deliver?

This post documents our implementation of the paper's static placement algorithm in our llama.cpp fork, the surprises we encountered, and the actual performance we measured on Qwen3.6-27B with partial offload. More importantly, it chronicles our journey from initial implementation through production deployment, including the critical fixes that transformed ATSInfer from a promising prototype into a production-ready system.

The claim

The paper's central hypothesis is straightforward: not all tensors need to live on the same device. By analyzing per-operator timing and solving a knapsack-like optimization, the scheduler can identify which tensors to keep on GPU (where compute is fast) versus CPU (where the PCIe transfer cost outweighs the GPU benefit).

The algorithm is elegant:

Where \(r_i\) is the reward for placing tensor \(i\) on GPU. Positive reward means GPU wins; negative means stay on CPU. The scheduler solves a constrained optimization (GPU memory budget) with adjacency penalties (switch cost) to minimize total execution time.

What we implemented

Phase 1: Mapping the fork

Before touching the scheduler, we mapped the integration points:

1. Scheduler Pass 5 (ggml-backend.cpp) — where backend assignment and split creation happen
2. CUDA backend streams — compute stream + transfer stream + VBR side-stream for turbo KV (codecs from spiritbuun/buun-llama-cpp[2])
3. Turbo KV cache — must be pinned GPU-only (VMM pages, tier transcoding would break)
4. MoE hot cache — atomic scheduling units with fork/join annotations

The fork already had the infrastructure hooks: ggml_backend_sched_set_tensor_backend() for placement, eval callbacks for profiling, and the two-stream CUDA model for overlap.

Phase 2: Static Placement Solver

We implemented the core solver in src/atsinfer-placement.{h,cpp}[3]:

// Knapsack DP solver at MB granularity
struct PlacementSolver {
    // reward: GPU time savings minus copy cost
    float reward(size_t tensor_idx, float current_memory);

    // adjacency penalty: switch cost between backends
    float switch_cost(size_t tensor_idx);

    // solve: maximize total reward subject to memory budget
    std::vector<Backend> solve(const std::vector<Tensor>& tensors, float budget);
};

The solver runs during the first decode via an eval callback, collecting per-node timing data. We calibrated CPU/GPU ratios based on empirical measurements: 12x memory-bound, 30x compute-bound (matmul).

Phase 3: Load-Time Placement

The paper's gains come from load-time placement, not runtime hints. In llama.cpp terms:

1. Run solver BEFORE llama_model::load_tensors()
2. For each weight tensor, solver decides GPU or CPU
3. load_tensors() uses solver's decision via tensor_buft_overrides
4. Weights load directly into their final location

This avoids the fundamental issue we discovered: runtime tensor modification is unsafe in ggml. The buffer system owns tensor lifecycles; modifying buffers at runtime breaks scheduler invariants.

The surprises

Surprise 1: Runtime hints are fundamentally broken

We initially tried ggml_backend_sched_set_tensor_backend() to override the scheduler's decisions at runtime. This failed spectacularly:

#0 ggml_backend_sched_split_graph()
#1 ggml_backend_sched_reserve()
#2 llama_model_load()

The scheduler's split_graph() memsets ALL hv_tensor_backend_ids to -1 before Pass 1 when is_reset == false. So our runtime hints were wiped before the solver could use them.

Fix: Use load-time placement via tensor_buft_overrides, not runtime hints. See commit fee98a844[4].

Surprise 2: GDN fused ops break naive promotion

Qwen3.6's GatedDeltaNet layers use fused ops that require backend alignment. Promoting their weights to GPU while compute stays on CPU causes:

error: fused op not supported on this backend

Fix: Detect GDN layers and exclude ALL tensors in those layers from promotion. Zero their rewards before solving.

Surprise 3: Turbo KV requires full offload

Turbo4 KV cache (from spiritbuun/buun-llama-cpp[2]) throws at init if any turbo KV lands on a non-VBR (CPU) backend. So partial offload + turbo = crash. The turbo codecs (turbo4, turbo3_tcq, etc.) require the TurboQuant CUDA interface; layers whose KV lands on CPU fall back to q8_0.

Fix: Either force all KV to GPU (defeats memory savings) or implement CPU turbo kernels (in progress).

Surprise 4: The +38% gain was real (but only for Qwen3.6)

With -ngl 48 (partial offload), we measured:

The key was contiguous block promotion: instead of per-tensor promotion, we promote entire layers from the GPU boundary. This creates one CPU↔GPU transition, not dozens.

The benchmark results

Production config (-ngl 999, full offload, MTP, 210k ctx)

With full offload, ATSInfer correctly recognizes all weights are on GPU and doesn't fight the existing configuration.

Hybrid config (-ngl 48, partial offload, Qwen3.6-27B Q2_K_XL)

The 38% gain comes from smart layer promotion: ATSInfer detects that 48 layers aren't worth keeping on CPU for this model size, promotes contiguous blocks to GPU, and amortizes the PCIe transfer cost over the entire layer.

Model zoo validation

The turbo4 crash story

During this work, we discovered a dangling Q tensor in the turbo flash-attn kernel from spiritbuun/buun-llama-cpp[2]. The bug was in ggml_cuda_flash_attn_ext_vec() (ggml/src/ggml-cuda/fattn.cu) which pointed a graph tensor's src[0] at a stack-local Q_rot:

ggml_tensor Q_rot;            // function-scope STACK local
dst->src[0] = &Q_rot;         // graph tensor src[0] -> STACK local

When the stack unwound, src[0] became a dangling pointer. A gdb hardware watchpoint caught the corruption:

Hardware watchpoint: *(void**)0x7fff90b58a78
Old value = 0x7fff90b58590   (real Q)
New value = 0x7fff96dc6d10   (a STACK address)
#0 ggml_cuda_flash_attn_ext_vec()   fattn.cu:261

Fix: Do not modify dst->src[0]. Copy the dst tensor and rotate the copy's Q:

ggml_tensor Q_rot;            // function scope (outlives dispatch)
ggml_tensor dst_rot;          // local copy of the op
dst_rot.src[0] = &Q_rot;      // only the COPY points at Q_rot

Additionally, turbo V output was left in the rotated domain. The K encode rotates with signed FWHT, but V dequant didn't un-rotate. Fix: Added inverse FWHT in dequantize_V_turbo4_0 and dequantize_V_turbo8_0 to transform V back to the original domain before the output projection.

The broader insight

The paper's static placement is equivalent to "smart n_gpu_layers". For models where all layers fit in VRAM, the optimal placement is full offload. ATSInfer's value is in automatically detecting this and adjusting n_gpu_layers, accounting for KV cache, compute buffers, and CUDA overhead.

For models that don't fit, the contiguous block approach promotes as many layers as possible from the GPU boundary, minimizing transitions. This is the sweet spot for consumer hardware: not enough VRAM for full offload, but enough for 60-70% of layers.

The seq_rm saga

One of the cleanest wins from this work was fixing checkpoint reuse for hybrid/recurrent models. The problem: Qwen3.6-27B's GDN layers have no per-token KV cache, so pos_min always returns the latest position, making checkpoint matching impossible. This was one of three independent root causes identified in our cache invalidation analysis[5]:

1. Checkpoint search predicate (primary killer): The search lambda used cur.pos_min < pos_min_thold, which always failed for hybrid models where pos_min equals the full sequence length.
2. Checkpoint creation threshold: The default checkpoint_min_step (64–256 tokens) was too high for the short turns common in hybrid model tool-calling.
3. Seq_rm wipe destroys restored checkpoint: common_context_seq_rm() always fails for hybrid models with partial ranges, and the server treated this as an error, wiping the cached state.

Multiple contributors tackled these issues across several patches and PRs:

• sanmai (e0b83880): Updated the search lambda to use cur.pos_max <= pos_next for hybrid models, lowered checkpoint minimum to 4 tokens, and added a guard to prevent memory wipe if seq_rm fails but a checkpoint was restored.
• spiritbuun (b9989525): Implemented the seq_rm wipe guard specifically.
• PR #22929[6]: Introduced role-based prompt splitting and strategic checkpoints before user messages.
• PR #13194[7]: SWA support and unified KV cache refactor.
• PR #24035[8]: The merged fix that stores pos_end in each checkpoint:

struct common_prompt_checkpoint {
    llama_pos pos_min;
    llama_pos pos_max;
    llama_pos pos_end;  // NEW: where the prompt ended at checkpoint creation
    int64_t n_tokens;
    std::vector<uint8_t> data;
};

Checkpoint selection now uses pos_end instead of pos_min:

if (is_recurrent_or_hybrid) {
    return cur.pos_max <= pos_next;  // Use pos_max, not pos_min
}

When pruning checkpoints, entries are removed if their saved prompt end exceeds the end of the new prompt, or if their memory range exceeds the current pos_next.

Beyond the upstream PR, our local fork adds boundary-anchoring: an extra checkpoint anchored at the common-prefix boundary (the divergence point). For workloads like codeneedle (big shared file + tiny changing suffix), this dropped re-processing from ~2048 tokens / ~3.9 s to ~16 tokens / ~0.15 s.

MoE hot cache integration

Our llama.cpp fork now has CLI access to the advanced MoE hot cache system:

llama-diffusion-cli \
  --model diffusiongemma-26B-A4B-it-Q4_K_M.gguf \
  --moe-hot-cache /tmp/perf.json \
  --moe-hot-cache-max-mib 2048 \
  --gpu-layers 999

The hot cache requires a performance JSON file with expert activation patterns:

{
  "schema": "llama.cpp.moe_layer_perf.v1",
  "layers": [
    {
      "layer": 0,
      "experts": [[0, 100], [1, 95], [2, 80]],
      "hot_experts": [[0, 80], [1, 75]],
      "cold_experts": [[2, 70]],
      "total_moe_time_per_call_us": 580
    }
  ]
}

This enables selective expert promotion to GPU, dramatically improving MoE performance on partial offload.

The MTP Speculative Decoding Journey

While ATSInfer's tensor placement was delivering solid gains, we hit a critical bottleneck: MTP (Multi-Token Prediction) speculative decoding was running on CPU, killing performance.

The Problem: Draft Context GPU Layers

Qwen3.6-27B-MTP uses a draft model that predicts multiple tokens ahead. The target model verifies these predictions in batch. For this to be fast, both target AND draft must run on GPU.

But our initial implementation had a critical bug: the draft context wasn't inheriting the user's --n-gpu-layers-draft setting. It defaulted to CPU execution.

Symptoms: - Draft generation: ~2 tok/s (CPU) - Draft acceptance: 0% (timing mismatch) - Overall decode: ~13 tok/s (worse than non-speculative!)

See our ATSInfer documentation[9] for full diagnostic details.

The Fix: Apply n_gpu_layers_draft to Draft Context

The fix was straightforward but critical. In common/speculative.cpp[10] around line 3096:

// Before fix (line 3096):
if (params.speculative.draft.n_gpu_layers >= 0) {
    mparams.n_gpu_layers = params.speculative.draft.n_gpu_layers;
}

// After fix (commit d1a54d8c7):
if (params.speculative.draft.n_gpu_layers >= 0) {
    mparams.n_gpu_layers = params.speculative.draft.n_gpu_layers;
}
// Same code, but now properly applied BEFORE llama_init_from_model()

The key insight: mparams.n_gpu_layers must be set before calling llama_init_from_model(), not after. The model initialization reads this value to decide how many layers to offload to GPU.

See commit d1a54d8c7[11]: fix(speculative): apply n_gpu_layers_draft to MTP draft context model params

Results After Fix

With the fix applied and ATSINFER_BUDGET_MB=20000 set:

Improvement: - Draft speed: 2 tok/s → 20 tok/s (10× faster) - Acceptance rate: 0% → 28-34% (working!) - Overall decode: 13 tok/s → 20 tok/s (+54%)

But GPU utilization was still only 25-30%, suggesting room for further optimization.

The Cache Type Mismatch Discovery

After fixing the GPU layers issue, we noticed another problem: the draft context was using f16 KV cache while the target used q5_1/q5_0 compressed cache. This caused:

1. Memory waste: Draft cache used 30% more VRAM than necessary
2. Precision mismatch: Draft predictions in f16 vs target verification in q5_1/q5_0
3. Conversion overhead: Extra type conversions during verify step

The Root Cause

The CLI flags --spec-draft-type-k and --spec-draft-type-v existed in common/arg.cpp[12] but were never applied to the draft context's cparams.type_k/v.

The Fix: Apply Draft Cache Types

In common/speculative.cpp[13] (lines ~3075 and ~3105):

// For external draft models (has_draft path):
if (params.speculative.draft.cache_type_k != GGML_TYPE_F32) {
    cparams.type_k = params.speculative.draft.cache_type_k;
}
if (params.speculative.draft.cache_type_v != GGML_TYPE_F32) {
    cparams.type_v = params.speculative.draft.cache_type_v;
}

// For MTP internal draft (spec_mtp path):
if (params.speculative.draft.cache_type_k != GGML_TYPE_F32) {
    cparams.type_k = params.speculative.draft.cache_type_k;
}
if (params.speculative.draft.cache_type_v != GGML_TYPE_F32) {
    cparams.type_v = params.speculative.draft.cache_type_v;
}

This ensures the draft context respects the user's cache type settings, reducing memory footprint and eliminating precision mismatch.

Expected impact: +10-15% speed improvement (~22-23 tok/s), reduced memory pressure.

Adaptive Draft N-Max Tuning (DSPark-Inspired)

Even with the cache type fix, we noticed that fixed n_max=2 wasn't optimal for all workloads. Sometimes acceptance was very low (<15%), meaning we were wasting compute generating drafts that got rejected. Other times acceptance was high (>50%), meaning we could safely generate longer drafts for more speedup.

The DSPark Paper Insight

The DSPark paper[14] (§5: Load-adaptive throughput) describes dynamic adjustment of draft length based on acceptance rate feedback. The key idea: adapt to workload characteristics in real-time.

Implementation

We added adaptive state tracking to the MTP implementation in common/speculative.cpp[13]:

struct adaptive_draft_state {
    std::deque<float> recent_acceptance_rates;  // Last 20 rounds
    size_t max_history = 20;
    float target_acceptance = 0.30f;             // Target 30% acceptance
    float min_acceptance = 0.15f;                // Reduce n_max below this
    float max_acceptance = 0.50f;                // Increase n_max above this

    int current_n_max;                           // Current adaptive value
    int base_n_max;                              // Original configured value
    int min_n_max;                               // Minimum allowed (base/2)
    int max_n_max;                               // Maximum allowed (min(base*2, 8))

    uint64_t total_drafts_generated = 0;
    uint64_t total_drafts_accepted = 0;
    bool enabled = false;                        // Feature flag
} adapt;

The adaptation logic runs in the accept() function:

void accept(llama_seq_id seq_id, uint16_t n_accepted, bool /*is_other*/) override {
    // ... existing code ...

    // Track acceptance rate for adaptive draft tuning
    if (adapt.enabled && n_rows > 0) {
        float acceptance_rate = (float)n_accepted / (float)n_rows;

        adapt.total_drafts_generated += n_rows;
        adapt.total_drafts_accepted += n_accepted;

        adapt.recent_acceptance_rates.push_back(acceptance_rate);
        if (adapt.recent_acceptance_rates.size() > adapt.max_history) {
            adapt.recent_acceptance_rates.pop_front();
        }

        // Adjust every 5 rounds
        if (adapt.recent_acceptance_rates.size() >= 5 && 
            adapt.recent_acceptance_rates.size() % 5 == 0) {
            int new_n_max = compute_adaptive_n_max();
            if (new_n_max != adapt.current_n_max) {
                SPC_INF("MTP adaptive: n_max %d -> %d (avg %.1f%%)\n",
                        adapt.current_n_max, new_n_max,
                        acceptance_rate * 100);
                adapt.current_n_max = new_n_max;
            }
        }
    }
}

The compute_adaptive_n_max() method implements the DSPark-inspired logic:

int compute_adaptive_n_max() {
    if (!adapt.enabled || adapt.recent_acceptance_rates.empty()) {
        return adapt.base_n_max;
    }

    float avg = average(recent_acceptance_rates);

    if (avg < min_acceptance) {
        // Too many rejections - reduce draft length
        return max(min_n_max, current_n_max - 1);
    } else if (avg > max_acceptance) {
        // High acceptance - increase draft length
        return min(max_n_max, current_n_max + 1);
    }

    return current_n_max;  // Keep current
}

Unit Tests

We created comprehensive unit tests in tests/test-adaptive-draft.cpp[15] covering:

1. Disabled adaptive returns base_n_max
2. Empty history returns base_n_max
3. Low acceptance (<15%) reduces n_max
4. High acceptance (>50%) increases n_max
5. Normal acceptance (15-50%) keeps n_max unchanged
6. Min boundary enforcement
7. Max boundary enforcement
8. Sliding window behavior
9. Average calculation correctness
10. Initialization validation

All 10 tests pass. See commit f2d440806[13].

Expected Impact

• Decode speed: +10-20% from adaptive tuning alone
• GPU utilization: Better saturation during high-acceptance periods
• Waste reduction: Less compute wasted on rejected drafts
• Combined with cache fix: Total expected +30-50% (~28-32 tok/s vs current ~20 tok/s)

Enabling Adaptive Tuning

The adaptive draft tuning is now available via CLI flag:

llama-server \
  --model Qwen3.6-27B-MTP.gguf \
  --spec-type draft-mtp \
  --spec-draft-adaptive \
  --n-gpu-layers-draft 999 \
  ...

Or via environment variable:

export LLAMA_ARG_SPEC_DRAFT_ADAPTIVE=1

This makes it easy for automated testing and benchmarking without code modification. See commit 6811dcdb0[16] for implementation details.

The Complete Optimization Stack

Combining all our optimizations, here's the complete picture:

Layer 1: ATSInfer Tensor Placement

• Smart n_gpu_layers detection
• Contiguous block promotion
• MoE expert hot cache
• Gain: +38% for partial offload, neutral for full offload

Layer 2: MTP Speculative Decoding

• Draft context GPU acceleration (commit d1a54d8c7)
• Cache type inheritance (commit f2d440806)
• Gain: +54% decode speed (13 → 20 tok/s)

Layer 3: Adaptive Draft Tuning

• DSPark-inspired n_max adjustment (commit f2d440806)
• Acceptance rate feedback loop
• Gain: +10-20% additional speed (expected 24-28 tok/s)

Combined Expected Performance

We're closing the gap to wolfy's ~42 tok/s baseline, with remaining optimizations (batch size tuning, parallel context reduction, memory pressure reduction) expected to bridge the rest.

The Broader Ecosystem

Our work sits at the intersection of several research threads:

1. ATSInfer [cite:feng2025hspp] — Static tensor placement
2. DSPark — Load-adaptive speculative decoding
3. Turbo KV — from spiritbuun/buun-llama-cpp[2]: turbo1–turbo8, TCQ, VBR, turbo flash-attn, FWHT rotation
4. MoE Hot Cache — Selective expert promotion
5. GDN Rows Mode — Recurrent state optimization

See also: Qwen3.6 technical report[17], llama.cpp main repo[18], independent validation[19].

Each contributes to the overall goal: maximizing inference speed on consumer hardware.

Lessons Learned

1. Read the Code, Not Just the Paper

The paper describes an elegant algorithm, but the devil is in the implementation details. We spent hours debugging issues that weren't mentioned in the paper: - Scheduler reset wiping runtime hints - GDN fused ops requiring backend alignment - Turbo KV requiring full GPU offload - Draft context not inheriting GPU layer settings

2. Measure Everything

Without detailed logging and profiling, we would have missed: - The 0% draft acceptance rate (draft running on CPU) - The cache type mismatch (f16 vs q5_1/q5_0) - The low GPU utilization (25-30% instead of 60%+)

Tools like nsys profile, custom eval callbacks, and detailed logging were essential.

3. Start Simple, Then Optimize

We initially tried complex runtime tensor migration. It failed. The simple solution (load-time placement via tensor_buft_overrides) worked perfectly.

Similarly, adaptive draft tuning started as a simple sliding window average. Future enhancements (system load monitoring, per-sequence adaptation) can build on this foundation.

4. Document As You Go

The ATSInfer documentation[9] and todo files (~/.todos/) became invaluable references. They captured: - Diagnostic findings - Fix rationales - Performance baselines - Future optimization plans

This documentation made it easy to write this blog post and share knowledge with the community.

Future Work

Immediate Next Steps

1. Test on otter_den — Empirical validation of cache type fix and adaptive tuning
2. Batch size tuning — Optimize --ubatch-size and --parallel for MTP
3. Memory pressure reduction — Smaller context, aggressive quantization
4. CLI flag for adaptive mode — Make it user-configurable

Medium-Term Enhancements

1. System load integration — Monitor CPU/GPU utilization for adaptive decisions
2. Per-sequence adaptation — Independent n_max per request
3. Verification budget (DSPark Algorithm 2) — Full load-aware dynamic transfer
4. Pipeline parallelism — Overlap draft generation with target verification

Long-Term Vision

1. Unified scheduling framework — Combine ATSInfer placement, MoE hot cache, and adaptive speculative decoding
2. Auto-tuning — Self-optimizing based on workload characteristics
3. Multi-model support — Extend beyond Qwen3.6 to other architectures
4. Community integration — Upstream improvements to llama.cpp mainline

References

[1] Feng, S. and others. "Automated Tensor Scheduling for Hybrid {CPU-GPU} {LLM} Inference on Consumer Devices". 2025.

[2] spiritbuun. "spiritbuun/buun-llama-cpp". 2024.

[3] Kade. "ATSInfer placement solver source". 2025.

[4] Kade. "Commit fee98a844 on llama.cpp fork". 2025.

[5] Kade. "Cache invalidation analysis for hybrid models". 2025.

[6] llama.cpp contributors. "PR #22929: role-based prompt splitting". 2025.

[7] llama.cpp contributors. "PR #13194: SWA support and unified KV cache". 2025.

[8] llama.cpp contributors. "PR #24035: avoid unnecessary checkpoint invalidation". 2025.

[9] Kade. "ATSInfer documentation". 2025.

[10] Kade. "Speculative decoding source". 2025.

[11] Kade. "Commit d1a54d8c7: fix speculative MTP draft params". 2025.

[12] Kade. "CLI argument parsing source". 2025.

[13] Kade. "Commit f2d440806: adaptive draft n_max tuning". 2025.

[14] Park, J. and others. "DSPark: Block-Diffusion Speculative Decoding". 2025.

[15] Kade. "Adaptive draft test source". 2025.

[16] Kade. "Commit 6811dcdb0: add --spec-draft-adaptive flag". 2025.

[17] Qwen Team. "Qwen3.6 Technical Report". 2025.

[18] Gerganov, G. and others. "llama.cpp: Inference of LLMs in C/C++". 2025.

[19] Ichim, D.. "Independent validation: codeneedle benchmark". 2025.