20 IBM

1. Company and research division. IBM Research (AI native systems; Spyre accelerator software) and IBM/Red Hat as llm-d co-founders. Torch-Spyre authors: D. Grove, A. Viros i Martin, A. Blanchard. Related IBM Research talks: MAKCI (S. Jain, R. Raj) and model-derived operator tests (K. Ishizaki et al.).

2. Date of publication / announcement. 20 October 2026: IBM Newsroom (llm-d 1.0) and IBM Research publication pages for PyTorch Conference 2026.

3. Research title and field. (a) llm-d 1.0, an open distributed LLM inference platform (CNCF). (b) Torch-Spyre, a PyTorch PrivateUse1 + Inductor backend for the IBM Spyre accelerator. Field: inference systems, accelerator software.

4. Research background. Agentic workloads are context-heavy and bursty. Managing the KV cache often costs more than generating the output. Out-of-tree accelerators need deep PyTorch integration to run the model ecosystem.

5. Scientific innovation. (a) Production-hardened distributed inference. Features: multi-tenant QoS (priority- and fairness-aware flow control), zero-downtime operation (router HA, coordinated prefill/decode rollouts), a v1 configuration API, and agent-aware serving (session-aware routing, turn-priority scheduling, native Responses-API state). It runs on vLLM and SGLang. (b) Spyre is a dataflow accelerator with 32 AI cores, on-chip interconnect and compiler-managed scratchpad. Torch-Spyre adds device-specific tensor layouts and scratchpad-optimised tiling integrated with upstream PyTorch. MAKCI provides analytical kernel cost models for Triton on Spyre without hardware profiling.

6. Methodology. (a) Demonstrations cited in the release: GLM 5.2 (about 753B parameters, open-weight MoE) on 'widely deployed NVIDIA GPUs' with hundreds of parallel agentic sessions; an IBM Research quantum-error-correction exploration workload. (b) Conference talk abstract; the operator-test framework covers 116 operators from 8 models.

7. Results. (a) Peak output above 6.6 million tokens per minute; up to 3,000 concurrent coding agents with zero preemptions; an estimated cost 5–10× lower per token than equivalent commercial API pricing at current cloud rental rates; 11.5× lower inference cost than an all-proprietary configuration, about 98% prefix-cache hit rate and about $0.11 per million tokens blended (IBM Research use case). The project has more than 500 contributors from more than 25 organisations. (b) Spyre 'can now run thousands of models from Hugging Face and vLLM'; no performance numbers in the abstract.

8. Scientific significance. This marks the maturation of open, Kubernetes-native inference infrastructure tuned for agents, a direct response to the shift in infrastructure demand from training to inference.

9. Technical limitations. The performance figures are contributor demonstrations with methodology in linked blogs that were not reviewed here, and the cost comparisons depend on rental-price assumptions. Torch-Spyre performance has not been disclosed.

10. Technology readiness and evidence type. llm-d 1.0: commercial (production-adopted open source; JetBrains, DigitalOcean and Roblox cited). Torch-Spyre: validated prototype / pilot (conference presentation). Evidence type: press release and conference abstracts (preliminary or marketing claims).

11. Competitive comparison. Comparable stacks are NVIDIA Dynamo (cited as complementary), SGLang and vLLM routers, and proprietary hyperscaler serving systems. No neutral benchmark is available.

12. Future research (1–5 years) — FORECAST. FORECAST (1–5 years): llm-d becoming a CNCF-standard inference layer across GPU, TPU and Instinct fleets; agent-session-aware scheduling as a default; Vera Rubin support (cited by NVIDIA).

13. Original references.

• https://newsroom.ibm.com/2026-10-20-llm-d-1-0-advances-open,-cost-efficient-ai-inference-for-production-scale
• https://research.ibm.com/publications/integrating-the-ibm-spyre-accelerator
• https://research.ibm.com/publications/makci-a-multi-abstraction-ai-kernel-cost-instrumentor-for-performance-modeling-in-ai-compilers
• https://research.ibm.com/publications/real-shapes-real-bugs-model-derived-operator-tests-for-quick-out-of-tree-accelerator-bring-up