05 Meta

1. Company and research division. Meta Infrastructure / Engineering at Meta, with Meta Superintelligence Labs and NVIDIA (co-engineering); separately, Meta AI research (RPL paper).

2. Date of publication / announcement. Prometheus: Engineering at Meta, 19 October 2026 (exact publication time not confirmed). RPL: AI at Meta publication page dated 20 October 2026.

3. Research title and field. (a) 'Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster'. Field: AI training infrastructure, reliability. (b) 'Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning' (RPL). Field: retrieval-augmented agents, RL.

4. Research background. At frontier scale, a single fault can halt synchronous training. Goodput and mean time between interruptions (MTBI) therefore become first-order determinants of effective compute.

5. Scientific innovation. (a) Disclosure of fleet-scale reliability engineering for NVL72 domains: in-rack sparing, parallelism-aware fault tolerance, firmware automation and power-transient management. (b) RPL lets a policy compose search primitives into executable programs with local state. 'Program mutation' is a grammar-constrained, execution-verified way to generate improved trajectories that are distilled back into the policy.

6. Methodology. (a) Hardware platforms: Catalina (GB200 NVL72 'Ariel', 800G scale-out per GPU), Clemente (GB300 NVL72) and the forthcoming San Miguel (Vera Rubin NVL72). Networking: building fabrics of up to 20,736 GPUs (DSF, then NSF on 51.2T switches) and BAG aggregation over 2 km to more than 100 km (Arista 7800 / Broadcom Jericho3). Training stack: PyTorch with FSDP, plus Tectonic and LogDevice storage. (b) Evaluation on open-domain QA and corpus-level reasoning at two model sizes (4B and 8B).

7. Results. (a) Goodput above 85% for pre-training and above 90% for RL (Muse Wonder). MTBI was about 47k GPU-hours during those runs and is now above 54k GPU-hours. Interruptions fell by 99% from a mid-2025 peak of about 1,000 events per 8K GPUs per day; the target is 2 events per 8K GPUs per day (about 100k GPU-hours MTBI). Sparing: a 64-of-72 slice plus about 2% spare racks means 523 racks (37,656 GPUs) for a 32,768-GPU job, about 13% overhead, of which 11% comes from slice geometry. Firmware P90 per-pod upgrade time fell from more than 7 h to 2 h. Power transients swing 30–40%. The San Miguel spec claims 2.52 EFLOPS FP4 per rack and a 10–20% performance-per-watt gain over GB300. (b) A 4B RPL model outperformed all 8B baselines on open-domain QA, according to the abstract; it also had the highest answer F1 and document-coverage F1 on corpus-level tasks, with fewer retrieval turns.

8. Scientific significance. This is rare quantitative disclosure of how reliability, rather than peak FLOPS, bounds frontier training, and it is highly relevant to co-design of scale-up domains. RPL shows that small models can learn retrieval strategies that beat larger fixed pipelines.

9. Technical limitations. (a) Company-reported figures that have not been audited. The primary page returned HTTP 403 to our fetch, so the text was verified via a verbatim mirror. (b) RPL is an arXiv preprint; the full paper was not reviewed beyond the abstract.

10. Technology readiness and evidence type. (a) Commercial (operational infrastructure); evidence: company engineering disclosure. (b) Lab prototype; evidence: preprint.

11. Competitive comparison. Comparable published figures are few. Earlier Meta 24K and 100K-GPU cluster reports, and Llama 3 training-interruption statistics (2024), provide baselines. No other hyperscaler disclosed comparable MTBI data at the time of writing.

12. Future research (1–5 years) — FORECAST. FORECAST (1–5 years): reliability (MTBI) becomes a procurement metric alongside FLOPS; larger scale-up domains drive in-rack sparing as standard; Vera Rubin integration in 2027; RPL-style programmable retrieval inside agent frameworks.

13. Original references.

• https://engineering.fb.com/ (Prometheus post, 19 Oct 2026; HTTP 403 to fetch; verbatim mirror https://enggist.vercel.app/post/97ee8863-91bf-44d6-ad38-b10f9a857944)
• https://ai.meta.com/research/publications/learning-to-program-retrievals-composing-search-primitives-via-reinforcement-learning/