{"id":23881,"date":"2026-09-10T15:42:35","date_gmt":"2026-09-10T15:42:35","guid":{"rendered":"https:\/\/scannn.com\/magic\/"},"modified":"2026-09-10T15:42:35","modified_gmt":"2026-09-10T15:42:35","slug":"magic","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/magic\/","title":{"rendered":"Magic"},"content":{"rendered":"\n

Research update on compute-efficient pretraining and scaling to trillion-parameter models.<\/p>\n

\n

Frontier pretraining is said to be a big-lab-only game. We don\u2019t have 100k chips yet, so there\u2019s only one way: algorithmic efficiency. After compounding for \u2026 a while \u2026, our pretraining recipe is now >10x more compute-efficient than that of leading open-weight base models.<\/p>\n

We match DeepSeek V4 Pro Base using ~50x fewer FLOPs \u2013 that\u2019s around half of GPT3\u2019s pretraining compute, or ~$0.5M on GB200. We continued scaling 10x (~$4M) and meaningfully outperformed all publicly available open base models on perplexity evals. By the scaling laws in Figure 1, training a model this capable would cost >$100M under DeepSeek V4 Pro\u2019s recipe (and this is ignoring how much data exists). Of course, we won\u2019t stop scaling there.<\/p>\n

We believe pretraining, agentic RL, and long-context are sufficient to build superhuman coding agents and automate AI R&D. We started with long-context<\/a>. Today\u2019s blog post is about pretraining.<\/p>\n

\n
\n