{"id":23189,"date":"2026-08-11T17:18:25","date_gmt":"2026-08-11T17:18:25","guid":{"rendered":"https:\/\/scannn.com\/no-local-models-will-not-win\/"},"modified":"2026-08-11T17:18:25","modified_gmt":"2026-08-11T17:18:25","slug":"no-local-models-will-not-win","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/no-local-models-will-not-win\/","title":{"rendered":"No, local models will not win"},"content":{"rendered":"\n
\n

Every time a new open-weight AI model is released, people say<\/a> that local models are the future. Why spend billions of dollars building out datacenters when everyone will just be able to run AI models on their laptops or phones? I think this idea is doomed. No matter how strong open-weight models get, most inference will always happen in AI datacenters.<\/p>\n

Local models are too weak to be widely used<\/h3>\n

Local models are never going to be as powerful<\/strong>. I think this point should be obvious: all of the current frontier models (closed and open-weights) are far too big to run on anything but a full GPU cluster in a datacenter. Of course, smaller models are getting more intelligent over time. In a year you might be able to run something about as strong as GPT-5.6-Sol on your laptop. But by then, you\u2019ll think of GPT-5.6-Sol as too weak to be useful.<\/p>\n

Many people deny this last point, but it\u2019s true: almost everyone\u2019s revealed preference is to use the strongest available model in their price range<\/strong>. If AI progress had stalled at GPT-4, I think we could have built some very powerful tools around it, but who\u2019d use GPT-4 today? As LLMs have gotten more capable, our expectations around them have grown: we now expect agentic systems to be able to solve more and more problems independently. It\u2019s intensely frustrating when they get confused or stall out. When given a choice, people are going to pick the model that frustrates them less, which is always going to be the bigger, more powerful one.<\/p>\n

Local models are more expensive and less efficient<\/h3>\n

On top of that, datacenter models are always going to be cheaper<\/strong>. I don\u2019t understand why people keep saying that local models are cheap: it seems to me to be the same mistake people make when they say that driving Uber is \u201cfree money\u201d (ignoring the costs of fuel and wear-and-tear on your car). For the setup price alone of a low-end home lab<\/a>, you could buy several years of a paid subscription to one of the AI providers. The power costs would come out to around $50-$300 per month, depending on how much inference you\u2019re running: again, the price of a couple more paid subscriptions.<\/p>\n

Why are datacenter models cheaper? It\u2019s not because datacenter inference is subsidized: inference is actually fairly cheap<\/a>. If you\u2019re running the same model locally and in a datacenter, the datacenter model will be inherently more efficient<\/strong>.<\/p>\n

The main reason is batching<\/strong>. A GPU can do hundreds of thousands of mathematical operations exactly as quickly as it can do one. However, for a single user\u2019s inference, each new token depends on the result of the previous one, so it can\u2019t be batched. What can be batched is the inference of hundreds of users together. This costs essentially as much time, power, and heat as just doing inference for one user at a time.<\/p>\n

When you\u2019re running your own inference at home, you\u2019ve got nothing to batch \u2014 at best you\u2019re running a few parallel AI agents \u2014 so utilization is terrible. There\u2019s a lot of potential inference that you\u2019re paying for but can\u2019t use: it\u2019s just being wasted. The only way around this is to get together with some friends and expose your local inference endpoint to them (at which point you\u2019re basically running your own crappy datacenter).<\/p>\n

The other reason is that datacenters have larger, more efficient GPUs to work with<\/strong>. The kind of consumer GPUs you\u2019d run local models on are gaming GPUs like the RTX 4090. A datacenter B200, designed for batched AI inference, gets about three times the flops and just under four times the memory bandwidth for the same amount of power. So between batching and GPU efficiency, you\u2019re using something like ~30x the resources to run your model locally.<\/p>\n

Incidentally, this is why I\u2019m suspicious of people who say that local models are good because they aren\u2019t as resource-hungry as those big bad datacenters. If you want to run LLMs efficiently, you should be trying to push as much of your use into AI datacenters as possible! Charitably, what they mean is that we should all be running smaller<\/em> models \u2014 but even then, you should ideally be using small models via, say, the GPT-5.6 Luna<\/a> API instead of hosting your own model.<\/p>\n

How might local models win anyway?<\/h3>\n

Is there a possible world in which local models win? I suppose so. One thing that could happen is that governments could ban the use of AI datacenters altogether: either due to concerns around the danger of AI, or simply bending to public pressure<\/a>. In that world, local models would be the only game in town.<\/p>\n

Alternatively, AI progress might somehow stall for very large models while progressing for small ones. I struggle to imagine how this might happen (barring government intervention, as above), but a world where a 30B parameter model could be a frontier model is a world where local models might be competitive.<\/p>\n

Or maybe models get so<\/em> good that a 30B model is genuinely smart enough to do everything, so nobody really needs a model like Opus or Sol unless they\u2019re trying to solve the Reimann Hypothesis. I don\u2019t really buy this. Models can do frontier mathematical work today while still being not smart enough to refactor large codebases as well as me, so it\u2019s hard to imagine a world where I don\u2019t just want to use the smartest model available.<\/p>\n

Local models are not useless<\/h3>\n

I do think there will always be a niche for local models. I\u2019m reminded of the surprisingly simple idea behind Thinking Machines\u2019 \u201cInteraction Models\u201d<\/a> (which OpenAI also does<\/a>, because it\u2019s obvious): for latency-sensitive applications like voice chat, you have a small, fast model handle the talking, which delegates to a large, slower model for the hard thinking. I wouldn\u2019t be surprised if most AI use in five years is mediated through a local model on your phone or laptop (though in this world almost all the work would still be done via AI datacenters).<\/p>\n

Some users will prefer local models even though they\u2019re weaker and more expensive. For instance, being able to steer the model locally<\/a> might be a killer feature for those users. Others might simply value having total control over their own infrastructure, or have unreliable internet. If you\u2019re one of those people \u2014 particularly if you only chat to the models instead of using them for research or coding \u2014 local models are a good choice for you. However, I think this is always going to be a niche group. The majority of users will continue to do their inference through datacenters.<\/p>\n<\/div>\n

Here’s a preview of a related post that shares tags with this one.<\/p>\n

Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Every time a new open-weight AI model is released, people say that local models are the future. Why spend billions of dollars building out datacenters when everyone will just be able to run AI models on their laptops or phones? I think this idea is doomed. No matter how strong open-weight models get, most inference […]<\/p>\n","protected":false},"author":16,"featured_media":23190,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23189","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23189","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23189"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23189\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23190"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23189"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23189"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23189"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}