{"id":23352,"date":"2026-08-17T15:07:49","date_gmt":"2026-08-17T15:07:49","guid":{"rendered":"https:\/\/scannn.com\/teaching-everyone-to-fish-for-tokens\/"},"modified":"2026-08-17T15:07:49","modified_gmt":"2026-08-17T15:07:49","slug":"teaching-everyone-to-fish-for-tokens","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/teaching-everyone-to-fish-for-tokens\/","title":{"rendered":"Teaching Everyone to Fish for Tokens"},"content":{"rendered":"\n
\n
Housekeeping: No voiceover for this post as I\u2019m traveling.<\/span><\/h5>\n

The oldest comparison people try to make is how what\u2019s happening with open models compares to foundational open-source software projects like the Linux operating system. There are fairly clean analogies, but they paint a narrow path forwards for the self-sustaining nature of the open-source model ecosystem, where once Linux got big enough it was going to be self-fulfilling as the best possible tool for many jobs. The open-source language model \u2013 i.e. only models that come with a full training recipe, data, code, etc. \u2013 is a closer analogue to the open-source operating system. The open weight models you use \u2013 those with just model weights and inference code to run them \u2013 are closer to specific versions of software that you install in a project built upon them.<\/span><\/p>\n

Share<\/span><\/a><\/p>\n

Model weights are very transient on average, but they still have a long shelf life, as with a lot of heavily used software. It\u2019s why many companies are still using workflows built on Llama 3, despite agentic behaviors taking off years later. The open-source recipe, typified in modern times by the Olmo models I helped build at Ai2, with its predecessors like Pythia from EleutherAI, is a resource intensive process that any company can pick up, modify, and press \u201crun\u201d on to produce a new set of model weights. In the best cases, the community can contribute improvements in data or training code back into the next model too! This is why Nvidia is investing so much in nearly open-source models \u2013 for their Nemotron models they release all the data they legally can and the training code, etc. Nvidia wants a world where countless people can build token machines, so intelligence is not monopolized. This is a world with massive demand for inference across many companies, all of which want to buy Nvidia\u2019s offerings.<\/span><\/p>\n

Open-source AI has a tricky future, as building the best models is extremely capital intensive. The ability to build competitive models has stayed more accessible in industry longer than many would\u2019ve expected. The default expectation for many is that training models is too expensive and the open-source recipe is too far behind, so building a new lab centered on some part of training LLMs will not be tractable.<\/span><\/p>\n

There are two futures from here. First is if \u201cit works\u201d \u2013 if the open-source recipe works for Nvidia, they\u2019ll be creating far more demand for their chips (and profits) than it costs to build the models. Right now it\u2019s <\/span>reported<\/span><\/a> that Nvidia is spending $26 billion on this endeavor. It\u2019s not clear if this will work, or if AI\u2019s capital intensiveness will drive more and more companies out of the training game. We <\/span>haven\u2019t seen many signs of this starting<\/span><\/a>. In fact, the companies bowing out \u2013 like <\/span>Databricks<\/span><\/a> and 01.ai \u2013 seem like anomalies.<\/span><\/p>\n

The open-source ecosystem will become increasingly dependent on Nvidia\u2019s financing in the coming years. This is an existential window, where within a few years the profits of this approach need to return to them, or another open model company needs to cultivate platform-like financial feedback loops on their openness. This economic reward needs to be proportional to the profits generated by Anthropic and OpenAI\u2019s APIs to keep pace over decades of language model development. This can be driven by competitiveness on performance or by the AI boom just being so big that the open model training, inference, and fine-tuning companies all have vast quantities of demand.<\/span><\/p>\n

The second future is if one of these two financially positive paths doesn\u2019t play out, open models will fork to a different development path than the leading closed models \u2013 one more focused on efficiency, modifiability, specialization, etc. I put this mentally as <\/span>my most likely outcome<\/span><\/a> \u2013 open models are still incredibly useful, but <\/span>fill a long-tail ecosystem<\/span><\/a> relative to the closed counterparts that have monopoly ownership stakes in the most valuable areas like knowledge work collaboration, drug discovery, SWE, etc. The long-tail is something like enterprise-specific agents that run on-prem with private data on repetitive business tasks.<\/span><\/p>\n

Part of why I think this open-source training will have a hard time catching on is because training is getting more complex and more abstracted. The current open model ecosystem is buoyed by an explosion in interest in post-training open models. These people take models like DeepSeek V4 Flash, Inkling Small, or GLM 5.X and finetune them for their specific agentic tasks (e.g. in Tinker, the most popular finetuning API today).<\/span><\/p>\n

Over the last few years, post-training largely referred to the whole process of modifying the base model to make it intelligent and usable. There is a shift happening where the ability to train a base model to be a general agentic reasoner is becoming opaque like at-scale pretraining practices from a few years ago. This could go so far as to change the established pretraining, midtraining, post-training lexicon that has been standard for a few years. It could come to be something closer to pretraining, reasoning training, and post-training.<\/span><\/p>\n

As there\u2019s less interest in training the entire model, there\u2019s less interest in investing in open-source AI. These are the only sort of hints we will get, but we cannot do much to fight the economic gravity of these situations. This trend is the next step in the number of open model builders who release base models (the model versions before core reasoning training) continuing to decrease. It goes hand in hand with open model builders experimenting with <\/span>revenue<\/span><\/a> <\/span>share<\/span><\/a> licenses for downstream use in products or inference. These are experiments in keeping the financing viable for building near frontier open-weight models \u2013 a lot hinges in the near future on how successful they are. These are the people that need to succeed for Nvidia\u2019s demand-growth strategy around open-source to succeed, and last.<\/span><\/p>\n

Along the way we\u2019re still in for a ton of action in open-weight models, as releasing access to intelligence is one of the strongest business strategies available. This additional type of player, who monetizes the AI indirectly, is typified by Meta and other hyperscalers with massive balance sheets. Meta <\/span>releasing its very-strong Muse Spark 1.2 model<\/span><\/a> as open-weights would severely hamper the revenue growth rate of their competitors in Anthropic and OpenAI who rely on selling tokens. These companies are both commoditizing their complements, but they\u2019re doing it in different ways. Nvidia wants to teach everyone to fish for tokens, so the ecosystem is self-sustaining, but Meta is strategically flooding the zone with tokens.<\/span><\/p>\n<\/div>\n

Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Housekeeping: No voiceover for this post as I\u2019m traveling. The oldest comparison people try to make is how what\u2019s happening with open models compares to foundational open-source software projects like the Linux operating system. There are fairly clean analogies, but they paint a narrow path forwards for the self-sustaining nature of the open-source model ecosystem, […]<\/p>\n","protected":false},"author":16,"featured_media":23353,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23352","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23352","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23352"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23352\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23353"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23352"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23352"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23352"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}