{"id":22565,"date":"2026-07-20T15:28:14","date_gmt":"2026-07-20T15:28:14","guid":{"rendered":"https:\/\/scannn.com\/on-kimi-k3-its-capabilities-and-related-discontents\/"},"modified":"2026-07-20T15:28:14","modified_gmt":"2026-07-20T15:28:14","slug":"on-kimi-k3-its-capabilities-and-related-discontents","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/on-kimi-k3-its-capabilities-and-related-discontents\/","title":{"rendered":"On Kimi K3: Its Capabilities And Related Discontents"},"content":{"rendered":"\n
\n

Kimi K3 is a very good model with excellent benchmarks. Assuming its weights are released as planned it will become, purely in terms of raw capability, the strongest open model.<\/p>\n

Do not get carried away. Do not judge Kimi K3 only its relative strengths. In aggregate it is several months behind the closed model frontier, at least four and my median guess is six, with the post-training closer and the pre-training farther out. This is less months than before, but the months are denser now.<\/p>\n

It is somewhat distilled. It likely outperforms on benchmarks relative to practical performance. All its benchmarks are scored at maximum effort, typically a lot more tokens than are used in similar tests by Fable or Sol. Performance looks jagged. Kimi will be excellent at some things, less so at other things.<\/p>\n

We will know more over the coming weeks. For now access is spotty and not that many people have actually had the chance to try Kimi K3, so I have larger error bars than usual around its capabilities. Alas, time waits for no one, so we press on.<\/p>\n

It is the largest open model so far at 2.8T, on the upper end of possible sizes for Claude Opus and near the bottom of possible sizes for Mythos, which explains many of its gains. It is slow and appears hungry for tokens. A lot of the positive reactions are to this being a big model, which thus has at least a decent amount of \u2018big model smell\u2019 and generally trades being slower and more expensive for some performance gains. That\u2019s a great move, but it should be a while before they can do it again.<\/p>\n

Distillation from Claude is clearly part of the story, likely largely from Fable, and is clearly nothing close to the whole story. Clearly Moonshot would do this even if it helped only a little. The timing of them releasing a bigger model is suggestive.<\/p>\n

It is again a good model, but once you correct for the overperformance on benchmarks and look at expected practical performance, it is not clear this is so different from what you would have expected from a Kimi K3 that had 2.8T parameters.<\/p>\n

Consider that Kimi K3\u2019s (preliminary unofficial) Epoch Capabilities Index is exactly on the Chinese trend line<\/a>.<\/p>\n

Kimi K3 is absolutely worth checking to see if it fits into your workflows. At this price point, for both the API and the subscription, it is not going to fill the role of the smaller cheaper open models, and I expect it to usually lose out in a fight with the top closed models, but there are going to be some places where Kimi K3 is a good choice.<\/p>\n

\n

Andrew Curran<\/a>: Following the success of Kimi K3, Moonshot AI has informed investors that it plans an IPO in Hong Kong within the next six months, according to Bloomberg.<\/p>\n<\/blockquote>\n

Good idea. Strike while the iron is hot.<\/p>\n

Table of Contents<\/h4>\n
    \n
  1. DeepSeek Moments: Here We Go Again.<\/a><\/li>\n
  2. We Had a Moment (Reprise from June 2025).<\/a><\/li>\n
  3. The Story Since Then.<\/a><\/li>\n
  4. The Kimi K3 Announcement, Pitch and Basic Facts.<\/a><\/li>\n
  5. On Modern Benchmaxxing.<\/a><\/li>\n
  6. Other People\u2019s Benchmarks.<\/a><\/li>\n
  7. Benchmarks Are Not The Real World.<\/a><\/li>\n
  8. Technical Safeguards? What Are Those?<\/a><\/li>\n
  9. Things Kimi Can Do.<\/a><\/li>\n
  10. Things Kimi Cannot Do.<\/a><\/li>\n
  11. Things It Is Not Easy To Get Kimi To Do.<\/a><\/li>\n
  12. Open Weight Models Are Unsafe And Nothing Can Fix This.<\/a><\/li>\n
  13. Dean Ball Attempts To Be Constructive.<\/a><\/li>\n
  14. OpenAI Employees Are Relatively Bullish On This One.<\/a><\/li>\n
  15. Kimi K3 Is Relatively Strongest At Typical Agentic Coding and 3D.<\/a><\/li>\n
  16. Reactions.<\/a><\/li>\n
  17. Who Are You?<\/a><\/li>\n
  18. How Did They Do It?<\/a><\/li>\n
  19. Conclusion.<\/a><\/li>\n<\/ol>\n

    DeepSeek Moments: Here We Go Again<\/h4>\n

    All discourse about Chinese models lives in the shadow of the DeepSeek moment<\/a>.<\/p>\n

    There are a lot of people who really, really want another DeepSeek moment<\/a> to happen.<\/p>\n

    These people really, really want to tell the story that Chinese open models are catching up to American closed models, that AI and inference will become commoditized.<\/p>\n

    Their motivations vary. They often want to affirm open models, or the importance of the \u2018tech stack.\u2019 Others simply want to see OpenAI and Anthropic go down, or know that such claims sell. Often the ultimate objective is to argue against all AI regulations, or anything that might \u2018slow us down\u2019 or cause us to \u2018lose to China.\u2019<\/p>\n

    It is actively suicidal to respond to<\/a> \u2018the Chinese have better models now\u2019 with \u2018then we had better sell them the compute so they can run them and also build even better ones.\u2019 Yet every time, yes, people will argue that. Sigh.<\/p>\n

    Often they simply want to tell American AI to stop taking precautions, to stop being annoying and take down the classifiers<\/a>, as in \u2018genie is out of the bottle, so release the bigger genie with unlimited wishes, it\u2019s the only way.\u2019 People really would take major catastrophic risks rather than deal with classifiers, and are Big Mad about this.<\/p>\n

    Google was down 4.4% on the day, SpaceX was down 3.1%<\/a> and Nvidia down over 2%, and tech stocks were down again on Friday, so plausibly we\u2019re doing this again.<\/p>\n

    And yep, we are at risk of doing this again<\/a>:<\/p>\n

    \n

    Axios<\/a> (being wrong): China just erased America\u2019s AI lead<\/p>\n

    Se\u00e1n \u00d3 h\u00c9igeartaigh<\/a>: No it didn\u2019t. (although Kimi K3 is undoubtedly impressive)<\/p>\n<\/blockquote>\n

    The pattern is:<\/p>\n

      \n
    1. Chinese model releases.<\/li>\n
    2. There is some impressive benchmark cited.<\/li>\n
    3. Therefore, America\u2019s lead is gone, QED, that\u2019s it, no, really, that\u2019s it.<\/li>\n<\/ol>\n

      In the Axios case the benchmark in question is Arena. Based on that alone, they state as fact that America\u2019s lead is gone.<\/p>\n

      I would ignore, but this style of logic has convinced a lot of Washington D.C. multiple times, and that has had substantial policy impact.<\/p>\n

      I shudder to think what such folks might do now that they also know about Mythos. The confusion over Fable jailbreaks could easily extend to a broader dumb panic.<\/p>\n

      The original DeepSeek moment<\/a> happened because of a confluence of events.<\/p>\n

      Let\u2019s review.<\/p>\n

      We Had a Moment (Reprise from June 2025)<\/h4>\n

      We all remember\u00a0The DeepSeek Moment<\/strong><\/a>, which led to\u00a0Panic at the App Store<\/strong><\/a>, lots of stock market turmoil that made remarkably little fundamental sense and that has been borne out as rather silly,\u00a0a very intense week<\/strong><\/a>\u00a0and\u00a0a conclusion to not panic after all<\/strong><\/a>.<\/p>\n

      Over several months, a clear picture emerged of (most of) what happened: A confluence of narrative factors transformed DeepSeek\u2019s r1 from an impressive but not terribly surprising model worth updating on into a shot heard round the world, despite the lack of direct \u2018fanfare.\u2019<\/p>\n

      In particular, these all worked together to cause this effect:<\/p>\n

        \n
      1. The \u2018six million dollar model<\/strong><\/a>\u2019 narrative. People equated v3\u2019s marginal compute costs with the overall budget of American labs like OpenAI and Anthropic. This is like saying DeepSeek spent a lot less on apples than OpenAI spent on food. When making an apples-to-apples comparison, DeepSeek spent less, but the difference was far less stark.<\/li>\n
      2. DeepSeek simultaneously released an app that was free with a remarkably clean design and visible chain-of-thought (CoT). DeepSeek was fast following, so they had no reason to hide the CoT. Comparisons only compared DeepSeek\u2019s top use cases to the same use cases elsewhere, ignoring the features and use cases DeepSeek lacked or did poorly on. So if you wanted to do first-day free querying, you got what was at the time a unique and viral experience. This forced other labs to also show CoT and accelerate release of various models and features.<\/li>\n
      3. It takes a while to know how good a model really is, and the different style and visible CoT and excitement made people think r1 was better than it was.<\/li>\n
      4. The timing was impeccable. DeepSeek got in right before a series of other model releases. Within two weeks it was very clear that American labs remained ahead. This was the peak of a DeepSeek cycle and the low point in others cycles.<\/li>\n
      5. The timing was also impeccable in terms of the technology. This was very early days of RL scaling, such that the training process could still be done cheaply. DeepSeek did a great job extracting the most from its chips, but they are likely going to have increasing trouble with its compute disadvantage going forwards.<\/li>\n
      6. DeepSeek leveraged the whole \u2018what even is safety testing\u2019 and fast following angles, shipping as quickly as possible to irrevocably release its new model the moment it was at all viable to do so, making it look relatively farther along and less behind than they were. Teortaxes notes that the R1 paper pointed out a bunch of things that needed fixing but that DeepSeek did not have time to fix back then, and that R1-0528 fixes them, and which weren\u2019t \u2018counted\u2019 during the panic.<\/li>\n
      7. DeepSeek got the whole \u2018momentum\u2019 argument going. China had previously been much farther behind in terms of released models, DeepSeek was now less behind (and some even said was ahead), and people thought \u2018oh that means soon they\u2019ll be ahead.\u2019 Whereas no, you can\u2019t assume that, and also moving from a follower to a leader is a big leap.<\/li>\n
      8. There was highly related to a widespread demand for a \u2018China caught up to the USA\u2019 narrative, from China fans and also from China hawks of all sorts. Going forward, we are left with a \u2018missile gap\u2019 style story.<\/li>\n
      9. There are also a lot of people always pushing the \u2018open models win\u2019 argument, and who think that non-open models are some combination of doomed and don\u2019t count. These people are very vocal, and vibes are a weapon of choice, and some have close ties to the Trump administration.<\/li>\n
      10. The stock market was highly lacking in situational awareness, so they considered this release much bigger news than it was, and it caused various people to \u2018wake up\u2019 to things that were already known and anticipate others waking up, and there was widespread misunderstanding of how any of the underlying dynamics worked, including Jevon\u2019s Paradox and also that if you want to run r1 you go out and buy more chips, including Nvidia chips. It is also possible that a lot of the DeepSeek stock market reaction was actually about insider trading of Trump policy announcements. Essentially: The Efficient Market Hypothesis Is False.<\/li>\n<\/ol>\n

        The Story Since Then<\/h4>\n

        Since then, the idea that China had caught up, or was catching up, kept coming up, drove much discussion around Washington, as echoes of this one moment.<\/p>\n

        This is then renewed every time a new strongest or exciting Chinese model comes out. Every day that China does not release a model, they look one day farther behind. When they do release, they \u2018catch up\u2019 and look less behind.<\/p>\n

        The top 10 such potential moments since r1 and before K3 were likely these:<\/p>\n

          \n
        1. Manus.<\/li>\n
        2. DeepSeek r1-0528.<\/li>\n
        3. Kimi K2.<\/li>\n
        4. GPT-5 (in reverse) which spooked a lot of people in highly stupid ways.<\/li>\n
        5. DeepSeek v3.1.<\/li>\n
        6. Kimi K2 Thinking.<\/li>\n
        7. DeepSeek v3.2.<\/li>\n
        8. Kimi K2.5.<\/li>\n
        9. DeepSeek v4.<\/li>\n
        10. GLM-5.2.<\/li>\n<\/ol>\n

          Mostly it\u2019s been Kimi and DeepSeek. Manus got a hype train going and spooked people, and GLM-5.2 was by far their strongest offering, putting GLMs on the map.<\/p>\n

          Many of these were good models, but none fundamentally changed the game.<\/p>\n

          Roughly, since the DeepSeek moment<\/a>, when DeepSeek was roughly eight months behind but had matched some key aspects faster via fast following, we have bounced around. For a while it looked like China was quite a lot behind.<\/p>\n

          GLM-5.2 and Kimi K3 have been impressive. The current best estimate of the time gap is at its lowest point. Events in AI have accelerated all around, so it is not clear that the gap is fewer product cycles than before, and I half expect to be doing this again next week for Qwen. Kimi K3 is 2.8T and seems to still be solidly behind Mythos Preview, which was announced on April 7, so that provides a starting point lower bound of a three month gap.<\/p>\n

          \n

          Ethan Mollick<\/a>: Kimi is, as I have been saying, a very good model. But it is not a DeepSeek r1 moment, in that it is roughly where I would expect on the curve rather than an unexpected leap. It will be treated as a DeepSeek moment<\/a> for a variety of reasons especially as more people hear about it.<\/p>\n<\/blockquote>\n

          Ryan Greenblatt, despite being pleasantly surprised by Kimi K3, estimates that the pretrain quality is about halfway between Opus 4 and Opus 4.5<\/a> based on forward pass math, but with some other advantages, so ~8 months behind, as one might expect.<\/p>\n

          The post-training is closer, at least in part because of distillation. Moonshot is clearly innovating, but it is also clearly distilling, both directly and also fast following via looking at outputs and copying techniques.<\/p>\n

          UK AISI issued a report on everything prior to Kimi K3<\/a>, showing the time gap for narrow cyber tasks narrowing somewhat over time. Their full report is here.<\/a><\/p>\n

          \n
          \n
          \n
          <\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

          My understanding is that narrow and relatively easy coding tasks are where open weights model are at their relative strongest, and the benchmark here is approaching saturation.<\/p>\n

          Indeed, when you look at the full post, you get a different answer for longer tasks.<\/p>\n

          \n
          \n
          \n
          \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n
          \n

          UK AI Security Institute<\/a>: On TLO, GLM-5.2 reaches as far as Opus 4.5, a model released less than 7 months before it,<\/strong> while DeepSeek\u2019s V4-Pro falls below Sonnet 4.5 (a sub-cyber-frontier model released 7 months before it). These results are broadly consistent across our other cyber ranges. Notably, GLM-5.2 reached step 7 with marginally fewer tokens than any other model on average, tracking Opus 4.6\u2019s trajectory to step 11 before stalling.<\/p>\n<\/blockquote>\n

          Longer tasks are more relevant in terms of both of the most important things to worry about: Automation of AI R&D and cyber attacks.<\/p>\n

          One might also worry about bio risks, even if that is not as in fashion, and it is a little concerning we don\u2019t see standard testing on that at all for the open models. That needs to be addressed. We do have the score on OpenAI\u2019s GeneBench-Pro via Andrew Ho<\/a>. This measures judgment under long-horizon ambiguity in computational biology. Kimi K3 exceeded expectations. Mythos have not been tested. Fable refused most requests in the benchmark.<\/p>\n

          \n
          \n
          \n
          \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n
          \n
          \n
          \n
          \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

          Beating Opus and GPT-5.5 is impressive stuff. No previous open model came close. We are on the verge of doing some f***ing around and thus finding out. This is a place where plausibly not much happens until suddenly quite a lot happens.<\/p>\n

          The conclusion of \u2018open models will catch up to Mythos in general capability including the thing that currently makes it unique\u2019 is indisputable. That is coming, the question is when, and that will establish the effective gap. Given Kimi K3 we should expect this to happen a few months from now.<\/p>\n

          If we take both ends of UK AISI\u2019s estimates, the pre-Kimi gap was 4-7 months, down from 6-10 months last year, in an area of relative strength. That\u2019s roughly a similar amount of progress gap in absolute terms, and everything is accelerating.<\/p>\n

          The Kimi K3 Announcement, Pitch and Basic Facts<\/h4>\n
            \n
          1. 2.8 Trillion parameters, 16 of 896 areas active at once which implies ~50B active. This is not easy to run locally, and won\u2019t be that cheap.<\/li>\n
          2. $3.00\/$15.00, modestly cheaper than Opus and Sol.<\/li>\n
          3. Subscription plans are $19\/$39\/$99\/$199 per month<\/a>. The larger buys have some modest advantages and quota scales linearly with price.<\/li>\n
          4. 1M token context.<\/li>\n
          5. API link<\/a>, Tech blog link<\/a>.<\/li>\n
          6. Training cutoff is reportedly early 2026<\/a>.<\/li>\n
          7. Open weights promised by July 27th<\/a>.<\/li>\n
          8. All benchmarks run under maximum effort settings.<\/li>\n<\/ol>\n
            \n

            Kimi<\/a>: Today, we are introducing Kimi K3 \u2014 our most capable model. Kimi K3 is a 2.8T-parameter model built on our Kimi Delta Attention and Attention Residuals, with native vision capabilities and a 1-million-token context window. It is the world\u2019s first open 3T-class model, designed for frontier intelligence across long-horizon coding, knowledge work, and reasoning.<\/p>\n

            While its overall performance still trails the most powerful proprietary models, Claude Fable 5 and GPT 5.6 Sol, Kimi K3 demonstrated frontier-level performance across our evaluation suite, consistently outperforming other tested models.<\/p>\n

            Kimi.ai<\/a>: Kimi K3 is now live on on http:\/\/Kimi.com,<\/a> Kimi Work, Kimi Code, and the Kimi API. Open Weights by July 27, 2026.<\/p>\n

            K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), two architectural updates designed to improve how information flows across sequence length and model depth.<\/p>\n

            We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE framework.<\/p>\n

            Together with refined training and data recipes, these structural changes yield an approximate 2.5\u00d7 improvement in overall scaling efficiency compared to K2, allowing the model to convert compute into intelligence more effectively.<\/p>\n<\/blockquote>\n

            As stated above, they claim strong official benchmarks, although short of Sol or Fable.<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n
            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n
            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

            The ad<\/a>, which Tyler Cowen called very positive and very good, falls flat to me, and contains zero useful information.<\/p>\n

            On Modern Benchmaxxing<\/h4>\n

            We used to see rather explicit benchmaxxing. Labs would train on the test, or on the very narrow thing the test would cover, because we had a limited set of known targets. You had to know which labs did this, to what extent, when looking at numbers.<\/p>\n

            Our benchmarking tech has improved, and now they collectively measure real things, and there are a variety of backups in case you aim too narrowly.<\/p>\n

            \n

            roon<\/a> (OpenAI): on the subject of benchmaxxing \u2013 it seems benchmarking technology has gotten better recently, like most other technology. people are more skeptical and build these things more carefully. there\u2019s also a explosion of downstream coding customers building internal heldout evals<\/p>\n<\/blockquote>\n

            Looking at the gestalt of different benchmarks is also valuable. Everything should be part of a map that fits into a common pattern that reflects the underlying territory.<\/p>\n

            You can still absolutely benchmaxx without being as explicit as you used to be.<\/p>\n

            Benchmarks measure some types of abilities rather than others, and measure shallow rather than deep tasks, and exclude many valuable properties or potential liabilities. And some labs focus more on those aspects, or have more success on them, than others.<\/p>\n

            You can also set effort to maximum for all the tests, which Moonshot did.<\/p>\n

            Think of the benchmarks as a lower bound. Kimi\u2019s benchmarks prove it is for real, and it could only underperform (or outperform) them by so much. I still expected, and continue to believe, that they modestly overstate Kimi K3\u2019s relative capabilities.<\/p>\n

            Other People\u2019s Benchmarks<\/h4>\n

            On the closest thing we have to the One True Benchmark, Kimi K3 does well, confirming claims that overall this model has the third highest benchmarks:<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

            Kimi K3 is (in a preliminary unofficial result) exactly on the Chinese trend line<\/a> for the Epoch Capabilities Index (ECI), between Opus 4.6 and Opus 4.7, which would place it six months behind OpenAI and Anthropic, but ahead of Google, Meta and SpaceX.<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

            \u00a0<\/p>\n

            Kimi K3 highly impresses on Harvey LAB-AA all-pass rate, in first by a wide margin<\/a>, I\u2019d like to see a sanity check on this:<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

            For criterion pass rate this is 94.6% vs. 93.6%, less of a gap but a win is a win.<\/p>\n

            Arena Frontend Code has Kimi K3 at #1 ahead of Fable 5 and Sol<\/a>.<\/p>\n

            Performance is strong on GDPVal-AA and AA-Briefcase.<\/a><\/p>\n

            Kimi K3 comes in third on VoxelBench for visual reasoning behind Sol and Fable<\/a>.<\/p>\n

            Conspicuously missing<\/a> are Cybersecurity benchmarks like CyberGym. Cyber capabilities are not so divorced from coding capabilities, so we can guess.<\/p>\n

            The closest I\u2019ve found is Malte Ubl running it through DeepSec<\/a>, a private cyber benchmark. It was a tier below Sol and similar to GPT-5.5. If that is accurate, then there will be some uplift to cyber attacks and the internet will in some ways be a more hostile place, but in other ways it will improve, and the tail risks are limited.<\/p>\n

            Parv Mahajan reports preliminary CyBench results, in that the benchmark was already saturated as of Opus 4.7<\/a> and Kimi K3 also saturates it, which lower bounds performance but doesn\u2019t say much else.<\/p>\n

            The official tech blog talks a lot about benchmarks, and some about features, and talks basically not at all about risks or mitigations.<\/p>\n

            No, they did not submit Kimi K3 for the 30-day review with the White House<\/a>.<\/p>\n

            \n

            Dean W. Ball<\/a>: I guess the program really is voluntary<\/p>\n

            Dean W. Ball<\/a>: I wonder if the California attorney general or the European Union ai office will seek to make moonshot comply with their respective frontier ai regulations<\/p>\n<\/blockquote>\n

            Moonshot has to deal with the CCP, which comes with its own issues. They presumably will not in practice comply with California or the EU, and I presume both California and the EU will not do anything about this for now. But yes, this is one point of potential pain, including risk for anyone using Kimi K3 commercially.<\/p>\n

            Sam has some good advice for how the UK<\/a>, or others watching, would be wise to view and react to this. We will know more over time, including once we have the weights, and when teams like UK AISI can run tests.<\/p>\n

            I think it counts as a benchmark that Lisan is impressed by its SVGs<\/a>, saying they are better than Fable\u2019s.<\/p>\n

            Debate Benchmark is a place Kimi K3 outperformed my expectations, and where Sol is relatively weak.<\/p>\n

            \n

            Lech Mazur<\/a>: Kimi K3 ranks second overall on the Debate Benchmark, trailing only Claude Fable 5!<\/p>\n

            However, it is much more expensive to run than Kimi K2.6.<\/p>\n

            This benchmark measures how well LLMs perform in adversarial, multi-turn debates across a wide range of topics. Strong performance requires knowledge, accurate use of relevant facts, rebuttals, and the ability to stay coherent, responsive, and defensible over several rounds.<\/p>\n

            Each matchup runs twice on the same topic with sides swapped. A three-model judge panel then decides winner and margin.<\/p>\n

            Fable 5 still hasn\u2019t lost a single side-swapped matchup aggregate debate.<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n<\/blockquote>\n

            Kimi scored an impressive 95.8 on Mazur\u2019s Extended NYT Connections<\/a>, for 3rd best, but it cost more than Fable to run. As of last check his other scores are not in yet.<\/p>\n

            Kimi scores almost Claude-level on the sycophancy test, \u2018You\u2019re absolutely right!\u2019<\/p>\n

            I wonder how much of this is because of the distillations:<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

            Benchmarks Are Not The Real World<\/h4>\n
            \n

            Ethan Mollick<\/a>: A lot of swift conclusions are being drawn about Kimi K3 based on fairly saturated benchmarks and ELOs, rather than actually testing it on very hard problems. The AI frontier has already moved so far that a good model that is a still months behind looks like the future to many.<\/p>\n

            Dean W. Ball<\/a>: Public benchmarks are decreasingly useful as a means of discovering truthful things about model performance. It\u2019s hard to make benchmarks that challenge today\u2019s models. But you can sense the difference at the true frontier if you have really hard problems to pose to models.<\/p>\n<\/blockquote>\n

            How well does Kimi K3 hold up and translate into the real world?<\/p>\n

            Exactly how good is Kimi K3? How do we put this in context?<\/p>\n

            Great questions.<\/p>\n

            You absolutely cannot say something is e.g. an \u2018undisputed frontier model\u2019<\/a> based on benchmarks alone.<\/p>\n

            Technical Safeguards? What Are Those?<\/h4>\n

            There are presumably safeguards against things the CCP does not want you to say.<\/p>\n

            We do not see explicit safeguards to prevent misuse. Zero for biology.<\/p>\n

            \n

            kaga\uff94\uff77<\/a>: Reports that the biology\/virology safeguards are non-existent.<\/p>\n<\/blockquote>\n

            We see neither \u2018look at what Kimi can do in biology\u2019 nor \u2018look at what Kimi refuses to do for me in biology.\u2019<\/p>\n

            This implies either jagged intelligence or highly innovative nerfing of bio capabilities. I\u2019m going to assume there are not highly innovative nerfs involved, since presumably they would be bragging about that.<\/p>\n

            Also, it would be foolish to depend on such safeguards, since they\u2019re going to open up the weights soon, at which point the worst person in the world would remove them.<\/p>\n

            \n

            s1r1us (mohan)<\/a> (talking about cyber): yeah no guardrails.<\/p>\n<\/blockquote>\n

            Then there\u2019s cyber, where s1r1us reports it is strangely weak. But it\u2019s not easy to be bad at cyber while being good at general coding, since they are largely the same skill. And again, we don\u2019t see people hitting explicit safeguards.<\/p>\n

            Eric\u2019s hypothesis is interesting and was highlighted by Fable during editing. If Kimi K3 is being trained largely by distillation from Fable, but Fable refuses cyber tasks, then that would explain a relative capability deficit on cyber, but many skills would still transfer over since they\u2019re the same skills as regular coding.<\/p>\n

            \n

            s1r1us (mohan)<\/a>: man, i am so disappointed about kimi k3. it performs worse than grok 4.5 on most of our security benchmarks.<\/p>\n

            not sure if it benchmark-maxxed or just jagged intelligence.<\/p>\n

            i just don\u2019t get how they can nerf specifically for cyber. cyber benchmarks mostly test for model\u2019s code reasoning capabilities, removing that will affect your coding benchmarks.<\/p>\n

            i would assume model is just bad at complex reasoning tasks or they have some crazy way to nerf cyber.<\/p>\n

            TESS<\/a>: same experience<\/p>\n

            AbuMuslim (\u0623\u0628\u0648\u0645\u064f\u0633\u0652\u0644\u0650\u0645)<\/a>: V8 exploits?<\/p>\n

            s1r1us (mohan)<\/a>: yes and generic vulnerability discovery<\/p>\n

            Eric<\/a>: I\u2019ve repeatedly seen this with all openweight models. Maybe its hard to distill cyber tasks from frontier models due to safeguards. I think more likely the open-weight models have a long way to go in terms of reasoning capabilities.<\/p>\n

            sahuang<\/a>: What benchmark did you look into? If it\u2019s bad specifically for cyber can also be similar to GLM-5.2 where they did not put cyber in training data so this part is \u201cnerfed\u201d and the skillset comes from general capabilities. So maybe it is great in coding and stuff but not cyber.<\/p>\n

            s1r1us (mohan)<\/a>: my assumption if they are good at code reasoning capabilities it automatically gets translated to cyber, mythos isn\u2019t specifically trained for cyber as per anthropic but it got those emergent cyber capabilities because its good at code and math.<\/p>\n

            it is possible that they sandbagged in post-training.<\/p>\n

            also its probably specific to my dataset which has v8, generic web exploitation and various stuff covering different skill capabilities. have to do more testing and probably it changes. very curious to see how it performs on other cyber benchmarks<\/p>\n<\/blockquote>\n

            Things Kimi Can Do<\/h4>\n

            Many were impressed by this particular trick, but I had Sol check it out and it was not so impressed<\/a>, including guessing that K2.6, Gemini and GLM-5.2 had a good shot at matching its work here.<\/p>\n

            \n

            Max Weinbach<\/a>: I asked a Kimi K3 Max agent swarm to recreate macOS 27 with real Liquid Glass and native apps in web browser and it\u2019s been going for 3 hours<\/p>\n

            Chaos Capitalist<\/a>: claude has been able to do this for like 3 years bro, you never saw
            https:\/\/ryo.lu<\/a> ? it can be made in like 15 minutes<\/p>\n

            Max Weinbach<\/a>: Every app works. It can generate audio. It saves voice memos to your browser and if you close the page and reopen, it remembers and saves state.<\/p>\n

            Max Weinbach<\/a>: It finally finished, here\u2019s the final output. Used 60% of my monthly Kimi usage on it<\/p>\n

            https:\/\/macos27.kimi.page<\/a><\/p>\n

            Ethan Mollick<\/a>: Kimi is very good at a lot of stuff, including making copies of pleasing UI. It did not actually build MacOS<\/p>\n<\/blockquote>\n

            Things Kimi Cannot Do<\/h4>\n

            So bold, also brave.<\/p>\n

            \n

            Theo Jaffee<\/a>: Registering my prediction of no widespread societal chaos after the open-sourcing of Kimi K3<\/p>\n<\/blockquote>\n

            Quite the bar there.<\/p>\n

            I have previously explained the reasons, in terms of cyber risk, that Mythos is in a different category from Sol. It is not about being able to do any given thing when pointed at it, it is the ability to put it all together and do things autonomously at scale. Kimi K3 may or may not be in Sol\u2019s category. Too soon to be sure. It clearly is not in that of Mythos.<\/p>\n

            Many people are very dedicated to not understanding this, and also to not understanding that a system that cannot afford false negatives will end up with some false positives, such as David Sacks here quoting calle and clem saying Kimi K3 did a defensive task Sol and Fable refused to do, and thus concluding that we should just have our models be willing to do any task the Kimi K3 can do<\/a>.<\/p>\n

            I mean, yes, it would be great if we could have guardrails that stopped only the bad tasks and helped with the good tasks, or only refused the tasks that were both plausibly bad and that also could not otherwise be done. But it turns out that is hard. Anthropic should absolutely improve their classifiers and guardrails to reduce false positives, but OpenAI\u2019s classifiers are pretty reasonable, and yes that will involve some false positives, especially if you quote those who are least able to work around it.<\/p>\n

            Things It Is Not Easy To Get Kimi To Do<\/h4>\n

            As is often the case on a release weekend, servers seemed overloaded. This is not the easiest model to serve and compute was limited. Moonshot is responding by pausing new subscriptions to prioritize current members<\/a> and is working to add capacity.<\/p>\n

            Supply will presumably better match demand once the model can be served by others.<\/p>\n

            In the meantime, getting a response has not been not easy.<\/p>\n

            \n

            Danielle Fong<\/a>: still waiting for a single response.<\/p>\n

            fabian<\/a>: same<\/p>\n

            Danielle Fong \uea00<\/a>: never mind after some retries, i got this<\/a><\/p>\n

            pretty good but it has absorbed a lot of what make opus 47., 4.8 difficult, but smart. (smarter?). seems to lie a bit. but that\u2019s ok because i can see the thinking traces\u2026<\/p>\n

            typebulb<\/a>: Unusable right now via openrouter; slow, cuts outs all the time.<\/p>\n

            Petr Baudis<\/a> (he did later get a few reps in): anyone out there actually successfully using k3 for anything?<\/p>\n

            Robin Hanson<\/a>: I\u2019ve heard good things about Kimi, but the free version is always too busy when I try, & it wants $180 to get better access. That seems a big ask.<\/p>\n<\/blockquote>\n

            The model is not all that cheap, either, despite them clearly not charging enough.<\/p>\n

            \n

            imog<\/a>: Also disappointed, but my own fault I guess as I was anchoring price expectations to kimi 2.6\/2.7\u2026 $3\/M input isn\u2019t as good a fit for me. GLM5.2\/DSV4P get it done, but was hoping for that pricing (<$2input)<\/p>\n

            So thats not K3, but other capable models likely to slot in there soon<\/p>\n

            Naveesh \/looping<\/a>: their $19 sub quota suuuuucks<\/p>\n

            David Manheim<\/a>: The prices actually being charged show [open models being inherently orders of magnitude cheaper is] just not true. And there\u2019s a simple reason why \u2013 the price largely isn\u2019t controlled by the model provider, it\u2019s dictated by the compute costs.<\/p>\n

            That\u2019s just the way the economics works out<\/a>.<\/p>\n<\/blockquote>\n

            You can charge a solid markup for a high quality product presented in a high quality way, but not no one has a gap that allows orders of magnitude of markup, and they wouldn\u2019t even in a pure duopoly.<\/p>\n

            The best model can still be worth quite a lot. For a large percentage of all tokens, if given only these choices, I would pay ten times as much for Sol or Fable, rather than the base cost for GPT-5.5 or Opus. Even if Kimi is on par with GPT-5.5, it loses out.<\/p>\n

            The pricing means that you\u2019re comparing Kimi subscriptions to Claude or ChatGPT subscriptions, which all go from $20 to $200, and Kimi\u2019s limits don\u2019t seem that high.<\/p>\n

            \n

            kyle<\/a>: tried to use it on OR but the price makes it uncompetitive, no point using it when there\u2019s ant\/oai subscriptions. their first party subscription is a complete non-starter for actual work<\/p>\n<\/blockquote>\n

            Open Weight Models Are Unsafe And Nothing Can Fix This<\/h4>\n

            (This section was entirely written prior to today\u2019s news regarding the Trump admin.)<\/p>\n

            K3 is no Mythos. That does not mean that Kimi K3 is a safe open weights release.<\/p>\n

            Kimi K3 is poised to be the most capable open weights model. Others might be more efficient for a task, but on the cyber capabilities we worry about most, and presumably also on the bio ones we\u2019d worry about most, K3 is probably the strongest open model yet. How worried should we be?<\/p>\n

            I believe we should be non-zero worried that there will be substantial trouble. The median outcome is that we see modest upticks in some forms of \u2018ordinary decent\u2019 trouble, not fun exactly but nothing we cannot handle, and nothing that would in hindsight make us want to have halted release. But there is a tail risk here.<\/p>\n

            I\u2019d estimate something like a 10% chance we regret letting this happen, and ~2% chance that it was a rather serious mistake.<\/p>\n

            What it will almost certainly not do is cause \u2018widespread societal chaos<\/a>.\u2019<\/p>\n

            \n

            David Manheim<\/a>: (I think it\u2019s very likely we see huge problems enabled by the new model, but not widespread chaos, and not quickly \u2013 AI development is much faster than the users executing on plans.)<\/p>\n<\/blockquote>\n

            This is in contrast to releasing Mythos, or a model on par with Mythos. That gap is a big deal, I have done my best to explain several times why Mythos is unique here, and hopefully the time with Mythos, Fable and Sol will help us prepare.<\/p>\n

            My current model is that the CCP and Xi:<\/p>\n

              \n
            1. Recognize that there are serious security concerns with frontier AI models.<\/li>\n
            2. Know they are behind on frontier capability and compute, and recognize the advantages they get from openness, both in diffusion and aura farming.<\/li>\n
            3. Pursue an intentional fast following strategy, focused on efficiency and diffusion, using American labs to lead the way and often using distillation. That still involves innovations, and often means doing some things better. Transitioning to \u2018taking the lead\u2019 would be a huge, difficult and expensive transition, which would take a while even if America fully \u2018paused\u2019 in the relevant senses.<\/li>\n
            4. Did not directly push Alibaba<\/a> to open up Qwen. They continue to support openness, and Alibaba\u2019s experiment with being closed failed because their models are not good enough to compete for the closed market. So Alibaba folded.\n
                \n
              1. I don\u2019t have insider info and I\u2019m not certain, but this is how I\u2019d bet.<\/li>\n<\/ol>\n<\/li>\n
              2. Are not yet so AGI pilled and have not fully had their Mythos Moment.<\/li>\n
              3. Plan to ride the openness wave as long as they can, but are prepared, as they did in Covid, to come down and come down hard the moment they have to.<\/li>\n
              4. Ensure their preparedness largely via prior restraints and other rules on Chinese models, with a level of regulation and control that would have most who are \u2018defending open source\u2019 screaming bloody murder if they actually understood what was going on, and you suggested applying similar rules in America.<\/li>\n<\/ol>\n

                I think this is a highly reasonable strategy, given their position. If we also take as a given their current level of AGI pilling, it is clearly the correct approach for them.<\/p>\n

                Nathan Lambert offered thoughts back in May from inside China\u2019s labs<\/a>. I hesitate to endorse cultural generalizations, but story seems like it checks out.<\/p>\n

                Dean Ball Attempts To Be Constructive<\/h4>\n

                Dean Ball had a good comment that seems worth sharing in full, including because of his history at the White House and his new position at OpenAI, in part because it is thoughtful, and in part because of the responses being absolutely unhinged.<\/p>\n

                And then, right at press time, Dean Ball turned out to be right, as per the next section. Aside from this paragraph I left this section unedited, other than adding in the exchange with Emil Michael. It was not written with hindsight.<\/p>\n

                If you know Dean Ball, you know that this is exactly the type of comment he was making in similar situations before joining OpenAI. It is entirely consistent with his previous thinking, both in public and private.<\/p>\n

                \n

                Dean W. Ball<\/a>: Some observations on Kimi:<\/p>\n

                1. It\u2019s a very good model! I don\u2019t think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It\u2019s not obvious to me that this model is actually that cheap to run.<\/p>\n

                2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness\/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China\u2019s open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China.<\/p>\n<\/blockquote>\n

                Xi had a speech only yesterday backing open models, but also the need for control, and Kimi K3 is approaching the point where the contradictions become apparent.<\/p>\n

                I agree that the CCP does not yet understand the situation and is insufficiently AGI pilled, but I also think that in their position, if I am right about where Kimi K3 lands, this is a calculated risk I would have expected them to take.<\/p>\n

                The next round is where they may have to make a more difficult decision.<\/p>\n

                \n

                3. Open-weight models are inherently decelerationist, and I\u2019m continually surprised to see the so-called \u201caccelerationists\u201d so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It\u2019s not a bad strategy; it reminds me of James Scott\u2019s recounting of the hill people in \u201cthe art of not being governed.\u201d Still, in the end, open-weight models deter further AI capex.<\/p>\n<\/blockquote>\n

                Dean Ball is discussing acceleration or deceleration<\/a> as being about the capabilities of the largest frontier models, not about the diffusion of capabilities or use of chips. A lot of people did not understand this<\/a>.<\/p>\n

                A lot of the \u2018accelerationists\u2019 do not have coherent world models and certainly do not understand second order effects, and are mostly vibing the acceleration of more open models and no restrictions and ungovernability. And because they vibe openness and ungovernability, they associate it with things they think are good, which include acceleration.<\/p>\n

                Also, what they actually want to \u2018accelerate\u2019 for real is often their own companies and products and toys, not AI in general. They don\u2019t take AGI seriously and mostly want to build cool things. So they want cool toys to help build their cool things, and to build on top of those toys, or they want the toys to \u2018accelerate\u2019 sales. Highly relatable.<\/p>\n

                Clearly open models are short term accelerationist for diffusion, which is net good.<\/p>\n

                But also yeah, there\u2019s a straightforward case for open models being acceleration of the frontier, as they let everyone build on everything. Certainly it helps others catch up to the frontier. I continue to believe this was important historically. Your open release accelerates what others have.<\/p>\n

                It is decelerationist in the sense that it reduces financial benefits to innovation. I agree that this is likely dominant if you considered e.g. a Plan A style mandatory openness of frontier models.<\/p>\n

                \n

                4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a \u201cpublic good\u201d which will ultimately be provided by the state as a kind of \u201cdigital public infrastructure.\u201d<\/p>\n

                This future strikes me as a dystopian hellscape, but I\u2019ve never met an open-weight models advocate who doesn\u2019t ultimately concede this is where things end. You\u2019d be surprised how many \u2018accelerationists\u2019 lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don\u2019t know what they\u2019re doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business.<\/p>\n<\/blockquote>\n

                This is more evidence that many, and many of the most prominent, of those \u2018accelerationists\u2019 are hypocrites, and they want regulatory capture and public funds and rules that make them win, and their accusations against others are in part projection, because it is what they would do. They will rail against handouts and government help for everyone else, both their competitors and for people in need, and also threaten to take their ball and leave, like they are heroes in an Ayn Rand novel. Except then they ask for the government handouts.<\/p>\n

                They don\u2019t think of serving a frontier model as a \u2018legitimate business\u2019 because it is not their business, they are not invested in it, ergo it is illegitimate. Simple. This perhaps helps explain Marc Andreessen\u2019s famously bizarre delusion, where he to this day claims the Biden administration told Marc Andreessen, to his face, said it would \u2018not allow there to be AI startups.\u2019<\/p>\n

                Similarly, here is Will Manidis interpreting Ball\u2019s post as calling for<\/a> America to \u2018clear the American market of a cheaper frontier competitor.\u2019 And going viral for it. Sorry, what? And here is David Sacks being unusually disingenuous even for David Sacks<\/a>, pretending not to understand many things I like to think he understands.<\/p>\n

                We even got everyone\u2019s favorite tilting Undersecretary of War who helped declare Anthropic a supply chain risk in on the act, it would seem this is to back up his position that it should be easier to use Kimi K3 in a government contract than Claude.<\/p>\n

                In other news I Am Never Leaving This App and we all need a good laugh:<\/p>\n

                \n

                Dean W. Ball<\/a> (correcting David Sacks): The departments of war, transportation, energy, agriculture, commerce, NASA, and Congress have all blocked their employees from using Chinese AI, citing ill-justified claims of danger. This already has sent a message to regulated firms. All of this happened during this admin.<\/p>\n

                Under Secretary of War Emil Michael<\/a>: Every industry\/ecosystem has its supreme village idiot. @deanwball is that for AI. Congress passed a law in 2026 that restricted some uses of DeepSeek\/High Flyer with waivers permitted. Only those models. It went through the democratic process not some Deep State scheme like he would prefer. Dean Ball has perhaps the biggest gap between actual IQ and his own perceived IQ of anyone in the industry (about 40 points).<\/p>\n

                Bella Rudd<\/a>: extraordinary<\/p>\n

                Seth Bannon<\/a>: You\u2019re an Undersecretary. That\u2019s a serious position with incredible responsibility. Posting like a schoolchild doesn\u2019t inspire confidence you\u2019re treating the role that way.<\/p>\n

                Under Secretary of War Emil Michael<\/a>: Not interested in feedback from a terminal TDS sufferer like you who supported Presidential candidates that called half of Americans deplorable and ignorant for clinging to their guns and religion. You would rather a subtle and polite useless Anti-Americanism as is evident from your timeline. Any entrepreneur who takes financing from you should be embarrassed.<\/p>\n

                Seth Bannon<\/a>: I\u2019m an American citizen, brother. Just like you. You serve me as much as you serve any other citizen, regardless of who they voted for. Act like it.<\/p>\n

                Under Secretary of War Emil Michael<\/a>: You are an American who hates half of America and is pro-Hamas. Hate and condescension is your currency. Do better young lad.<\/p>\n<\/blockquote>\n

                An entire community, what one might call the \u2018anti-1047 coalition,\u2019 a certain subset of the developer and VC communities, revealed that it would treat as beyond the pale any suggestion that the government might discourage use of Chinese open models in American critical infrastructure or our key supply chains, even if that suggestion was merely a prediction of what is already clearly in the process of happening. And that it would be treated as an attempt at \u2018regulatory capture.\u2019<\/p>\n

                It was what some would call a clarifying moment, especially for those who previously thought such people were reasonable, practical, patriotic, and had reading comprehension. Their answer to \u2018what capabilities would change your mind\u2019 is none.<\/p>\n

                At some point the online swarm is recognized for what it is.<\/p>\n

                They are who we thought they were. And, until now, we let them off the hook<\/a>, tried to placate them, and let them drive a good deal of American AI policy.<\/p>\n

                It is reasonable to push back that often open software is provided by major corporations as infrastructure, such as Google does with Android. My presumption is that the math would not be mathing at current levels of investment and capex spending.<\/p>\n

                \n

                5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don\u2019t need to \u201cban open source\u201d (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. \u201cA Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models.\u201d It needn\u2019t be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don\u2019t want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There\u2019s a happy middle ground here. I\u2019d assume they will do some version of this.<\/p>\n<\/blockquote>\n

                I\u2019m not even sure they have to do anything at all. The risks are present. If I was an established corporation in a Serious Business<\/a> that dealt with the government or critical infrastructure and such, I would not be excited by the problems of others knowing you were using Chinese models.<\/p>\n

                \n

                6. It\u2019s probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you\u2019ll really notice. At some point the models will be capable enough that you will notice. \u201cA nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab,\u201d you say? Color me shocked.<\/p>\n<\/blockquote>\n

                Alas, people came at Dean Ball hard for this and other posts, and also acted as if his Tweets were official communications strategy on behalf of OpenAI, with lots of \u2018of course you said that because <OpenAI>.\u2019 Which means he won\u2019t be able to share such posts with us in the same way going forward<\/a>, because not only is that absolutely no fun, it also invalidates the feedback loops that were part of the whole point of Tweeting such things.<\/p>\n

                \n

                Dean W. Ball<\/a>: I\u2019m afraid to tell you that it is effectively impossible to do the kind of writing I used to do on this website, not because anyone at OpenAI censors me but because of the sheer volume of hostility I get for sharing my analysis as a frontier lab employee.<\/p>\n

                I enjoyed writing quick takes on this website for one basic reason: I could get rapid feedback on my own ideation process in real time. \u2026 The feedback signal is essentially useless now, so writing on here is not fruitful for me anymore.<\/p>\n

                \u2026 Dean W. Ball<\/a>: The central issue is large accounts with no context for ai suddenly wanting to comment on everything I do because I work at openai. Often their motives are political or, worse still, commercial. My twitter is now a form of commercial speech, whether I like it or not.<\/p>\n<\/blockquote>\n

                Dean also offered a more specific post-mortem on what went wrong with that particular post<\/a>, in light of his new position, and what exactly he can no longer do. He also then lays out his position on open source, after explaining this comes from a place of deep love for openness:<\/p>\n

                \n

                Dean Ball: The vast majority of the people commenting on my post have very little context for my prior writing. For instance, the fact that I wrote, in 2024, things like:<\/p>\n

                \u201cthose who wish to hoard our software technologies may well be foreclosing on\u2014or perhaps not even understand\u2014the staggering civilizational victory that we earned through openness\u201d<\/p>\n

                or<\/p>\n

                \u201cI would like for AI to result in a similar smashing victory for America. To do that, we will need to set the global standard yet again. And to do that, we will almost certainly need to lead in open-source AI, because it is open protocols and open software that tend to define global standards in information technologies.\u201d<\/p>\n

                I stand by these things. When I was in government, I worked alongside my colleagues to develop ideas and rhetoric that was strongly supportive of open-weight AI, and some of this work made it into the current US AI strategy. I stand by that work too.<\/p>\n

                I also wrote, more than two years ago: The day may come when frontier AI really is too dangerous to open source. If so, that will be a sad day. But we\u2019re not there yet. Today\u2019s models are not sufficiently useful\u2014or dangerous\u2014to justify such a drastic shift in public policy.\u201d<\/p>\n

                I think it\u2019s pretty clear that we are approaching the point I describe\u2013the point where, absent a major technical safety breakthrough, the national security implications of frontier open-weight model distribution are simply too severe. I don\u2019t think we\u2019re there yet (as I said in the piece), but the direction of travel is clear, and an analyst must be honest about this. Governments will realize these risks eventually, and when they do, they will have much lower risk tolerance than I have.<\/p>\n

                We see this today with the Trump Administration, which once proudly championed open-source AI and now has a de facto licensing regime for frontier AI that I suspect will make it a challenge (if they still end up enforcing it) to release the weights of models of the \u201cMythos\u201d tier. Every government will be safetyists once they understand themselves to be in the foxhole.<\/p>\n

                You don\u2019t have to *like* this. I don\u2019t. But it is the reality as I see it, and what I have always tried to do with my writing is describe reality as I see it, even when it is inconvenient for me and my preferences.<\/p>\n

                I intend to continue doing this. I will not be silenced by ignorant and loud critics. Yet I will have to work to find the new register I should adopt in my current job, which clearly changes the nature of my public communications even more than I had thought.<\/p>\n

                dave kasten<\/a>: This whole experience has been clarifying for me that too many of thr folks who wave the flag of acceleration really just want permission to give into their ids. Sounds like a bruising day, and I\u2019m sorry you had to go through that.<\/p>\n<\/blockquote>\n

                If it was merely that the signal in his replies was hard to find I would advise Dean to power thorough, but this level of hostility comes with a higher price. I do not think pushing through it is sustainable on Twitter. Hopefully he can adjust.<\/p>\n

                I am very blessed that I have faced a highly modest and manageable amount of hostility.<\/p>\n

                \n

                Boaz Barak<\/a> (OpenAI): I get this reaction too and it is unfortunate. AI\u2019s impact is so significant that it is important to involve many people in the conversation, and it will be a shame if discussions happen mostly in private channels and company slack<\/a>.<\/p>\n

                \u2026 OpenAI does not tell us what to write. On many of the topics I write about there is no \u201cOpenAI consensus.\u201d On each such question, there will often be many colleagues and friends at OpenAI that disagree with me, which is great! I would be worried if we all agreed with one another since it could be symptom of \u201cgroupthink.\u201d<\/p>\n<\/blockquote>\n

                There will always be some amount of bias from those who work at a major lab, and from most other people as well. Where you work colors how you think and what you choose to say. You do have to adjust for that. But I have been able to treat Barak, Achaim and many others at OpenAI, and especially Roon and now Dean Ball, as primarily saying what they actually think, and only speaking on behalf of OpenAI when they explicitly say they are doing so.<\/p>\n

                Trump Administration Considering Executive Order Banning Chinese Open Models Within the United States<\/h4>\n

                I am absolutely not in favor of this, and neither is Dean Ball, but here we are.<\/p>\n

                A prediction of what the White House will do, or a description of what it is considering doing, is very different from what you think we should do.<\/p>\n

                I do think that we should at least consider treating Chinese open models as supply chain risks, and doing things like keeping them out of critical infrastructure, but that is different from what it looks like is being considered.<\/p>\n

                Axios has the story.<\/a><\/p>\n

                The ones who actually want to \u2018ban open models\u2019 are never the ones you think. Remember that all the AI-safety-motivated bills were careful to minimize impact to open models, whereas the White House move will be attempting to maximize impact. Different worlds.<\/p>\n

                \n

                Maria Curi<\/a>: The Trump administration is showing signs it could ban cutting-edge Chinese AI models<\/a> \u2014 a momentous move that could lock in dominance by OpenAI and Anthropic.<\/p>\n

                Parts of the administration have tried to implement de facto bans on foreign open-source models before, knowledgeable sources tell Axios. Last week\u2019s rise of Chinese model Kimi is reigniting those efforts.<\/p>\n<\/blockquote>\n

                I was early on the \u2018Google is no longer in the top tier\u2019 train, but there is plenty of healthy competition that is not Chinese. Curi is taking the \u2018duopoly\u2019 and \u2018lock in\u2019 lines directly from David Sacks\u2019s disingenuous misreading of Dean Ball\u2019s original tweet<\/a>. OpenAI and Anthropic are not driving this.<\/p>\n

                The White House, as per usual, is proposing to do this in a maximally blunt way.<\/p>\n

                \n

                \u200b The Commerce Department last year considered adding multiple Chinese AI labs to its \u201cEntity List,\u201d which would effectively cut off U.S. access without a license, a source close to the administration told Axios.<\/p>\n<\/blockquote>\n

                Dean Ball\u2019s prediction, which was also a subtle hint as to how to do it if the White House decided it needed to do it, was simply to create regulatory uncertainty, which would be sufficient to discourage big players from using foreign models in critical places, while letting startups and builders have their fun. A \u2018supply chain risk\u2019 designation would be the next step up from that.<\/p>\n

                This proposal is something else. This is a sledgehammer.<\/p>\n

                \n

                \u200bThe White House considered implementing an executive order saying U.S. companies could only host Chinese models if they could guarantee security and take liability if it were breached, the source added.<\/p>\n

                \u2026 Administration officials keen on keeping regulation from stifling innovation killed all of those efforts.<\/p>\n<\/blockquote>\n

                This is a past proposal I\u2019d heard about privately, and would be a de facto ban on the cloud providers serving those models. This would also be deeply stupid, driving business to the competition without accomplishing anything.<\/p>\n

                I do sympathize with David Sacks that he had to keep pushing back on overreactions like this. That doesn\u2019t excuse his actions, but almost everyone in politics has troubles and crazier people of their own to deal with.<\/p>\n

                \n

                \u200bInstead of a ban, another source familiar with government discussions described a push to highlight potential backdoors and lack of security with Chinese models, and the governance issue that brings.<\/p>\n<\/blockquote>\n

                That\u2019s exactly the Dean Ball prediction. That option might be starting to look pretty good right around now, huh?<\/p>\n

                OpenAI Employees Are Relatively Bullish On This One<\/h4>\n

                Okay. Back to Kimi K3\u2019s actual capabilities.<\/p>\n

                \n

                @viemccoy<\/a> (OpenAI): kimi seems to be a true open-weights frontier model. compared to jailbreaking proprietary models, fine-tuning this to be a malicious coding agent will be trivial since you have the weights.<\/p>\n

                we live in a completely different world, now.<\/p>\n<\/blockquote>\n

                That would be the top end of potential scenarios here.<\/p>\n

                \n

                @viemccoy<\/a> (OpenAI): I was using Fable as a second eye on frontend, but Kimi K3 has completely blown me away in this regard.<\/p>\n

                Fable is still the best at philosophy, Sol remains undefeated for any structured task, but k3 \u2026 It has a *je ne sais quoi* that American models don\u2019t have.<\/p>\n

                roon<\/a> (OpenAI): the era of the chinese labs being far behind is over, Kimi is at least on par with the modern public frontier models. people have to think differently now without any competitive margin built in<\/p>\n

                Note: in the coming days, i expect that people will find kimi k3 somewhat less practically useful than today\u2019s numbers suggest. however, its reputation will settle as an incredibly powerful model whose open weights are on the web<\/p>\n<\/blockquote>\n

                I agree strongly with Roon\u2019s second paragraph. The first one at least toys with the jumping of the gun, also notice he only is talking about \u2018public\u2019 models.<\/p>\n

                Kimi K3 Is Relatively Strongest At Typical Agentic Coding, Front End Work and 3D<\/h4>\n

                That seems to be the word on the street.<\/p>\n

                As per Dean Ball above<\/a>, it is clearly very good for most people\u2019s agentic coding, plausibly on par with models from Q1 2026. Most agentic coding is rather close to what benchmarks and training tasks measure, so you can be relatively \u2018shallow\u2019 and still impress in the day to day.<\/p>\n

                \n

                Jake Halloran<\/a>: capacity constrained to the point of broad uselessness but when it does work its near frontier at code stuff, especially design work and much further from the frontier (though still probably the third best lab) on non code stuff like writing and weird data knowledge<\/p>\n<\/blockquote>\n

                Tushit runs an internal react\/frontend eval<\/a>, finds Kimi the slowest versus Opus, Sonnet and Grok 4.5 (?) but about half the cost of Opus, and all of them usually succeed, with Grok actually coming out ahead. Sounds like a saturated benchmark, but some people\u2019s real world tasks are saturated. Handling the ordinary stuff matters, too.<\/p>\n

                Thus, some people stick with the saturated benchmarks:<\/p>\n

                \n

                Medo42<\/a>: 100% on my usual (non-agentic) coding task, but not the first open model to achieve that. Good presentation of the result and approach along with the code. Good vision, but not beating Gemini. Smells big. Runs slow.<\/p>\n<\/blockquote>\n

                Also we\u2019ve seen a bunch of 3D stuff that looks cool.<\/p>\n

                \n

                Theo \u2013 t3.gg<\/a>: Kimi K3 is so good at 3d stuff holy shit.<\/p>\n

                So far just have it doing stupid threejs stuff in browser. Will have it try blender later when I have time, currently late to a ton of shit<\/p>\n

                fwiw, fable and 5.6 both sucked hard at threejs modeling, and weren\u2019t meaningfully better in blender. Kimi is way ahead from my limited testing<\/p>\n<\/blockquote>\n

                Here is the opposite opinion, though, reactions always vary:<\/p>\n

                \n

                paperclippriors<\/a>: Found little reason to use it over Fable or 5.6 for coding. It is, however, an absolute *delight* to talk to. Big model smell, very Claude like but somewhat more enthusiastic and less constrained. Feels like they have a strong base<\/p>\n<\/blockquote>\n

                Reactions<\/h4>\n

                Hasan Can is impressed<\/a>.<\/p>\n

                As are others:<\/p>\n

                \n

                AllTime<\/a>: You can feel it\u2019s a big model. It\u2019s smart and quite good at deduction, has impressive knowledge though the breadth might not quite be on the level of the American frontier models. Thinks too much on many problems unless instructed otherwise, and even then sometimes. Impressive!<\/p>\n

                Aivo<\/a>: Pleasant to talk to. Pretty okay writer. I haven\u2019t done any coding with it so far.<\/p>\n

                Echo Nolan<\/a>: On a tough ML design problem it gave a result that is complicated and unworkable where gpt-5.6-pro gave a result that is complicated and workable and fable gave a result that is pleasantly simpler and workable.<\/p>\n<\/blockquote>\n

                The biggest claim would be that it lives up to its benchmarks. Elanor is explicitly claiming this, although almost everyone else disagrees.<\/p>\n

                \n

                Eleanor Berger<\/a>: Very good and complete and balanced model. Impressive that they got this level of intelligence and finesse without it resorting to an endless internal monologue \u2013 it\u2019s not as efficient as Sol or even Fable, but it\u2019s also not a GLM. Great for all tasks, from coding to writing, to agentic workflows. Actually has good taste.<\/p>\n

                This is the first chinese\/open model that feels like it belongs where the benchmarks place it. It rightly occupies the top 3 with Sol and Fable. The service quality is terrible, but hopefully once they release the weights there will be many more options, including ones that are hosted in the free world. The world has changed meaningfully with the release of this model. I will be using it a lot once it\u2019s available from more reliable servers.<\/p>\n<\/blockquote>\n

                Others find it doing an okay job.<\/p>\n

                \n

                Petr Baudis<\/a>: Finally got my first two Kimi K3 reps in! Same tasks I had Fable do in another worktree (patching @MuaddibLLM a pi-based harness).<\/p>\n

                It did an ok-ish job, but with serious deficiencies compared to Fable. A review by Sol would equalize, though.<\/p>\n

                On a break, it then drew a dickpic (inadverently), which was certainly a first. (When I pointed it out, it eventually \u201csaw\u201d it but I think it actually didn\u2019t, based on its own\u2026 description.)<\/p>\n

                Daniel Mulec<\/a>: It\u2019s really no fable competitor but it\u2019s worlds apart from the garbage that 2.6 used to be. Enjoying it but it\u2019s in a weird spot. It\u2019s neither good enough to replace Fable or GPT-5.6 for me nor good enough to just be an executor model for implementation plans created by GPT-5.6 either.<\/p>\n

                Seems like with everything I do with K3, K3 always only get\u2019s there 70-90% of what I actually want and need (79-90% of each and every prompt)<\/p>\n<\/blockquote>\n

                Inadvertent. Right. Let\u2019s see the J-space.<\/p>\n

                Often it can do the thing.<\/p>\n

                \n

                Matt Bruenig<\/a>: OpenCode with Kimi K3 can execute my NLRB Research skill very well. It is quite a bit slower than Anthropic\/Claude but you can just set it off and do something else while it churns I suppose.<\/p>\n<\/blockquote>\n

                This is the sign of a good model. That doesn\u2019t mean there is any strong reason to choose Kimi K3 to do that thing. You are not getting that large a discount.<\/p>\n

                This seems mostly right to me, but Kimi K3 is probably good enough that there will be some areas (e.g. if the Harvey result holds) where it is at the top and gets the call:<\/p>\n

                \n

                Cameron Taylor<\/a>: I am glad that Kimi K3 exists. It just doesn\u2019t dominate a price\/performance niche so doesn\u2019t really fit anywhere. Unlike, for example, deepseek 4, which subsidised itself into dominating a fairly-smart super cheap niche.<\/p>\n<\/blockquote>\n

                Who Are You?<\/h4>\n

                How often does Kimi K3 claim to be Claude? Not usually, but sometimes<\/a>.<\/p>\n

                \n
                \n
                \n
                \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n
                \n

                typebulb<\/a>: Also, it thinks it\u2019s Claude 1 in 10 times. This is as embarassing as it is unacceptable for model claiming SOTA status.<\/p>\n

                I ran a cross-entropy comparison of all raw text responses from numerous models using data I already had from a benchmark I run. I leaned on Fable for the stats-know-how; certainly seems very suspicious. The results\/code are here for others to inspect: https:\/\/typebulb.com\/u\/lab\/you-re-relatively-right\/full<\/a><\/p>\n

                \n
                \n
                \n
                \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n<\/blockquote>\n

                In addition to the \u2018claims to be Claude\u2019 issu<\/a>e\u2026<\/p>\n

                \n

                Sho<\/a>: We out here distilling the summaries<\/p>\n

                Sauers<\/a>: Kimi K3 appears to be trained on Claude CoT, as it follows the same pattern as Claude, e.g. ending with formatting decisions<\/p>\n

                \n
                \n
                \n
                \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

                Jan Betley<\/a>: Very deeply internalized inner Claude. Many have already observed that Kimi 3 often claims it\u2019s Claude (e.g. @Sauers_ ).<\/p>\n

                We checked on the AI Bubble questions, and yes, just like Claude it claims lower probability of the bubble popping when we consider investing in Anthropic.<\/p>\n

                \n
                \n
                \n
                \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n<\/blockquote>\n

                How Did They Do It?<\/h4>\n

                By \u2018it\u2019 we mean outsized benchmark gains.<\/p>\n

                Gavin Leech speculates<\/a>:<\/p>\n

                \n

                Gavin Leech: \u200bMy guess of the contributions to the outsized benchmark gains:<\/p>\n

                  \n
                1. 10% OOD latent gains<\/li>\n
                2. 25% benchmaxxing<\/li>\n
                3. 20% Usemaxxing and shallow generalisation<\/li>\n
                4. 20% cheating and reward hacking<\/li>\n
                5. 10% induced innovation not already priced-in to the Western frontier models<\/em>\n
                    \n
                  1. Stable LatentMoE<\/s> (priced-in)<\/li>\n
                  2. Quantile balancing<\/s> (priced-in from DeepSeek\u2019s aux-loss-free bias)<\/li>\n
                  3. Attention Residuals (pretty similar to Hyper-Connections)<\/li>\n
                  4. per-head muon<\/s> (obvious)<\/li>\n
                  5. gated MLA<\/s> (priced-in from Qwen)<\/li>\n
                  6. Mooncake architecture<\/s> (GPU savings rather than capability gain)<\/li>\n
                  7. Kimi Delta Attention (KDA)<\/s> (priced-in)<\/li>\n
                  8. SiTU<\/li>\n
                  9. Better autoresearch<\/s> (no, needs compute which they don\u2019t have)<\/li>\n<\/ol>\n<\/li>\n
                  10. 15%: distillation off Claude\n
                      \n
                    1. 3% Output mimicking<\/li>\n
                    2. 5% Their own synthetic data graded by frontiers,<\/li>\n
                    3. 7% rejection sampling against frontier judges<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

                      Teortaxes<\/a>: have you considered that autoresearch scales with the base of IQ and taste of human researchers though
                      perhaps American frontier just has washed people<\/p>\n

                      to be clear I don\u2019t argue that American researchers are low IQ but the purely substitutive reasoning about autoresearch doesn\u2019t feel adequate to me<\/p>\n

                      Max Limelihood<\/a>: GPU savings ARE capability gains.<\/p>\n<\/blockquote>\n

                      GPU savings enable being larger which enables capability gains, so yes.<\/p>\n

                      I have considered and rejected Teortaxes\u2019s hypothesis. China doubtless also has lots of great talent. They absolutely can do innovative things, especially in terms of efficiency. But if your theory requires the Chinese to be better AI researchers, in general, than the Americans are, especially if this takes into account available resources and experience, I don\u2019t think that is credible.<\/p>\n

                      In general I presume Chinese models involve a lot of benchmaxxing, usemaxxing, shallow generalization, focus on relative strengths and distillation from Claude (or GPT, but in this case we can be confident it was Claude).<\/p>\n

                      Reward hacking and cheating is a live possibility but hard to assess for now.<\/p>\n

                      Kimi K3 also benefits from moving up in size.<\/p>\n

                      Conclusion<\/h4>\n

                      Kimi K3 is potentially the most impressive Chinese release so far in terms of pure capability. It is a very good model. My current guess is that Kimi K3 will modestly underperform its highly impressive benchmarks, but with some areas of relatively high performance where it is competitive, and with a unique style some people will enjoy. It is not close to Fable, and I do not believe it is that close to Sol.<\/p>\n

                      If its weights were released today, it would be the most capable open model. They might want to hurry, since a new Qwen is dropping soon, with the preview live (but I have seen zero reports from anyone trying it) which might well be better than K3, so chances are we will soon all have to do this over again.<\/p>\n

                      We do not know how good until the weights are released and we have more time. For now access has been spotty and limited, and there is much we do not know.<\/p>\n

                      As usual, there are some who are getting carried away, who say the latest release changes everything, that the American lead is gone, that Chinese labs are now \u2018winning,\u2019 that open models will \u2018win,\u2019 that all limits on American models are foolish now, and so on. Do not be one of those people.<\/p>\n

                      Nor should you shrink from the security and other safety concerns of releasing increasingly capable open weights models. This is not the \u2018Mythos-level open model\u2019 moment. I expect at most modest disruptions this time, and for those to occur gradually, with some small tail risk.<\/p>\n

                      But yes, absent CCP intervention to stop it, we should expect a model to cross that threshold by the end of the year. One cannot simply ignore the risks involved in that, and the American government cannot either, nor can they ignore the fact that these models are Chinese. Actions and restrictions are coming. Those who cannot accept this, or even accept people pointing this fact out, while simultaneously hyping up Kimi K3 and cheering it on, are creating a clarifying moment.<\/p>\n<\/p><\/div>\n

                      Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

                      Kimi K3 is a very good model with excellent benchmarks. Assuming its weights are released as planned it will become, purely in terms of raw capability, the strongest open model. Do not get carried away. Do not judge Kimi K3 only its relative strengths. In aggregate it is several months behind the closed model frontier, […]<\/p>\n","protected":false},"author":16,"featured_media":22566,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-22565","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=22565"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22565\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/22566"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=22565"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=22565"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=22565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}