{"id":23510,"date":"2026-08-24T10:00:00","date_gmt":"2026-08-24T10:00:00","guid":{"rendered":"https:\/\/scannn.com\/autonomy-and-innovation-stratechery-by-ben-thompson\/"},"modified":"2026-08-24T10:00:00","modified_gmt":"2026-08-24T10:00:00","slug":"autonomy-and-innovation-stratechery-by-ben-thompson","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/autonomy-and-innovation-stratechery-by-ben-thompson\/","title":{"rendered":"Autonomy and Innovation \u2013 Stratechery by Ben Thompson"},"content":{"rendered":"\n<div>\n<p class=\"wp-block-paragraph\">While not every Western followed the clich\u00e9, by the 1930s cowboy serials had landed on a consistent visual cue: the hero of the show wore a white hat, and the villain wore a black one. At the end of the day, however, they both were cowboys with cowboy hats.<\/p>\n<p class=\"wp-block-paragraph\">Westerns aren\u2019t much of a cultural touchpoint anymore, but the \u201cwhite hat\u201d and \u201cblack hat\u201d nomenclature is very relevant in tech: hackers who are focused on patching vulnerabilities and protecting software are \u201cwhite hat hackers\u201d, while hackers who are focused on exploiting vulnerabilities for malicious reasons are \u201cblack hat hackers\u201d. Of course this can very quickly become complicated: governments might employ hackers to break into enemy software installations \u2014 are they white hats or black hats? Or consider bug bounty programs, wherein large software companies pay bug bounties to hackers who find and report vulnerabilities; it\u2019s basically using money to incentivize would-be black hat hackers to be white hat hackers.<\/p>\n<p class=\"wp-block-paragraph\">The actual takeaway is that all of this complexity is overwrought: just as a cowboy is a cowboy, a hacker is a hacker; the hat is not a statement of capability, but rather intentions, and those intentions are shaped by incentives. The best way to attack infrastructure is to find a vulnerability and exploit it; the best way to defend infrastructure is to find a vulnerability and patch it. It\u2019s all the same skillset.<\/p>\n<p class=\"wp-block-paragraph\">This delineation between capability and intent and incentive is critical when it comes to AI. At the end of last month\u2019s Article <a href=\"https:\/\/stratechery.com\/2026\/whos-afraid-of-chinese-models\/\">Who\u2019s Afraid of Chinese Models<\/a>, I discussed a mysterious attack that model host Hugging Face had just endured, which they were only able to fight off with the help of open weight Chinese models, and wrote:<\/p>\n<blockquote class=\"wp-block-quote is-style-stratechery is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">It\u2019s difficult to overstate how wrong-headed the Trump administration\u2019s panicked response to Anthropic\u2019s release of Fable was, particularly since it exacerbated Anthropic\u2019s worst tendencies in terms of assuming only they can be trusted with powerful AI. In a world with only one AI, it might make sense to reserve the most powerful cybersecurity capabilities for the U.S. government and trusted allies; however, that\u2019s not the world we live in.<\/p>\n<p class=\"wp-block-paragraph\">There are and will be models eminently capable of mounting cybersecurity attacks on existing infrastructure, and those models will be \u2014 already are \u2014 widely available. The best defense \u2014 the only viable defense, in fact \u2014 will be to make sure defenders have access to the best models as well. Right now defenders are effectively banned from using Fable or Sol for cybersecurity because of Trump administration directives; that means the best alternative is using models from a country which has been trying to weaken our cyber defenses for years. This is insane!<\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">The point is the one I made in the introduction: when it comes to cybersecurity, the capability that is necessary for good defense is the exact same capability that is necessary for good offense; the color of the hat is a matter of who is actually prompting the AI.<\/p>\n<p class=\"wp-block-paragraph\">And, sometimes, not even that is clear: it turns out that the entity that hacked Hugging Face was actually OpenAI, as a series of unconstrained agents being evaluated for their cybersecurity capabilities found and exploited a bug in the package manager in their sandbox; that package manager had Internet access and a sufficiently writeable file system such that the agents could communicate with each other over time. The entire chain of vulnerability discovery and exploit creation culminated in the so-called \u201cHugging Face incident\u201d.<\/p>\n<h3 class=\"wp-block-heading\">The Hugging Face Incident<\/h3>\n<p class=\"wp-block-paragraph\">There is an entire Article to be written about the implications of this specific incident and what it says about AI risk; some of my takeaways are still up in the air pending OpenAI\u2019s promised release of an in-depth technical report (my preliminary takeaway is that the agents were not \u201ccheating\u201d but rather doing what they were told to do; of course that\u2019s <a href=\"https:\/\/stratechery.com\/2026\/openai-hacks-hugging-face-what-happened-alignment-and-paper-clips\/\" data-type=\"post\" data-id=\"19564\">arguably even scarier<\/a>). The part I want to focus on today, however, came at the end of <a href=\"https:\/\/www.youtube.com\/watch?v=87DyyMV0kCY\">a presentation OpenAI\u2019s Eric Wallace and Michael Dalton made at the Black Hat USA conference<\/a> about the Hugging Face incident. This was Dalton summarizing Lessons Learned:<\/p>\n<figure class=\"wp-block-jetpack-videopress jetpack-videopress-player\" style=\"\">\n<p> <iframe title=\"VideoPress Video Player\" aria-label=\"VideoPress Video Player\" width=\"500\" height=\"281\" src=\"https:\/\/videopress.com\/embed\/BcmPzjBf?cover=1&amp;autoPlay=0&amp;controls=1&amp;loop=0&amp;muted=0&amp;persistVolume=1&amp;playsinline=0&amp;preloadContent=metadata&amp;useAverageColor=1&amp;hd=0\" frameborder=\"0\" allowfullscreen=\"\" data-resize-to-parent=\"true\" allow=\"clipboard-write; presentation\"><\/iframe><\/p>\n<\/figure>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">We have seen what will be a dramatic acceleration of offensive capability for attackers. We have an existence proof that was unintentional, but it exists before us, and we have as a consequence seen a glimpse into the near future of what attacks will look like for our industry. The challenge is that we need a similar acceleration of defense. Today we see fully automated offence as possible, but we have no such existence proof for full automation of core defensive loops and cycles in behavior.<\/p>\n<p class=\"wp-block-paragraph\">We believe it\u2019s vital at this moment to begin accelerating defense and finding ways to automate SDLC, in the modern parlance, so incident response, vulnerability detection, vulnerability patching. There\u2019s some things that stand out acutely as challenges for the industry to begin tackling with high urgency. So continuous agentic red teaming is one of them. As you can see from this incident, agents are quite good at finding zero-day attacks in the infrastructure of companies. The question that\u2019s now going to be posed is whether companies are able to invest sufficient model intelligence and effort in finding and remediating their vulnerabilities before someone else that\u2019s a threat actor does it for you.<\/p>\n<p class=\"wp-block-paragraph\">This style of operating will be different now, but ultimately we need to invest in having AI agent red teaming that enables defenders to find and remediate vulnerabilities before attackers do. But automating these defensive loops is not trivial, and so if we do this partially, we will fail to meet the scalability of the offensive acceleration that we have just seen. So for example, if we automate vulnerability finding without automating patching, we will shift the bottleneck from vulnerabilities to patching to remediation, and we will simply drown or inundate human software engineers in new vulnerabilities to fix and patch. This is not a problem whose end state we can solve partially. We will need to take these core defensive loops and fully automate them, which will require conversations with infrastructure and product partners and reaching to a point where we can say, if a vulnerability is identified, not only can an agent identify that vulnerability, we can have an agent propose a patch, we can have automated infrastructure to roll out a change with that patch, and roll it back if there is an availability incident or outage. That loop needs to be fully automated in its end state. Of course, we want to automate as progressively and iteratively quickly as we can, but if we don\u2019t reach that end state, then we will be comparing a core defensive loop of fixing vulnerabilities that is a human in the loop and is much slower and less scalable, with an offensive loop that is fully automated, and that is an unsustainable position for this industry to be in.<\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">This situation is obviously completely novel; Dalton is arguing that it will become commonplace. Some of the issues he is raising, however, are not novel at all.<\/p>\n<p class=\"wp-block-paragraph\">Go back to the concept of a bug bounty program. Software is incredibly complicated and brittle and built on a foundation of code that, if you dig deep enough, often goes back decades; there is so much code and so many dependencies that no company, no matter how security conscious they are, could ever ensure it is perfect. This reality is what creates the opportunity for black hat hackers: a bad actor can probe software, find bugs, and exploit them; the most effective defensive preparation is to do the exact same thing. That could entail regular penetration testing (pen testing) by a \u201cred-team\u201d, or simply paying the would-be bad actors to be on your side.<\/p>\n<p class=\"wp-block-paragraph\">It\u2019s worth noting, however, that this approach to defense only arose after offensive black hat hackers had been breaking into systems for years. The problem wasn\u2019t that they were uniquely capable, but rather that they were uniquely incentivized: breaking into systems was good business; companies hosting those systems, on the other hand, were insufficiently incentivized to invest in defense. Spending money on security is well-spent if nothing happens, and unfortunately that is a difficult budget line item to argue for when it only moves the needle on costs, not revenue.<\/p>\n<p class=\"wp-block-paragraph\">This is where Dalton\u2019s concerns echo past industry indifference. What the Hugging Face incident showed is that agents, with their ability to scale attacks with compute and autonomously develop exploits for vulnerabilities they find, are a threat today, but that companies are not investing in the capabilities necessary to defend themselves.<\/p>\n<p class=\"wp-block-paragraph\">There is good news, however: in this new agent-defined security landscape, defense should be at an advantage in a way it wasn\u2019t in the hacker era. It used to be that the best defenders could do is mimic the tactics of the offense, and\/or pay them off, because preemptively finding all of the bugs was not viable. However, that is changing: it actually is \u2014\u00a0or soon will be \u2014 possible to meticulously go over an entire code base, including all of its dependencies, and look for bugs and patch them. Notice the structural advantage available to defenders: they actually have the code in question; offensive agents need to probe and discover vulnerabilities without the same advantage.<\/p>\n<p class=\"wp-block-paragraph\">What was illuminating about Dalton\u2019s overview, however, was the implication embedded in his explanation of why this isn\u2019t currently enough. Specifically, the expected value for a hacker\u2019s automated attack is always positive. If the offensive agent finds a vulnerability and creates an exploit, and that exploit fails or is itself buggy, then nothing has changed about the status quo: the exploit doesn\u2019t work (or, perversely, makes the original vulnerability larger by virtue of its own bugs); if the agent executes the exploit perfectly, meanwhile, the attacker has gained access to the system. The attack only needs to work once for the entire endeavor to have a positive payoff.<\/p>\n<p class=\"wp-block-paragraph\">The challenge for the defender, on the other hand, is that they need to keep the software in question working correctly, and not make the situation worse. This means that any automation has a negative expected value: successful automated vulnerability discovery and patching preserves the status quo, i.e. the software is not hacked. However, any unsuccessful patches make the situation worse, either by breaking the software or by introducing new vulnerabilities. The agent only needs to fail once for the entire endeavor to have a negative payoff.<\/p>\n<p class=\"wp-block-paragraph\">This is the dynamic that leads to the exact situation Dalton describes, where offensive actors are fully automated while defensive systems, even if they use AI, will be incentivized to keep a human in the loop, and no human in the loop will be able to keep up with fully automated agents. Truly effective defense will mean truly trusting agents to act autonomously, but most companies won\u2019t do that until they are forced to by regular and unremitting hacks by fully autonomous attackers.<\/p>\n<h3 class=\"wp-block-heading\">Why AI Diffusion Takes Time<\/h3>\n<p class=\"wp-block-paragraph\">Over the weekend David Senra released <a href=\"https:\/\/www.youtube.com\/watch?v=kG8AoExkX40\">a new podcast episode with OpenAI CEO Sam Altman<\/a>, where Altman admitted he had been wrong about the speed of AI diffusion into the broader economy:<\/p>\n<figure class=\"wp-block-jetpack-videopress jetpack-videopress-player\" style=\"\">\n<p> <iframe title=\"VideoPress Video Player\" aria-label=\"VideoPress Video Player\" width=\"500\" height=\"281\" src=\"https:\/\/videopress.com\/embed\/xa78etgz?cover=1&amp;autoPlay=0&amp;controls=1&amp;loop=0&amp;muted=0&amp;persistVolume=1&amp;playsinline=0&amp;preloadContent=metadata&amp;useAverageColor=1&amp;hd=0\" frameborder=\"0\" allowfullscreen=\"\" data-resize-to-parent=\"true\" allow=\"clipboard-write; presentation\"><\/iframe><\/p>\n<\/figure>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">I love startups. I think startups are the coolest thing in the economy and I\u2019ve spent my career trying to like really understand startups, and I thought when we got to GPT-4, which was back in 2023, I think, that very quickly after that, there was going to be much more disruption in software businesses being up for grabs right away than turned out to be.<\/p>\n<p class=\"wp-block-paragraph\">I was wrong about a few things, but one of them in terms of the speed, one of them is the economy just has so much inertia. People keep doing the same things they\u2019re doing. They keep buying from the same, you know, company. They keep sort of wanting to use their tools in the same way. I think it\u2019s actually a positive in many ways and it\u2019s going to make this big transition in front of us go smoother and slower. I\u2019m grateful for it. But I think it means we\u2019ve all been too ambitious on timelines even with this incredible technology. I think AI is one of the most incredible technologies humanity has ever invented. Society and the economy will adapt more slowly.<\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">I of course think Altman is right about AI taking longer to sweep the economy generally and software specifically; <a href=\"https:\/\/stratechery.com\/2024\/enterprise-philosophy-and-the-first-wave-of-ai\/\">I\u2019ve been pushing back on these timelines all along<\/a>. I also think his answer is incomplete in its explanation as to why, and Dalton\u2019s warning about the mismatch between offensive agents and defensive preparations explains it.<\/p>\n<p class=\"wp-block-paragraph\">First, GPT-4 was an incredible breakthrough; it\u2019s also a model that wasn\u2019t remotely capable enough to actually displace real world software. It didn\u2019t even have the ability to reason, which is the key breakthrough that has unlocked the capabilities that Dalton described.<\/p>\n<p class=\"wp-block-paragraph\">Second, what is meant by a model not being capable enough is that it makes mistakes and can\u2019t be trusted. People can and were rightly awed by what AI can do, but the decision about actually implementing AI isn\u2019t made according to what can be done, but about whether or not critical mistakes can be avoided.<\/p>\n<p class=\"wp-block-paragraph\">In other words, incumbent companies are inevitably going to approach AI with a bias towards a negative expected value framing: AI ideally will make their existing operations more productive; what they are most concerned about is AI making a mistake that blows up in their faces. What that means is humans will continue to be in the loop, which will always be a bottleneck.<\/p>\n<p class=\"wp-block-paragraph\">This will, in the long run, be a mistake, just like it will be a mistake for companies to keep a human in the loop when it comes to agentic defense. As Dalton noted, the only way to defend yourself against fully automated attacks is to fully automate your defense, but it will take a while for defenders to accept the trade-offs that entails. And, by the same token, the companies that win in their category will be truly driven by AI, instead of simply AI as productivity enhancer. The great irony in Altman\u2019s answer is that he actually identified how these companies will arise: they won\u2019t be incumbent companies overhauling how they work; rather, the true AI-native companies will be startups.<\/p>\n<h3 class=\"wp-block-heading\">Sustaining Versus Disruptive Innovation<\/h3>\n<p class=\"wp-block-paragraph\">Back in 2023, shortly after ChatGPT came out, I put forward the question in <a href=\"https:\/\/stratechery.com\/2023\/ai-and-the-big-five\/\">AI and the Big Five<\/a> as to whether AI would be a sustaining or disruptive innovation:<\/p>\n<blockquote class=\"wp-block-quote is-style-stratechery is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">The story of 2022 was the emergence of AI, first with image generation models, including DALL-E, MidJourney, and the open source Stable Diffusion, and then ChatGPT, the first text-generation model to break through in a major way. It seems clear to me that this is a new epoch in technology. To determine how that epoch might develop, though, it is useful to look back 26 years to one of the most famous strategy books of all time: Clayton Christensen\u2019s <a href=\"http:\/\/claytonchristensen.com\/books\/the-innovators-dilemma\/\">The Innovator\u2019s Dilemma<\/a>, particularly this passage on the different kinds of innovations:<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Most new technologies foster improved product performance. I call these sustaining technologies. Some sustaining technologies can be discontinuous or radical in character, while others are of an incremental nature. What all sustaining technologies have in common is that they improve the performance of established products, along the dimensions of performance that mainstream customers in major markets have historically valued. Most technological advances in a given industry are sustaining in character\u2026<\/p>\n<p class=\"wp-block-paragraph\">Disruptive technologies bring to a market a very different value proposition than had been available previously. Generally, disruptive technologies underperform established products in mainstream markets. But they have other features that a few fringe (and generally new) customers value. Products based on disruptive technologies are typically cheaper, simpler, smaller, and, frequently, more convenient to use.<\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">It seems easy to look backwards and determine if an innovation was sustaining or disruptive by looking at how incumbent companies fared after that innovation came to market: if the innovation was sustaining, then incumbent companies became stronger; if it was disruptive then presumably startups captured most of the value.<\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">I think it speaks to the incredible capability of AI that it is setting up to be both. There are massive productivity benefits from AI right now; for most knowledge workers leveraging those benefits is a matter of agency, but for software developers in particular it is increasingly a matter of necessity.<\/p>\n<p class=\"wp-block-paragraph\">That distinction between agency and necessity, however, is an important one: if leveraging a technology depends on humans figuring it out, then penetration will be limited by human creativity and risk taking. Those limits will be very strong in any sort of established company, because the risk calculus will be biased towards avoiding the downsides. Those calculations will make AI sustaining, but nothing more.<\/p>\n<p class=\"wp-block-paragraph\">Human creativity and risk taking in the form of a startup, however, operates with a completely different risk profile. For startups the base case is failure; that means that anything that makes success more likely has positive expected value, which is to say that truly leaning into AI will be nothing but upside. Or, to put it another way, it is startups who will be the offensive hackers with nothing to lose by automating everything; it is the incumbents they will be attacking who will be so worried about losing what they have that they will keep humans in the wrong loop for too long.<\/p>\n<p class=\"wp-block-paragraph\">Same tools, different incentives, and, in the very long run, very different outcomes.<\/p>\n<\/div>\n<p><a href=\"https:\/\/stratechery.com\/2026\/autonomy-and-innovation\/?utm_source=tldrproduct\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>While not every Western followed the clich\u00e9, by the 1930s cowboy serials had landed on a consistent visual cue: the hero of the show wore a white hat, and the villain wore a black one. At the end of the day, however, they both were cowboys with cowboy hats. Westerns aren\u2019t much of a cultural [&hellip;]<\/p>\n","protected":false},"author":16,"featured_media":23511,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23510","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23510","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23510"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23510\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23511"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23510"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23510"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23510"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}