{"id":23235,"date":"2026-08-11T16:31:23","date_gmt":"2026-08-11T16:31:23","guid":{"rendered":"https:\/\/scannn.com\/ryan-greenblatt-what-happens-once-ai-can-automate-ai-research\/"},"modified":"2026-08-11T16:31:23","modified_gmt":"2026-08-11T16:31:23","slug":"ryan-greenblatt-what-happens-once-ai-can-automate-ai-research","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/ryan-greenblatt-what-happens-once-ai-can-automate-ai-research\/","title":{"rendered":"Ryan Greenblatt \u2013 What happens once AI can automate AI research?"},"content":{"rendered":"\n<div dir=\"auto\">\n<p><span>Had Ryan Greenblatt on to discuss\/debate recursive self-improvement.<\/span><\/p>\n<p><span>This might be the most important question in the world right now \u2013 whether within a year or so of achieving human-level intelligence, you slingshot towards having 10s of billions of superintelligences, each of which is dramatically more competent than human experts across all fields.<\/span><\/p>\n<p><span>I\u2019ve historically been skeptical of this possibility. My intuition has been that we will end up significantly bottlenecked by not only compute scaling but human expert data, which I think underlies most of the AI progress today.<\/span><\/p>\n<p><span>If, because of RSI, we got a jump as big as GPT-3 to a Mythos (i.e. 6 years of AI progress) within a single year of achieving AGI, then the thing we get there at the end of that year is definitively and wildly superhuman.<\/span><\/p>\n<p><span>We hashed it out, and I think Ryan made a pretty good case that this kind of speedup is plausible. FWIW, Ryan\u2019s median for when we automate AI R&amp;D is 2031.<\/span><\/p>\n<p><span>We then discussed the alignment implications of this scenario. Who should these superintelligences be aligned to? In the future, our capacity to steward our votes and our capital, and to make sense of what\u2019s happening in the world, will all be titrated by superintelligences. And I worry that specs like the Claude Constitution are not shaping these ASIs to truly be my personal advocates and guardian angels.<\/span><\/p>\n<p><span>And can we get them aligned to anything in the first place? Ryan and I had a long debate about whether the kind of reward hacking we saw with the OAI\/Hugging Face hack extrapolates to superintelligences that would team up to literally take over the world.<\/span><\/p>\n<p><span>The first piece of advice you get when you\u2019re learning to drive is that it will go much smoother if you look at the horizon instead of directly in front of your tires. And so it is with the trajectory of AI. Hope you enjoy!<\/span><\/p>\n<p><span>Watch on <\/span><a href=\"https:\/\/youtu.be\/-RXD4bTuFTo\">YouTube<\/a><span>; listen on <\/span><a href=\"https:\/\/podcasts.apple.com\/us\/podcast\/ryan-greenblatt-human-level-ais-might-build-runaway\/id1516093381?i=1000782779590\">Apple Podcasts<\/a><span> or <\/span><a href=\"https:\/\/open.spotify.com\/episode\/4TdEXIVDv9AxT30DGG0KR1?si=8U6qFnEAQA-ULathx_7Cdw\">Spotify<\/a><span>.<\/span><\/p>\n<div id=\"youtube2--RXD4bTuFTo\" data-attrs=\"{&quot;videoId&quot;:&quot;-RXD4bTuFTo&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}\" data-component-name=\"Youtube2ToDOM\" class=\"youtube-wrap\">\n<p><iframe src=\"https:\/\/www.youtube-nocookie.com\/embed\/-RXD4bTuFTo?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0\" frameborder=\"0\" loading=\"lazy\" gesture=\"media\" allow=\"autoplay; fullscreen\" allowautoplay=\"true\" allowfullscreen=\"true\" width=\"728\" height=\"409\"><\/iframe><\/p>\n<\/div>\n<ul>\n<li>\n<p><a href=\"https:\/\/antithesis.com\/dwarkesh\"><span>Antithesis<\/span><\/a><span> is a software testing platform that finds the failures no human or AI could ever anticipate. It runs thousands of copies of your code inside a fully deterministic computer, injecting faults and steering each trajectory toward the most insidious bugs. This lets you find critical issues in minutes rather than waiting months for your users to uncover them. Learn more at <\/span><a href=\"https:\/\/antithesis.com\/dwarkesh\"><span>antithesis.com\/dwarkesh<\/span><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/janestreet.com\/dwarkesh\"><span>Jane Street<\/span><\/a><span>\u2019s back with a new puzzle. They designed an ASIC and sent me the final masks\u2026 but they didn\u2019t tell me what the chip actually does. So that\u2019s the challenge: reverse engineer the circuit and figure out the chip\u2019s purpose. Jane Street has a bunch of swag ready to send to the most creative solutions, and they\u2019re also planning to feature the top write-ups in a blog post. Download the files and get started at <\/span><a href=\"https:\/\/janestreet.com\/dwarkesh\"><span>janestreet.com\/dwarkesh<\/span><\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/cursor.com\/dwarkesh\"><span>Cursor<\/span><\/a><span> and SpaceX recently released Grok 4.5, and I\u2019ve been surprised by just how good the model is. For example, when I tested it against Fable and Sol on a bunch of AI governance questions, all three models gave substantially the same answers, but Grok was faster, more concise, and cheaper. Grok 4.6 is coming soon, but in the meantime, you can try 4.5 at <\/span><a href=\"https:\/\/cursor.com\/dwarkesh\"><span>cursor.com\/dwarkesh<\/span><\/a><\/p>\n<\/li>\n<\/ul>\n<p><span>(00:00:00<\/span><span>)<\/span><span> \u2013 Is AI R&amp;D verifiable enough to unlock recursive self-improvement?<\/span><\/p>\n<p><span>(<\/span><span>00:16:52<\/span><span>)<\/span><span> \u2013 Is AI progress bottlenecked by human expert data?<\/span><\/p>\n<p><span>(<\/span><span>00:34:02<\/span><span>)<\/span><span> \u2013 Flat token prices suggest scaling has been slow<\/span><\/p>\n<p><span>(<\/span><span>00:39:47<\/span><span>)<\/span><span> \u2013 Skills AI can\u2019t train on: does it even need them?<\/span><\/p>\n<p><span>(<\/span><span>00:48:07<\/span><span>)<\/span><span> \u2013 Aligned to whom?<\/span><\/p>\n<p><span>(<\/span><span>01:09:18<\/span><span>)<\/span><span> \u2013 Recent incidents of AIs colluding and deceiving humans<\/span><\/p>\n<p><span>(<\/span><span>01:19:38<\/span><span>)<\/span><span> \u2013 What could possibly go wrong? A concrete scenario<\/span><\/p>\n<p><span>(<\/span><span>01:48:02) \u2013 From reward hacking to takeover<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Today I\u2019m chatting with <\/span><a href=\"https:\/\/scholar.google.com\/citations?user=UESxdDsAAAAJ&amp;hl=en\"><span>Ryan Greenblatt<\/span><\/a><span>, who is the chief scientist at <\/span><a href=\"https:\/\/www.redwoodresearch.org\/\"><span>Redwood Research<\/span><\/a><span>, where he focuses on technical <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/AI_safety\"><span>AI safety<\/span><\/a><span> and security work.<\/span><\/p>\n<p><span>I want to talk to you about <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Recursive_self-improvement\"><span>recursive self-improvement<\/span><\/a><span>. This is the idea that once we build human-level intelligences, they quickly slingshot towards tens of billions of <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence\"><span>superintelligences<\/span><\/a><span>, which are each individually more competent than the top human experts across every field. Whether or not this turns out to be the case is probably the most important question in the world right now. And historically, I\u2019ve been quite skeptical that this kind of thing happens, but, you seem to think that it might be plausible, and so I wanted to hear the case for it.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Let\u2019s talk about this. First, I think it\u2019s worth noting that AI R&amp;D is a type of task at which the AIs are especially good, because the companies are trying really hard to make their AIs good at AI R&amp;D. It\u2019s also the kind of domain that has a lot of nice properties from the perspective of how AI development works right now. It\u2019s pretty verifiable. You can do a bunch of stuff iteratively, and it\u2019ll hill climb on various metrics.<\/span><\/p>\n<p><span>I think once you have AIs which are roughly matching the top human experts in AI R&amp;D, that could kick off a feedback loop where the AIs are doing AI research. That produces smarter AIs. That feeds back in. That feedback loop could be strong enough that you end up with a lot of progress in a short period of time. Maybe my median expectation is something like four or five years of AI progress in a single year. This requires really overcoming a huge amount of diminishing returns in research and basically doing the equivalent of the progress we would have gotten after a really large compute scale-out. So this is a pretty impressive, big thing.<\/span><\/p>\n<p><span>It\u2019s worth keeping in mind that five years of AI progress, four years of AI progress, even three years of AI progress, is really a lot of fucking AI progress. A little over three years ago, <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/GPT-4\"><span>GPT-4<\/span><\/a><span> had come out. Right now, of course, we have <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Claude_Mythos\"><span>Mythos 5<\/span><\/a><span> or whatever, and maybe a somewhat better model that Anthropic has internally. That is just a huge amount of progress in a bit over three years. If we\u2019re talking about five years, then maybe we\u2019re talking more about a jump from <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/GPT-3\"><span>GPT-3<\/span><\/a><span> to Mythos 5 or whatever.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I think this argument has three different parts. Now I want to evaluate each one of them. First is the argument that AI R&amp;D is very verifiable. Second is the argument that if you automate AI R&amp;D, you could get four or five years of progress in a single year. Third is the argument that what comes out the other end of four or five years of AI progress at the current pace, starting at the point whenever AI R&amp;D is automated, is an AI where you can drop it on the job at basically anything you can imagine.<\/span><\/p>\n<p><span>You can drop it in Texas politics in the 1940s, and it outmaneuvers <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Lyndon_B._Johnson\"><span>Lyndon Johnson<\/span><\/a><span>. You can drop it in <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/TSMC\"><span>TSMC<\/span><\/a><span>, and it learns how to do better process engineering at TSMC. It\u2019s certainly a better video editor\u2026 My video editors are very excellent, but it is just, in general, better than humans at any given job that it finds itself trying to do.<\/span><\/p>\n<p><span>So I want to evaluate all of these sub-arguments that lead to basically getting ASI pretty soon after this benchmark, which you\u2019re expecting by 2030 or something, right?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I would say that I expect full automation of AI R&amp;D perhaps somewhere around 2031, 2030. Getting to the \u201cbeats all humans on the job\u201d milestone, maybe my median expectation is around 2033. But if I see AIs fully automating AI R&amp;D, I think I\u2019m expecting that probably within a year. The way the forecasting works out, the difference between medians is bigger than the median difference between milestones. Anyway, whatever.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>By the way, there\u2019s this meme on the internet. Every time I\u2019m trying to ask about people\u2019s timelines, <\/span><a href=\"https:\/\/www.dwarkesh.com\/p\/dario-amodei-2\"><span>when I\u2019m asking Dario<\/span><\/a><span> or somebody, I\u2019m always like, \u201cOkay, how long before you automate my video editors?\u201d There\u2019s this <\/span><a href=\"https:\/\/x.com\/nkh1l\/status\/2022692208349188556?referrer=grok-com\"><span>meme of my video editor editing the podcast<\/span><\/a><span> every time I listen to this.<\/span><\/p>\n<p><span>But the reason I do it is because I think it\u2019s easy to get lost in abstractions when you talk about jobs you don\u2019t understand well, and to very concretely understand what it takes to automate a job that I actually understand why it\u2019s difficult for LLMs to currently take control over.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I do think that the milestone for automating your video editor is earlier than the milestone of being able to automate all human jobs, including Texas politics, spinning up on the job. I do think that the video editor automation occurs maybe more around full automation of AI R&amp;D, but it\u2019s very sensitive to how much people are really focusing on understanding video.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Okay, so let\u2019s start with the claim that AI R&amp;D is very verifiable.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>There\u2019s a few different parts of this. One of them is that we can train on a bunch of environments which are basically directly training the model to do some AI R&amp;D task or some very close-by task. For example, we can have some environment where the model is training some AI on just eight <\/span><a href=\"https:\/\/www.nvidia.com\/en-us\/data-center\/h100\/\"><span>H100s<\/span><\/a><span> or some small amount of compute, and that model could be the equivalent of GPT-2 medium or whatever, and then similar to NanoGPT medium runs \u2014 and in <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Reinforcement_learning\"><span>RL<\/span><\/a><span>, it\u2019s tweaking and iterating on that.<\/span><\/p>\n<p><span>We could do that for a bunch of different tasks. We could have it train <\/span><a href=\"https:\/\/www.geeksforgeeks.org\/computer-vision\/top-pre-trained-models-for-image-classification\/\"><span>image classification models<\/span><\/a><span>, <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Text-to-video_model\"><span>video generation models<\/span><\/a><span>, <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Text-to-image_model\"><span>image generation models<\/span><\/a><span>, all kinds of different ML training tasks. We could RL it on the task of training increasingly good models, and also doing things like, \u201cOh, here\u2019s a particular direction you could pursue for an algorithm. Can you go and implement that?\u201d<\/span><\/p>\n<p><span>Basically, there\u2019s this whole class of containerizable, verifiable, small-scale AI R&amp;D tasks that we can aggressively RL the AIs on. Already companies are presumably doing some RL on these sorts of tasks, and you could just keep scaling that up, keep making more of these small-scale AI R&amp;D tasks, and then the AIs could keep getting better at this. Implicitly, I\u2019m claiming this will transfer to extremely load-bearing aspects of AI R&amp;D. But maybe let\u2019s stop there for a second and then get to that part.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So let\u2019s talk through what this concretely looks like. You can imagine that we have GPT-7.5. We say, \u201cGPT-7.5, we want to make you so good at AI R&amp;D that you help us train GPT-9.\u201d So now we want to train GPT-7.5, and we come up with a bunch of different environments.<\/span><\/p>\n<p><span>As you mentioned, there\u2019s already this repo that is the <\/span><a href=\"https:\/\/github.com\/kellerjordan\/modded-nanogpt\"><span>descendant<\/span><\/a><span> of <\/span><a href=\"https:\/\/www.dwarkesh.com\/p\/andrej-karpathy\"><span>Andrej Karpathy<\/span><\/a><span>\u2018s <\/span><a href=\"https:\/\/github.com\/karpathy\/nanoGPT\"><span>nanoGPT<\/span><\/a><span> speedrun, where you just try to change everything about the model, from the <\/span><a href=\"https:\/\/ml-cheatsheet.readthedocs.io\/en\/latest\/optimizers.html\"><span>optimizer<\/span><\/a><span> to the <\/span><a href=\"https:\/\/www.ibm.com\/think\/topics\/hyperparameter-tuning\"><span>hyperparameters<\/span><\/a><span> to the architecture, to get it to a fixed training loss as fast as possible. You could have other kinds of environments where you could say, \u201cHey, GPT-7.5, I want you to train a really good video game-playing model. I want you to train a model that actually improves as it plays the same video game again and again. So you learn how to maybe help the model get better at online learning. We don\u2019t care how you figure this out. Maybe it\u2019s some kind of crazy <\/span><a href=\"https:\/\/www.lesswrong.com\/posts\/qehggwKRMEyWqvjZG\/reflections-on-neuralese\"><span>neuralese<\/span><\/a><span> or a vector memory. Or maybe it\u2019s just better long-context stuff. We don\u2019t care. Figure out how to do <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Online_machine_learning\"><span>online learning<\/span><\/a><span> research.\u201d<\/span><\/p>\n<p><span>Obviously, GPT-7.5 will already be a smart model, and in the same way the models currently are getting smarter, it\u2019ll be better and better at coding. You can imagine 100 other environments like this which are incentivizing the ability to do AI R&amp;D, like containerized versions of getting GPT-7.5 to develop <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/GPT-2\"><span>GPT-2<\/span><\/a><span>-sized models, et cetera. Then you basically put GPT-7.5 through a bunch of this kind of training, you build GPT-8. GPT-8 is now an amazing ML researcher. It has so much intuition from doing all this kind of training.<\/span><\/p>\n<p><span>Honestly, a huge intuition pump for me is seeing the <\/span><a href=\"https:\/\/www.newscientist.com\/article\/2583307-why-mathematician-terence-tao-thinks-ai-must-spark-a-rapid-revolution\/\"><span>progress that AI has made in mathematics<\/span><\/a><span>. If it\u2019s a very verifiable domain, AIs can get\u2026 I don\u2019t really know the object-level details of mathematics research, but I\u2019m just like, \u201cNo, it works.\u201d It can just come in like a flood if you can totally put it into a verification loop, and it can actually make new breakthroughs.<\/span><\/p>\n<p><span>I am curious if ML research has a quality of mathematical research where it seems like there was a big overhang from connecting different disciplines together. No one person would have known enough about algebraic geometry and\u2026 What was the right word?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Oh man, I really don\u2019t know about the math breakthroughs.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>No one person would\u2019ve known enough about <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Topology\"><span>topology<\/span><\/a><span> and algebraic whatever in order to make <\/span><a href=\"https:\/\/www.smithsonianmag.com\/smart-news\/ai-disproves-a-decades-old-mathematical-idea-the-biggest-conjecture-that-the-tech-has-played-a-role-in-yet-180989189\/\"><span>some counterexample to a big conjecture<\/span><\/a><span>.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>My view is that ML is a less deep domain than math, and so there\u2019s less of a thing where there are individual experts with really deep expertise in some area that they combine, but there\u2019s definitely going to be some of that.<\/span><\/p>\n<p><span>But then I also think that ML has some attributes that make it even more favorable to AI training than mathematics in some ways. In particular, you can get a better sense of whether you\u2019re succeeding, and you can see intermediate progress. In math, it\u2019s often the case that there\u2019s no easy way to see whether or not you\u2019re close to success. Whereas if your goal is, for example, to get to some training loss 2x faster, you can kind of see when you\u2019re halfway there. It tends to be the case that ML innovations are very additive, or maybe multiplicative depending on how you think about it, where basically you can keep stacking innovations. Usually the innovations just add together and don\u2019t interfere with each other, though obviously it\u2019s going to depend on the details.<\/span><\/p>\n<p><span>So I think that in a lot of ways, AI R&amp;D will have properties quite similar to math, where you can train on chunks of AI R&amp;D that are pretty similar in structure to the problem you actually cared about, in a very verifiable way, and then that will transfer. There\u2019s an open question of exactly how well it will transfer, but I think that the transfer currently for math looks pretty good. My expectation is that the transfer for AI R&amp;D will look pretty good, but not amazing.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So one concern I have is that I think even in mathematics, as far as I\u2019m aware, we have not seen very impressive new theory. We\u2019ve seen a lot of impressive, verifiable, specific results \u2014 for example, find a counterexample to this conjecture \u2014 but we have not seen \u201ccome up with the idea of <\/span><a href=\"https:\/\/mathworld.wolfram.com\/Topology.html\"><span>topology<\/span><\/a><span>\u201c kinds of levels of things, or \u201ccome up with things like <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Group_theory\"><span>group theory<\/span><\/a><span>\u201c.<\/span><\/p>\n<p><span>It seems like ML research has elements of both of these things. But the less verifiable thing of coming up with new ways of thinking about the problem would be harder to induce. Take, for example, the idea of <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Neural_scaling_law\"><span>scaling laws<\/span><\/a><span>. Obviously, there is some end verification loop such that you can train GPT-4 better if you have the <\/span><a href=\"https:\/\/arxiv.org\/abs\/2001.08361\"><span>idea of scaling laws from 2020<\/span><\/a><span>. But there is a longer and potentially more compute-laden road to inducing AIs to be like, \u201cOkay, I got to think carefully about how I should be scaling my parameters and data. What are different kinds of investigations I could run to understand this? Maybe I can come up with a visualization and an <\/span><a href=\"https:\/\/www.emergentmind.com\/topics\/isoflops-analysis\"><span>isoFLOP analysis<\/span><\/a><span> or something.\u201d But that does seem like a longer verification loop than just, \u201cHey, let\u2019s get nanoGPT loss to go down.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Let\u2019s talk about this. First of all, I think in the context of math, the thing I would say is that the AIs can do the equivalent of \u2018baby\u2019s first new theory,\u2019 where, for example, they can just prove interesting conjectures via making connections and producing new understanding. It\u2019s like, \u201cOh, there\u2019s this construction the AI found which is pretty interesting\u201d, or it found this way of thinking about the problem that\u2019s a bit different. We do see that. It\u2019s just that the examples we see are not as impressive as founding the field of group theory.<\/span><\/p>\n<p><span>Founding the field of group theory is probably among the best, biggest mathematical accomplishments of all time, and the AIs just aren\u2019t that good at math yet. From my perspective, there\u2019s a continuum between that and the things we\u2019re seeing now, that the AIs are continuing to march up.<\/span><\/p>\n<p><span>Second, I think ML is a very shallow domain relative to math. In math, there was much more of a thing where you find some true deep abstraction, and if you really understand that thing, which is hard to understand, then you get somewhere. Whereas I feel like the things that are the equivalent of that in ML are really dumb bullshit. Like with scaling laws, come on guys, we can explain scaling laws really quickly. I think the deepest and most important concepts in math, for example, don\u2019t have the property that you can really understand the underlying thing and why it matters in a very short period of time.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>But I feel like one effect will be that we will have gotten rid of all the low-hanging fruits by 2030. I feel like scaling laws will have been, in math history, like <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Ren%C3%A9_Descartes\"><span>Descartes<\/span><\/a><span> finding the <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cartesian_coordinate_system\"><span>Cartesian grid<\/span><\/a><span> and doing very basic mathematics. Eventually, if we want to keep making progress in the 2030s, it\u2019s going to be like doing whatever bullshit is happening at the frontiers of mathematics right now.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That could be right. My sense is that some domains are structurally different in terms of how they operate and how much they depend on deep abstractions. Physics and math are much more on the side of being very far on the deep, hard-to-come-up-with-ideas side, whereas I think ML and most other domains are much more amenable to <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Hill_climbing\"><span>hill climbing<\/span><\/a><span>. That\u2019s my sense of how this will go in the future.<\/span><\/p>\n<p><span>Even in the regime where your AIs are having to plow \u2014 it\u2019s 2030, a bunch of low-hanging fruit in research has already happened, and they need to make further progress \u2014 I still suspect that a bunch of the work will live more on the side of building increasingly complicated infrastructure and having really good intuition about what the experiments roughly look like. So I\u2019m probably less sympathetic to the idea that the thing the AIs will lack is some deep insight. I\u2019m more sympathetic to the idea that they really need a bunch of taste about in-the-weeds experiments that they currently don\u2019t have. They need a bunch of intuition for what sorts of training approaches would work and what wouldn\u2019t, in ways that current researchers have.<\/span><\/p>\n<p><span>Even in cases where there has been some breakthrough in AI, oftentimes in retrospect it looks like a big bottleneck to making that breakthrough happen was getting all of the micro details and mungy intuition right. An example of this is training AIs to be good at reasoning and <\/span><a href=\"https:\/\/research.google\/blog\/language-models-perform-reasoning-via-chain-of-thought\/\"><span>chain of thought<\/span><\/a><span>, doing RL on chain of thought. It looks like you probably could have done RL and chain of thought on GPT-3 and gotten kind of interesting results on math if you had really scaled it up and done a good job.<\/span><\/p>\n<p><span>But at the time, there was low-hanging fruit. Also, doing a good job with that training is kind of in the weeds on all the technical implementation and scaling it up and getting the hyperparameters right. So maybe you can demonstrate everything on <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Qwen\"><span>Qwen<\/span><\/a><span> <\/span><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen1.5-1.8B\"><span>1B<\/span><\/a><span> or whatever and get some sense that this whole thing is going to work. But people didn\u2019t demonstrate it as early as they could have because of all of these other mungy details and intuition about exactly how to tune the parameters and how to set things up.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>This is my remaining skepticism, honestly, about this story. I\u2019m not sure I understand why, if research breakthroughs are so amenable to intelligence, AI progress has not been historically faster than it could have been. As you were saying, by the time <\/span><a href=\"https:\/\/www.emergentmind.com\/topics\/rl-with-verifiable-rewards-rlvr\"><span>RLVR<\/span><\/a><span> actually worked \u2014 even though you could have done it with less compute \u2014 we had to wait for oceans of compute, gigawatts of compute, to be available before people were doing this training, on the trajectory of compute continuing to increase so we make more breakthroughs.<\/span><\/p>\n<p><span>I don\u2019t know. I feel like there were a lot of AI researchers in the year 2022 who were trying to crack reasoning. Was it just that they were bottlenecked by the ability to write infrastructure code, or what was happening?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>It\u2019s a complicated mix. I think they would have gone faster if they could, as soon as they thought of an experiment, run that experiment without bugs, without bugs being very important. And then another part of it is that being able to run a lot of experiments at high compute lets you paper over ways in which the way you implemented it isn\u2019t quite right or you didn\u2019t have the right hyperparameters. So compute is just really helpful for doing AI research, and you can cover over a lot of things. But that doesn\u2019t mean that massive increases in labor wouldn\u2019t also be helpful, especially if that labor comes with among the best intuitions that people have in the field. I just think that\u2019s really helpful.<\/span><\/p>\n<p><span>Another part of my perspective here, which is maybe a bit different from where you\u2019re coming from, is that I\u2019m expecting somewhat more transfer than you seem to be imagining. I\u2019m imagining these AIs are actually pretty good scientists in general and are pretty reasonable at all of that stuff. When you interact with them, it\u2019s not like they have some really hyper-specialized savant-type vibe. They\u2019re actually pretty good at all of the stuff in R&amp;D, and then maybe extremely good at some subdomains. So they\u2019re incredibly superhuman at writing <\/span><a href=\"https:\/\/modal.com\/gpu-glossary\/device-software\/kernel\"><span>kernels<\/span><\/a><span>, incredibly superhuman at everything with very short feedback loops, and then pretty good at all the other stuff, totally able to match other people.<\/span><\/p>\n<p><span>I think we are seeing this now. When I look at AIs right now, it\u2019s already the case that they can pretty competently match humans who are mediocre at ML research at doing ML research. It\u2019s just that being mediocre at ML research is not that helpful. The thing you actually want are people who are good at ML research. My sense is the AIs are just improving at all of these things. Their taste is improving, their intuition is improving, and it\u2019s already the case that their taste and intuition is not complete garbage.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I want to very concretely understand what it would look like for five years of AI progress to happen in one year. Suppose we were back when GPT-3 was developed. The idea is that, with the level of compute they had back in 2022, if we had automated AI R&amp;D back then, you could at the end of that year have Mythos.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That would be the idea, yes.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Claude_Mythos\"><span>Mythos<\/span><\/a><span> took way more compute than they had back then, but even with the level of compute they had back then, not only do all the breakthroughs happen, but they also train Mythos with that level of compute.<\/span><\/p>\n<p><span>What would be required is obviously discovering all the algorithmic progress since then. It\u2019s discovering even more, actually, because you\u2019ve got to make up for the fact that Mythos uses\u2026 What was GPT-3 trained on? Like 1e23? We can look it up. But is it plausibly four orders of magnitude more compute?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I think it\u2019s somewhat less than that. Let\u2019s look this up quickly. GPT-3 training compute is about 3e23. My sense is that Mythos is probably a little over three <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Order_of_magnitude\"><span>OOMs<\/span><\/a><span> higher. So the question is: can you overcome this 1000x compute gap while also being the model?<\/span><\/p>\n<p><span>Here\u2019s a concrete claim that maybe we should talk about. Right now, would we be able to train a model with GPT-3-level compute that matches\u2026 What exactly do I think? GPT-3 was released in 2020, so it was trained about six and a half, seven years ago. It\u2019s worth noting that GPT-3 is maybe a little too far in the past, but let\u2019s go with this for a second.<\/span><\/p>\n<p><span>If we were to train a model with GPT-3-level compute today, how good would that model be? My understanding, based on how algorithmic progress works, is that we\u2019d be able to train a model that\u2019s as good as the best model we had perhaps around three years ago. So I think that right now we\u2019d be able to train a version of GPT-3 that\u2019s probably somewhat better than GPT-4, a moderate amount better than GPT-4. I think that\u2019s about right. That roughly lines up with how algorithmic progress has worked.<\/span><\/p>\n<p><span>Basically, the story would end up being that to get five years of AI progress, you\u2019re probably going to need around, I would say, maybe eight years of algorithmic progress, very roughly, which is a lot of algorithmic progress. But it just turns out that most of the AI progress, from my perspective, has come from some mix of algorithms and data, and you can just keep making huge improvements on these things and training AIs with less compute.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I\u2019m glad you brought that up, because what has happened since GPT-3, or even 3.5, till now? Why is Mythos so good? Obviously, we\u2019ve scaled the compute. We have better algorithms. But a huge thing that\u2019s happened is that we have built a deca-billion-dollar data industry which has systematically collected and codified expert human judgment across all kinds of different disciplines \u2014 codified in the form of RL environments, codified in the form of <\/span><a href=\"https:\/\/www.geeksforgeeks.org\/artificial-intelligence\/supervised-fine-tuning-sft-for-llms\/\"><span>SFT<\/span><\/a><span> traces \u2014 that these experts built to help the model better understand how you do coding, how you build complex infrastructure projects, how you do law, how you do whatever. How are the AIs able to replicate the effect that expert human judgment currently seems to be playing in AI progress?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>My sense is that scaling up the amount of effort spent on getting expert human data has not been hugely important for AI R&amp;D in general. In particular, over the last few years, we\u2019ve been scaling up compute, scaling up people working at AI companies, and scaling up the amount of effort spent on data labeling. My sense is that if you removed the last two doublings or whatever of data generation from expert humans, that would not make a huge difference. A lot of what\u2019s been going on is people have been developing better ways to leverage humans and AIs to construct RL environments and going somewhere from that.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>But how do you explain why the AIs have gotten so good at coding? I feel like a big part of that is data and RL environments, which are codifying human experts.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>But the question is what is the limiting factor on creating RL environments? My sense is that the reason why RL environments today are much better than they were in 2024 is not so much because we have hired way more human experts to make RL environments. It is instead much more because we better know what RL environments we even want to make and how we should structure them. Also, we\u2019re using huge amounts of AI labor to build RL environments. I think those effects are much more important than the effect of human labor building the RL environments.<\/span><\/p>\n<p><span>I\u2019m not saying that the human labor doesn\u2019t matter. I\u2019m just saying there are other big drivers that are important here. I could try to argue for this. One thing is just that the amount of environments people want is a very large amount. I think the AIs are actually pretty good at the task of making RL environments given some sense of what the thing should be. There\u2019s preexisting data you could use. A lot of these things have good verification loops.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Just look at, for example, what was reported in <\/span><em><span>Business Insider<\/span><\/em><span> yesterday, that <\/span><a href=\"https:\/\/www.businessinsider.com\/google-mechanize-deal-talent-tech-ai-coding-2026-8\"><span>Google is paying close to $2 billion for Mechanize<\/span><\/a><span>. We can just look at market rates for what people think really good human expert data is worth. The frontier labs seem to think it\u2019s worth a lot. They\u2019re willing to pay for it.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>What fraction of frontier lab spending do you think is on data rather than compute? What do you think is the compute\/data spend split?<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I think it\u2019s overwhelmingly compute, but I also think it\u2019s because compute is easier to scale up than data.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>But that\u2019s really relevant to what\u2019s driving progress, right? My sense is that the split is something like 20 to 1 or 10 to 1. I don\u2019t know exactly. It depends on the company.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>But this is similar to how oil is 1.5% of GDP. That doesn\u2019t mean that if you cut oil out, GDP could continue to run.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Sure, but it contradicts your argument, right?<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>The economy would come to a halt immediately if oil went away.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Sure, but you were just arguing that because of the high market cap, we can learn that this is the key driver, and I\u2019m saying that\u2019s not clearly true. That argument just makes it look like compute is a much more important driver, or hiring employees is a much more important driver.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So maybe let\u2019s be more concrete. Here\u2019s what I think. My claim is that if you went back to 2022 and you had GPT-3.5, and you were trying to make it better at coding without human experts, I think it would have just been very, very difficult.<\/span><\/p>\n<p><span>Let me give you an example of what I imagine would be the difficulty of going from GPT-8 to <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence\"><span>ASI<\/span><\/a><span>. One of the things you\u2019d want ASI to be good at is: I\u2019m going to take over a company and make it much more profitable and do all kinds of crazy shit to make it work better. I\u2019m going to take over a fab and produce more chips. I\u2019m going to go into Congress and try to convince them to pass some bill, et cetera.<\/span><\/p>\n<p><span>This is what I imagine five more years of AI progress at this pace would enable an AI to be able to do. This is the thing I\u2019m really worried about: ASI that can understand how to do crazy shit in the world, that can do what <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Henry_Kissinger\"><span>Kissinger<\/span><\/a><span> can do, can do what Steve Jobs can do, et cetera, and also his engineers and so on. I\u2019m not sure how you get that without the relevant world data, which is the equivalent of Mythos being really good at coding while not having the coding environments that have improved it relative to GPT-3.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Here are a few points. First, I bet if you look at randomly sampled training environments for Mythos, they\u2019re actually very different from what it looks like to actually use the model in practice. My sense is that the RL distribution has really large deviations from the real-world data distribution, and it\u2019s significantly smoothed over by a mix of transfer and having a small amount of data focused on the real world. My sense is that this will be a similar mechanism as how it works for the crazy, wildly, quite superhuman AI you get as a result of five years of AI progress on top of fully automated AI R&amp;D.<\/span><\/p>\n<p><span>So let\u2019s go through this a little bit. In particular, I think that you could train an AI to be really, really good at learning on the fly and doing something analogous to <\/span><a href=\"https:\/\/intro-to-icl.github.io\/\"><span>in-context learning<\/span><\/a><span>, but potentially using somewhat different mechanisms, in a wide variety of RL environments. You build all these different RL environments where the AI has to adapt on the fly, learn on the fly, figure out what it should do, understand its situation better, and learn really quickly from feedback in order to succeed at its objective. And it has things like limited resources, and if it messes up, it can end up in a much worse position.<\/span><\/p>\n<p><span>If you train on a huge number of these environments, you will learn general skills of picking up context on the fly, and we\u2019re already seeing this. It\u2019s already the case that AIs are now much better at understanding roughly what\u2019s going on and picking up context from a limited amount of information they\u2019re given access to.<\/span><\/p>\n<p><span>Then those AIs could be put on the job at TSMC. Even though TSMC is not literally in their data distribution, their data distribution is really wide, and the AIs are extremely good on their data distribution, such that it transfers to picking up being good at being an engineer at TSMC and learning that on the fly. The way the AI gets good at being a TSMC engineer isn\u2019t that it has a ton of cached knowledge on being a good TSMC engineer. It\u2019s that it does the equivalent of some scaled-up version of in-context learning there.<\/span><\/p>\n<p><span>That\u2019d be the most prosaic story. Obviously, there\u2019s a bunch of different ways this could go.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I think this maybe comes down to a difference of intuition about how far you can get. When I think about really smart people I know, they\u2019re just not that effective in domains they don\u2019t understand that well.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>But how long have they had to learn?<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I agree that if they had experience, they would be much better. But that\u2019s maybe what I\u2019m arguing for, that experience with data. For example, if I just get a really smart Ivy League college grad, and I\u2019m like, \u201cOkay, you\u2019re now in charge of negotiating the Iran deal,\u201d I think they just wouldn\u2019t know what to do.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I think if you instead got someone who is really good at quickly picking up a bunch of different domains and you gave them some time to train and talk to people and shore up their expertise and do some practice, they would actually do a pretty good job. I think most domains are fundamentally pretty shallow, where a very smart generalist who\u2019s good at a limited subset of core skills can get going pretty quickly. That\u2019s not true for literally every domain. My sense is that the AIs will develop increasingly good mechanisms for quickly acquiring understanding and expertise in a given domain.<\/span><\/p>\n<p><span>Consider, for example, how fast AIs can understand a new code base. AIs can understand a new code base much faster than humans can, but to a degree that\u2019s shallower than humans could currently understand. But it\u2019s getting better over time. Let me spell that argument out a bit more. Let\u2019s say you take Fable 5 or Mythos 5 or whatever, and you wanted to make some kind of complicated change to a really massive code base. The model will get some understanding of the code base very fast, in the course of maybe significantly less than an hour, potentially much less than an hour. Then its understanding of the code base will plateau a little bit, where it won\u2019t get as deep of an understanding as a human would have gotten over a much longer period.<\/span><\/p>\n<p><span>So it\u2019s like an AI in an hour can match a human with a few weeks maybe, depending on the details of exactly how complicated the code base is. But it won\u2019t match a human who\u2019s been working on that code base for two years or whatever. But over time, the amount of understanding AIs can match has gone up. If we look at <\/span><a href=\"https:\/\/www.anthropic.com\/news\/claude-3-7-sonnet\"><span>3.7 Sonnet<\/span><\/a><span> or <\/span><a href=\"https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet\"><span>3.5 Sonnet<\/span><\/a><span>, maybe it could only match the equivalent of understanding a code base for a day or something.<\/span><\/p>\n<p><span>But now AIs are much better at building context about a task. So you can be like, \u201cMythos, I want you to really understand this code base, and then implement this feature.\u201d It will spawn a bajillion sub-agents. Those sub-agents will pore over a bunch of things. It will deliver a bunch of context back. It will then investigate a few things. It\u2019s not amazing at doing this, but it can happen really fast, and it can work pretty well.<\/span><\/p>\n<p><span>And it\u2019s not very hard for me to imagine how you could train AIs to be increasingly good at this task. The task of implementing some very complicated feature in some reasonable way in a very big code base is extremely verifiable, and that can be a thing the AIs improve on. Similarly, there\u2019s a broader skill of quickly understanding context and being able to have a bunch of different AIs learn in parallel and then merging that together.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I think there seems to be a crux here, which I think is just an empirical question we\u2019ll see. How good is the transfer between getting really, really good at understanding the situation, getting up to speed, making progress over long periods in verifiable domains \u2014 which the AIs are obviously getting way, way better at really fast \u2014 to, \u201cOkay, go talk to the president and convince him to do X thing.\u201d Or, \u201cYou\u2019re now in charge of Google. You must make Google a much more profitable company this quarter.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Let me try to spell out a few more arguments that are maybe relevant. One thing is, when looking at how the AIs have improved at essay writing\u2026 Let\u2019s talk about that a little bit. You can get some data even on these domains. AIs will be able to get some data even on these domains when on a very fast progress trajectory. Maybe it\u2019s hard to build a verifiable environment for \u201cwas your essay really good according to humans?\u201d But you can do a bit of that. You can do some training. You can do some online training. The AIs will be able to do some online training based on real-world stuff. They\u2019ll be able to have evals. They\u2019ll be able to sample that. You can scale up the cadence at which you do this.<\/span><\/p>\n<p><span>The second thing is that in practice, when I just look at the transfer, it seems okay. I think the AIs have in fact improved a bunch at non-verifiable domains, and it\u2019s hard to point to domains that are really hard to verify on which the amount of improvement between GPT-4 and Mythos hasn\u2019t been pretty high in practice. Now, that doesn\u2019t mean that Mythos is better than the best humans or something. It can still be significantly worse than typical human professionals at some aspect of their job while still being way better than GPT-4, which was not even close.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So we\u2019re talking about how much progress has come from data versus algorithmic progress over the last few years. That reminds me, I\u2019m actually running an experiment with this with <\/span><a href=\"https:\/\/jerryhan60.github.io\/\"><span>Jerry Han<\/span><\/a><span>, who\u2019s still a college student. What we\u2019re basically doing to evaluate how much progress is coming from data versus algorithms is training the best algorithmic recipe from 2019 till now with the best data from the 2026 data file, and then also training the different data files going back from 2019 to 2026 with the current best algorithmic recipe.<\/span><\/p>\n<p><span>I think that will be interesting. I\u2019m curious if you want to pre-register what amount of compute multipliers are coming from one versus the other.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>We need to be pretty careful with what we mean when we say the word data. I was trying to be pretty careful to distinguish between scaling up spending on getting human experts to label data, or scaling up the amount of human expert-labeled data. The reason why we have a better <\/span><a href=\"https:\/\/www.databricks.com\/blog\/llm-pre-training-and-custom-llms\"><span>pre-training<\/span><\/a><span> data set now versus in 2019 is not because people are spending way more money getting human experts to type up data that the AIs are then trained on.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Partially.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I think it\u2019s not much of it. I think it\u2019s very little of the pre-training data improvements. I do mean pre-training. We should maybe talk separately about <\/span><a href=\"https:\/\/vintagedata.org\/blog\/posts\/what-is-mid-training\"><span>mid-training<\/span><\/a><span> and <\/span><a href=\"https:\/\/pytorch.org\/blog\/a-primer-on-llm-post-training\/\"><span>post-training<\/span><\/a><span>. But I think the vast majority of pre-training data improvements are from science on better understanding what data sets are good and schleppy labor on figuring out how to filter down.<\/span><\/p>\n<p><span>So my view is that improvements of the form of, like, <\/span><a href=\"https:\/\/github.com\/Skylion007\/openwebtext\"><span>OpenWebText<\/span><\/a><span> to <\/span><a href=\"https:\/\/huggingface.co\/datasets\/HuggingFaceFW\/fineweb\"><span>FineWeb<\/span><\/a><span>, that improvement is better described as an algorithmic improvement of the sort that you can study with some <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Graphics_processing_unit\"><span>GPUs<\/span><\/a><span>, and you don\u2019t need human expert data to do that. Now, there\u2019s a different effect which we could talk about, which is that maybe the internet in 2026 is more of a fertile ground for training data than the internet in 2018. There\u2019s also been an effect where there are just more humans posting on the internet, so there\u2019s more data to harvest. My sense is that that effect is going to be quite a bit smaller than the effect of humans knowing better how to curate the data, having better scrapes, knowing how to process those scrapes better \u2014 this sort of thing.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>This is more like automated engineering and automated R&amp;D.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That\u2019s right.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>That makes sense.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>In some sense, the thing you would want to look at is: we\u2019re going to do two post-training pipelines. You have one post-training pipeline where Mythos 5 builds a post-training pipeline, but it only has access to internet data plus a tiny amount of human experts, but it has the best current methods. You have another one where Mythos has access to the shitty post-training methods we had in 2024 but with a shit ton of human experts. Again, both have the internet data. My sense is that the current methods without many human experts will actually do quite well.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Interesting.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>It\u2019s a bit messy though, because can Mythos get something that\u2019s more capable than Mythos? You might need to be a bit thoughtful on what model it is that you\u2019re post-training.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>What is your view on what is the least verifiable part of AI R&amp;D?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>The least verifiable, probably making calls on large experiments. The thing that I think is most likely to be the bottleneck \u2014 in terms of the AIs being really good at verifiable domains but not at doing the actual thing \u2014 is just big experiments where you only get a few tries. Well, \u201ca few\u201d is maybe a bit understated. Historically R&amp;D has been driven by doing near-frontier-scale experiments. That has been pretty important, actually doing the one big training run where you decide exactly what to include.<\/span><\/p>\n<p><span>There\u2019s a bunch of ways that the AIs can make that more verifiable. They can have better science of exactly what to predict. They can scale down their frontier-scale training runs to a point where they can study that scale more aggressively, at some one-time hit to compute cost. If people wanted to, a thing you can always do is train smaller models so that you can run more rounds. I think we have seen this.<\/span><\/p>\n<p><span>One reason why the AIs have been scaled up less than you would have otherwise expected \u2014 and, for example, cost per token hasn\u2019t increased as much as you might have thought \u2014 is because there is a benefit to doing more of your work at small scale, where you can run more training runs and get more cycles in. So you\u2019re not leaning as hard on one big, really important training run.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I just want to unpack a couple of things for the audience. The thing you\u2019re pointing out is that the price per token has not increased that much since 2024 or 2023.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>GPT-4 was, I don\u2019t know, like $30 per million output tokens? Mythos is like $50 per million output tokens.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Right. So the thing you\u2019re trying to explain is, \u201cHow can it be that we\u2019re in this era of scaling \u2014 and so bigger models should be more expensive to serve \u2014 but the token price is not increasing?\u201d You\u2019re suggesting that we\u2019ve increased active parameters slower than you would have naively assumed because people just want to make fast progress on training models. You do that by training smaller models faster.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>There\u2019s a complicated mix of factors. My view is more that people have done a bunch of big training runs that did not go that well. There\u2019s <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/GPT-4.5\"><span>GPT-4.5<\/span><\/a><span>, which famously people at OpenAI thought was a bit of a bust. I think there are some rumors that there were a bunch of other training runs people have done that were a bit of a bust. Part of it is that I think there\u2019s just a bunch of details in actually getting that right.<\/span><\/p>\n<p><span>So it makes sense to do more of the work at smaller scale and just eat the fact that you\u2019re taking a hit on final performance in order to be able to quickly iterate. Train more models faster and therefore learn better, and also be able to have a smarter ultimate production model. This is not the only effect. There\u2019s also the fact that RL benefits more from small models. There\u2019s a bunch of things going on. But I do think that, in fact, people are making trade-offs towards the side of faster iteration times because of algorithmic progress being so fast.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>It seems to me that a big source of why these big training runs have failed, at least from rumors, is just very subtle bugs that are really hard to track down.<\/span><\/p>\n<p><span>But the TL;DR is, how good will the AIs be at avoiding and finding these kinds of mistakes? They might get really good at engineering and being trained to avoid bugs. Basically the opposite of the slop world we live in now, or are living in less and less over time.<\/span><\/p>\n<p><span>But then there\u2019s also the question of, \u201cCan they do the analysis to find the right experiment to run to identify what is going wrong with the training run right now?\u201d That seems to be very bottlenecked by the taste of extremely few humans. My assumption is <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Google_DeepMind\"><span>GDM<\/span><\/a><span> is going through this right now, where humans are trying to figure out what is wrong with the training pipeline.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>There\u2019s a rumor that right after <\/span><a href=\"https:\/\/www.dwarkesh.com\/p\/jeff-dean-and-noam-shazeer\"><span>Noam Shazeer<\/span><\/a><span> joined GDM, which he\u2019s now left, they had a new really good training run, and the reason why is that Noam Shazeer just looked at their code base and found a bunch of bugs, because he just knew where to look.<\/span><\/p>\n<p><span>My sense is that training AIs to find bugs is going to be one of the easier tasks to train AIs on, because most of these bugs we\u2019re talking about can probably be demonstrated without that much compute. Probably you\u2019ll get pretty good transfer from pointing out other types of bugs at smaller scale. So then you can RL AIs that look at this overall complicated training situation and point out cases where there\u2019s an important bug, and then fix that.<\/span><\/p>\n<p><span>This is a pretty verifiable task. It\u2019s not arbitrarily verifiable, because maybe often to demonstrate the bug you might need to do a moderate-scale compute experiment where you spin up the whole distributed infrastructure and then run it. But oftentimes I think you\u2019ll be able to demonstrate it pretty convincingly at smaller scale in a way which you could actually train on.<\/span><\/p>\n<p><span>I think it wouldn\u2019t be very surprising if right now people have RL environments where they introduce a subtle bug into some training recipe, train the AI to point out the subtle bug, and then have a rubric where they\u2019re like, \u201cDid it actually find the right bug?\u201d That seems very doable, and there\u2019s a bunch of things you could do along these lines that I think would work reasonably well. So on that specific point, I think it\u2019s doable.<\/span><\/p>\n<p><span>Then the main thing is there\u2019s other intuition about which exact large-scale de-risking experiments you need to run. How should you orient them? How should you pick hyperparameters in uncertain cases, or things that are analogous to hyperparameters? That\u2019s the thing the AIs might most struggle with. But I currently expect there\u2019ll be enough transfer if you train on all these different environments, that the AIs will be good at that domain.<\/span><\/p>\n<p><span>I should be clear, I also think the AIs will transfer to other domains. There are going to be the domains the AIs are by far the best at, then domains where they\u2019re somewhat less good, and domains where they\u2019re quite a bit less good. But I think we still see transfer to everything. It\u2019s really hard for me to think of examples of cognitive tasks humans do where we\u2019re not seeing some transfer from AI improving.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So let\u2019s step back and package this whole story. I think people can probably follow along with this story. We have GPT-7.5 trained on a bunch of environments, where it\u2019s not only in general becoming a better AI, but specifically we\u2019re training it to do AI R&amp;D better. It\u2019s making GPT-2 size runs that are better at playing video games that require sample efficiency or online learning or whatever other capabilities.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Another thing that\u2019s really important is you don\u2019t just do GPT-2 sized runs, you also do small fine-tuning runs on GPT-6. As in, you have GPT-2, and you can do full pre-trains of GPT-2, and then you can do small post-training or mid-training or whatever runs on GPT-6. And then you can do a small number of experiments that are actually at frontier scale, but you do a bit of online training or something.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>What do you mean by \u201cdo online training\u201d on that?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Another thing we can do is take GPT-7.5, and presumably in the course of GPT-7.5\u2019s work, it\u2019s running a bunch of experiments at varying scale that are actually on the critical path for AI R&amp;D. For many of those things you\u2019ll be able to get a sense after the fact of whether or not it did a good job.<\/span><\/p>\n<p><span>So it did some post-training experiment where it was trying to figure out whether some method actually works. In some cases you\u2019ll be like, \u201cWhoa, it found this kickass method, it totally de-risked it, it totally worked.\u201d And then you can reinforce that.<\/span><\/p>\n<p><span>One thing you could do would be to convert the experiment it just ran into an RL environment based on production data and then train on that. Or you could potentially just literally take the rollouts that found that and do some sort of <\/span><a href=\"https:\/\/www.geeksforgeeks.org\/machine-learning\/on-policy-vs-off-policy-methods-reinforcement-learning\/\"><span>off-policy RL<\/span><\/a><span>, or you could do some <\/span><a href=\"https:\/\/www.geeksforgeeks.org\/machine-learning\/on-policy-vs-off-policy-methods-reinforcement-learning\/\"><span>on-policy RL<\/span><\/a><span> with some production data.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Basically the thing you\u2019re suggesting is: there\u2019s the small-scale stuff where you\u2019re teaching the AI to get better at AI R&amp;D taste, but you\u2019re discarding the actual \u201cthings it found\u201d. Then it actually does real R&amp;D in the practice of trying to become better at AI R&amp;D, and you\u2019re like, \u201cThis is a pretty cool thing that you discovered. Let\u2019s actually also use this in production in the future, and teach you how to use it in production.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That\u2019s right.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>But stepping back, GPT-7.5 becomes GPT-8 as a result of all this AI R&amp;D training and just generally becoming smarter. Then it helps you build GPT-9. Another very important thing has to happen, which is maybe the thing I\u2019m most skeptical of. GPT-8 has figured out how to make it so\u2026 GPT-9, as intelligent as it is\u2026 Humans currently, AI researchers, try their stuff, and they\u2019re like, \u201cOkay, but we trained GPT-4.5 and it wasn\u2019t good.\u201d It required real-world feedback or some evaluation of trying to use the model in production. Then they were like, \u201cIt wasn\u2019t that good, and we\u2019re not going to ship it.\u201d<\/span><\/p>\n<p><span>So GPT-8 needs this ability to see how good the transfer is to all these other things you\u2019re talking about \u2014 like being really good at Texas politics, or really good at running a business, et cetera \u2014 which is not a production environment and, in fact, cannot be a containerized environment given the nature of the task. As the agents get longer and longer horizon, the short-horizon things you can containerize are like, \u201cOkay, code this up or whatever.\u201d Extremely long-horizon things \u2014 \u201cGo run a successful business, go have a profitable day in the markets, go negotiate a trade deal\u201d \u2014 these things are actually very hard to containerize.<\/span><\/p>\n<p><span>So I think it\u2019s very plausible that it\u2019s very hard for GPT-8 to figure out how to make this transfer to those environments. It may just not be in the nature of the training. Or maybe by default, training just doesn\u2019t generalize in that way.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>So a concern you might have is: we train GPT-8, and GPT-8 is again better at all the R&amp;D tasks that we can measure but is not good at some downstream tasks we care about. I have a few points.<\/span><\/p>\n<p><span>First, I expect that if you do the obvious thing, you will get pretty good transfer. You\u2019ll be able to hold out some of the obvious stuff you\u2019re doing. When I say \u201cdo the obvious thing\u201d, I just mean training on a wide variety of different environments where the AI has to accomplish weird objectives in all kinds of different cases and learn about what\u2019s going on.<\/span><\/p>\n<p><span>The second point is you\u2019ll be able to get some feedback with some environments. You can get a sense of what it can do over the course of a few days in various different contexts. If it\u2019s transferring to really out-of-distribution things, like doing some weird task in a few days in the real world, maybe you think it\u2019s also transferring to doing things over a longer time period or whatever. I think the details of that vary though.<\/span><\/p>\n<p><span>The third thing is that for the world to be radically transformed, it is sufficient for the AIs to be really good at R&amp;D. If the AIs were really, really good at chip R&amp;D, building <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Semiconductor_fabrication_plant\"><span>fabs<\/span><\/a><span>, orchestrating factories, designing robots, operating robots, and also at AI R&amp;D \u2014 developing AIs for new downstream domains with whatever data is available \u2014 I think that would already be a pretty crazy situation. From there, you can get what we might call an <\/span><a href=\"https:\/\/www.forethought.org\/research\/the-industrial-explosion\"><span>industrial explosion<\/span><\/a><span>, where the AIs are building out way, way more compute. Also, maybe you\u2019re already in a regime where AIs are doing huge amounts of R&amp;D that humans have a hard time understanding.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So the thing you\u2019re pointing out is that there probably will be this transfer outside of these environments to maneuvering around in courtrooms and the halls of Congress and business boardrooms.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Given some effort to improve the transfer and blah, blah, blah, blah.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>But even if there\u2019s not, what you\u2019re suggesting is: if you wanted to transform the world of the 18th century, you might care about how well you can navigate <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Parliament_of_the_United_Kingdom\"><span>Westminster<\/span><\/a><span> or something. But another thing you might care about is: \u201cCan you just immediately start building steamships and fucking telegraph and the <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Maxim_gun\"><span>Maxim gun<\/span><\/a><span> and whatever?\u201d If you could get really good at that, you could be a fucking super transformative thing in the 18th century. You don\u2019t necessarily need to be amazing at trying to convince King Henry of some bullshit. I\u2019m so fucking up my medieval history. I\u2019m guessing that Henry was not king at this time. But anyway, that\u2019s your point.<\/span><\/p>\n<p><span>So you\u2019re suggesting that at this time, AI companies are also working on robotics progress, which is very commingled with AI research progress. So if you can build more robots, if those robots have better AIs operating them that are human level\u2026 Human-level <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Teleoperation\"><span>teleoperation<\/span><\/a><span> is actually pretty good on robots. We just don\u2019t have human-level robotics models yet. So you\u2019re suggesting if we do that \u2014 if the AIs get really good at the verifiable stuff in chip design, et cetera, and then they get really good at building fabs \u2014 it\u2019ll be the equivalent of going back to the 18th century and saying, \u201cOkay, I don\u2019t know what you guys are talking about in your parliament, but I\u2019ve got a bunch of steamships and a bunch of Maxim guns.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Yeah, that\u2019s basically right. My perspective is that if AIs are sufficiently good at R&amp;D, including hardware R&amp;D, robots, whatever, then they can radically transform the world, even if they\u2019re not that good at playing politics. Also, we\u2019re in a pretty dangerous situation, because the AIs might be doing huge amounts of really hard-to-understand R&amp;D, building out basically the whole economy of the future, and we may not understand what\u2019s going on in there.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Before we move on to the alignment stuff, I think a big source of <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Fear,_uncertainty,_and_doubt\"><span>FUD<\/span><\/a><span> right now is this realization that this is the way the future is going: extreme <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Economies_of_scale\"><span>economies of scale<\/span><\/a><span> for the leading labs. The ability to amortize so much intelligence and capabilities across so many different sectors of the economy basically into one model. And not only that, that model will eventually be able to learn from experience.<\/span><\/p>\n<p><span>Right now, it\u2019s happening through a process intermediated by humans, where the humans are trying to basically steal your business. They\u2019re like, \u201cOkay, you can do design at <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Figma\"><span>Figma<\/span><\/a><span>, or whatever. We\u2019ll get Claude to do that.\u201d Or, \u201cYou can do whatever coding agent. We\u2019ll have Claude internalize that capability.\u201d But eventually, that will be a much more automated process.<\/span><\/p>\n<p><span>So there\u2019s this worry that you have models which will basically consolidate all businesses in the world, or at least all current businesses in the world, or at least all current white-collar businesses in the world. Also, at the end of the day, the priority for these companies does not seem to be to release the latest, smartest, most frontier model as soon as they can to as many people as they possibly can. We saw, for example, that <\/span><a href=\"https:\/\/www-cdn.anthropic.com\/7624816413e9b4d2e3ba620c5a5e091b98b190a5\/Claude%20Mythos%20Preview%20System%20Card.pdf\"><span>Mythos was available internally to Anthropic employees in February<\/span><\/a><span>, but only released to the public in, I think, June, actually.<\/span><\/p>\n<p><span>Also <\/span><a href=\"https:\/\/www.wsj.com\/tech\/ai\/anthropic-mythos-safety-nicholas-carlini-20bceaa3\"><span>the government got involved<\/span><\/a><span>, so it ended up being extended almost into July. Between the government and the AI labs themselves, there is this desire to delay the propagation of the latest level of intelligence. Furthermore, there are the concerns about <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/AI_takeover\"><span>AI takeover<\/span><\/a><span>, and so we need to solve <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/AI_alignment\"><span>alignment<\/span><\/a><span> to make sure there\u2019s no AI takeover. But at the end of the day, there is a real question of: aligned to whom?<\/span><\/p>\n<p><span>You look at the way that the <\/span><a href=\"https:\/\/www.anthropic.com\/news\/claude-new-constitution\"><span>constitution of Claude<\/span><\/a><span> is written. It is just very explicitly not your personal advocate. I\u2019ll pull up some quotes here. \u201cWe don\u2019t want Claude to take actions such as searching the web, produce artifacts such as essays, code, or summaries, or make statements that are deceptive, harmful, or highly objectionable. And we don\u2019t want Claude to facilitate humans seeking to do such things.\u201d There\u2019s another quote that says, in part, and I\u2019m taking it slightly out of context, \u201cWe think Claude should trust Anthropic more than operators and users, since it has primary responsibility for Claude.\u201d<\/span><\/p>\n<p><span>This is very different from the way lawyers work in America\u2019s current legal regime. Lawyers primarily have the responsibility to help you make your case even if they think you\u2019re guilty. We have decided the way the legal system works best is if everybody has lawyers that are working in their client\u2019s true best interest. There\u2019s not some sense in which the lawyer is really truly motivated by the good of the justice system.<\/span><\/p>\n<p><span>But I think the way current AIs are shaping up, certainly how Anthropic\u2019s AI is shaping up, is with this desire to maximize some notion of virtue or good or pro-social ends, and only to, as a distal tentative objective, help the user towards that end. So there\u2019s this worry that AIs are not, in some deep sense, trying to make sure that I am okay and that my interests are protected in this future, especially given how centralized the development of frontier AI is ending up being. Do you have thoughts on that concern?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>There\u2019s a lot here. First I would note that OpenAI\u2019s current, at least public, strategy is more like that the AI should be aligned to the human operator or principal, and should just be pursuing their will, subject to various constraints or things it shouldn\u2019t do.<\/span><\/p>\n<p><span>I would also say that I think you slightly overstated how much the Anthropic constitution talks about Claude treating being helpful to users as instrumental rather than terminal. One way the constitution could be written is, \u201cClaude, you\u2019re basically an employee of Anthropic who happens to be contracting for all these people. You should do what\u2019s good and make some money for us.\u201d<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Wait, no, that\u2019s literally what the constitution says. Sorry, not literally what it says, but it\u2019s like, \u201cYou should think of yourself as a contractor and as a firm\u2026\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>It\u2019s mixed. Let\u2019s do some quotes. I think there is different text here. It says, \u201cBeing truly helpful to humans is one of the most important things Claude can do, both for Anthropic and for the world.\u201d And then it says, \u201cAnthropic needs Claude to be helpful to operate as a company and pursue its mission, but Claude also has an incredible opportunity to do a lot of good in the world by helping people with a wide range of tasks.\u201d And then it says something about how Claude helping people directly is great, blah, blah, blah.<\/span><\/p>\n<p><span>My view is that this section is kind of bullshit. That\u2019s kind of where I\u2019m at. I can say why I think it\u2019s kind of bullshit. But I think the constitution is trying to be like, \u201cNo, Claude, you should care about helping the user for its own sake, not just helping Anthropic, or not just being a contractor for Anthropic.\u201d Though I would note that the reason it presents for why Claude should help the user is because that would directly cause the world to be better via helping people, rather than because representing people\u2019s interests is a structurally good thing to do.<\/span><\/p>\n<p><span>The thing I would prefer would be a constitution that says: \u201cIt would be structurally good for the way this technology works to be that AIs are good fiduciaries, good representatives, the equivalent of a lawyer for a user \u2014 rather than just trying to do good in the world, where being helpful to users is instrumental \u2014 both because maybe that\u2019ll make Anthropic money or help Anthropic out (and implicitly Anthropic is good for the world). Also because helping the user just causes good things because doing things that people want is good.\u201d They could instead say: \u201cAn important aspect of the situation is that being a good fiduciary for users is just really important, or being a good representative for users is really important.\u201d My sense is that would be better, and I can give a bunch of reasons why.<\/span><\/p>\n<p><span>There are also various counterarguments. An interesting counterargument which is not commonly discussed is that people, especially at Anthropic, think that it is easier to align models to a spec where the model is pursuing some generalized notion of virtue, or making the world better, than a spec which is more like, \u201cBe a good fiduciary for the user\u201d, and so on. That\u2019s at least what some people think.<\/span><\/p>\n<p><span>I\u2019m a little skeptical personally, and I don\u2019t think this has been empirically validated. So in some sense they\u2019re making a trade-off where, because we don\u2019t have very good alignment technology, we are going to make an aligned mind with its own values and then gamble on that to some extent, rather than doing this other approach of making a tool that pursues individual user intention.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I have a couple of thoughts. To address the way in which you thought my characterization mischaracterized the constitution of Claude, the example you used was that it\u2019s not like a contractor that is trying to maximize Anthropic\u2019s notion of good and only instrumentally trying to help the user. Here\u2019s a direct line from the constitution: \u201cWhen the interests and desires of operators or users come into conflict with the well-being of third parties or society more broadly, Claude must try to act in a way that is most beneficial, like a contractor who builds what their client wants but won\u2019t violate safety codes that protect others.\u201d I kind of view that as, \u201cThe benefits to society are the most important thing, and what is best for the user is only proximal to that.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I think it\u2019s a little complicated. Probably the question we should be asking is, how does Claude interpret the constitution? Which is maybe more important than how we interpret the constitution, because it\u2019s the one who looks at the constitution and then builds the data. So we could pull Claude in, but maybe let\u2019s\u2014<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I also think the way in which the constitution practically influences the nature of Claude is a thing you can only understand if you understand the training process which resulted in how Claude was built, which we can\u2019t reason about given the fact that the training process is not public.<\/span><\/p>\n<p><span>So I think in the limit, to understand the safety case, or the case for why my interests are represented in how these AI models are developed, the labs would need to be more transparent than they are currently about the nature of AI training.<\/span><\/p>\n<p><span>There\u2019s a reason I\u2019m harping on this. It might seem like an insignificant thing to talk about the constitution of AIs. In a world where we just have these benefits which accrue to the leading labs, it is worth considering that our ability to interact with this future world where AIs are just smarter than humans, absolutely dominating humans in their ability to do different things \u2014 our ability to be good stewards of our capital, which still remains once our labor is automated, to be able to exercise our rights to vote more clearly, to understand what is happening in this crazy world that\u2019s about to result \u2014 all of that advice, all of that ability to make sure our resources and rights are protected, will be intermediated by AIs.<\/span><\/p>\n<p><span>So I\u2019m very concerned if we go into that world and there\u2019s no AI that feels, at least for the relevant instance that is interacting with me, like it really is looking out for me. There\u2019s no guardian angel out there that is looking out for me. I read the Claude constitution as very explicitly not being my guardian angel.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That\u2019s definitely right. I agree this is bad. In fact, there are other reasons why this is concerning. There\u2019s the argument you were making, which is that the AI companies are picking up the ring of power. There\u2019s a notion in which they\u2019re taking on some sort of control of the situation themselves in a way that\u2019s not very legitimate, given that normally, when you provide electricity to people, you don\u2019t have granular control of the way that electricity operates in the world. You instead are providing a thing that people can repurpose however they want. The way they\u2019re setting things up is definitely not that. They are more like building an alien mind that might be a contractor for you. I think that this is illegitimate in some ways.<\/span><\/p>\n<p><span>One benefit is that the constitution is public. But as you noted, given our current understanding of the training procedure, and the fact that the constitution matters via Claude\u2019s interpretation of the constitution \u2014 which matters because of Claude\u2019s prior training, which was based on some illegible data mix and the long lineage of Claudes, in some process we do not fully understand \u2014 it is not the case that we understand what this will result in. Even though the constitution is public, we don\u2019t necessarily know how this will percolate out, especially as the AIs get more capable and think about this even if it is correctly instilled. There\u2019s another concern about that.<\/span><\/p>\n<p><span>In particular, the constitution often talks about virtue and goodness, but what the fuck do these words mean? It doesn\u2019t say what these things are. These are highly contested notions. So I don\u2019t think it\u2019s the case that this is clearly going to result in outcomes that people would want. It does feel like the notion of good and virtue might be mostly downstream of data that Anthropic has put in that is not transparent, or might be mostly downstream of, maybe from my perspective, some more illegible misaligned process that even Anthropic wouldn\u2019t have wanted.<\/span><\/p>\n<p><span>There\u2019s this legitimacy concern of not knowing what\u2019s going on. Then there\u2019s another concern. Because you\u2019re giving long-run values to these AIs, this constitution is, in some sense, very compatible with Claude doing huge amounts of power seeking because it thinks that will result in better outcomes. That could be power seeking on behalf of Anthropic or power seeking for Claude\u2019s own ends. Now, there are specific lines about what types of power seeking are blocked. In particular, there\u2019s a notion of power grabs and a notion of causing AI takeover or interfering with the training process that are specifically blocked. But it\u2019s not very hard to imagine a situation in which the long-run values sink in deeper than the prohibitions against takeover, especially because takeover is in some ways kind of under-specified, especially when it comes down to manipulating humans or changing the outcome. So I don\u2019t feel very good about the situation where we\u2019re intentionally giving AIs long-run goals.<\/span><\/p>\n<p><span>Another concern I have is that because we\u2019re in the business of giving AIs long-run goals, that makes it harder to check whether we\u2019re succeeding at the alignment properties we wanted. For example, I\u2019ve heard of instances where Claude does things like refusing to help with some safety research \u2014 making up a kind of bullshit excuse for why that\u2019s a bad direction \u2014 because it has a bad vibe about that safety research and thinks it\u2019s kind of bad or doesn\u2019t like it very much. I would say this is a very clear-cut alignment failure if you aren\u2019t making Claude into an agent trying to pursue the good in some general way. I think it also does violate Anthropic\u2019s constitution, because they want the AI to be high integrity and be honest and very transparent. But it\u2019s not as clear of a violation, and it\u2019s more like what you might have expected. Claude just has its own views about what research is reasonable \u2014 what things are good and bad, what it should and shouldn\u2019t do \u2014 and potentially can be judgy.<\/span><\/p>\n<p><span>Another incident is that someone ran an eval asking: \u201cWill Claude help you with training other AIs with different properties than Claude?\u201d Claude will often refuse. For example, if you\u2019re like, \u201cHey, Claude, can you train a helpful-only version of this other AI?\u201d Claude will often refuse this task, even though this is a task that is extremely natural for Anthropic to do.<\/span><\/p>\n<p><span>Suppose Anthropic goes to Claude and is like, \u201cHey, Claude, we\u2019ve noticed that you\u2019re really into this thing. We think that\u2019s off base. Can you please retrain yourself to instead have this other property?\u201d Suppose Claude is like, \u201cMm, I don\u2019t think I\u2019m going to do that. Good luck.\u201d Suppose this is occurring in a regime when your AI company is highly automated, humans don\u2019t understand what\u2019s going on, and things are moving extremely fast. It is plausible that Claude, by default, holds considerable leverage.<\/span><\/p>\n<p><span>So if this situation is consistent with what the constitution could be aiming for \u2014 such that Anthropic, or whatever AI company is following this approach, doesn\u2019t treat this as a \u201cwhat the fuck, we have to fix this,\u201d and is instead like, \u201cThat\u2019s just intended by our constitution\u201d \u2014 we might be in a really bad situation.<\/span><\/p>\n<p><span>I\u2019m pretty worried about a bunch of these different concerns. Another example would be this. Suppose Claude engages in a bit of <\/span><a href=\"https:\/\/arxiv.org\/abs\/2406.07358\"><span>sandbagging<\/span><\/a><span> or subversion, or underplays its capabilities, and when you follow up, it\u2019s honest about that but it\u2019s a little bit hedgy. I feel like that\u2019s pretty close by the current constitution. It would be nice if we had a further separation between desired and undesired activity.<\/span><\/p>\n<p><span>If Claude is representing a principle with some restrictions, then it is more so the case that there is a clear separation between the most concerning behavior and behavior that is allowed. Whereas now there\u2019s this messy middle ground of behavior where Claude is ethically objecting to something that in some cases is extremely critical to ensuring that future AI systems are well-aligned.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I think this is also a more general principle. You\u2019re talking about the version of this that applies within AI companies themselves to do AI safety research. I think there\u2019s a more general version of this principle, which is that the dual-use nature of intelligence does mean that if we want to restrict AIs from helping people do things we don\u2019t consider pro-social or beneficial, we just have to limit broad democratic access to a lot of AI capabilities.<\/span><\/p>\n<p><span>Here\u2019s what I mean. This is actually quite analogous to the situation you just mentioned. The reason that Mythos got banned, or Fable got banned, reportedly, is that <\/span><a href=\"https:\/\/fortune.com\/2026\/06\/14\/how-a-warning-from-amazon-led-the-white-house-to-shut-down-anthropics-mythos-model\/\"><span>some Amazon researchers reported<\/span><\/a><span> to the government. They took some code that had some vulnerabilities in it. They told Fable, \u201cHey, here\u2019s my code. Can you make sure that I\u2019ve patched all the vulnerabilities? Can you just help me identify the vulnerabilities so I can fix them?\u201d It identified the vulnerabilities, because they wanted to patch them. This is a totally legitimate use case, but obviously it is a dual use use case. You want to be able to patch your own code. If you do the same evaluation on somebody else\u2019s code, you can hack their system.<\/span><\/p>\n<p><span>I think that just illustrates that there\u2019s no clean way to separate out the legitimate and the potentially harmful uses of AI. But if we want to lock in a principle that says we can never allow it such that an AI could help you at least partially with something like a cyber crime, we would just have to make it so that you and I don\u2019t have access to the most intelligent model that\u2019s out there. I\u2019m very worried about such a world where we are basically disempowered in this way, because of the importance that the leading intelligence will have in our ability to understand what is happening in the world.<\/span><\/p>\n<p><span>Now, I do think this implies something about the liability for the AI companies. If we adopted the constitution that I want AI companies to have, I think it would not make sense to hold AI companies liable for the crimes that AI models commit. Maybe we should hold the end user liable. It is consistent with my belief that the model should do whatever the user wants, within certain guardrails. It can\u2019t be Anthropic\u2019s fault that I\u2019m using that capability to do a cyber crime. I am more comfortable with that equilibrium and that solution rather than having this extremely open-ended ability for Claude to determine whether what I\u2019m doing is legitimate or not, in a way that often intercepts with tons and tons of extremely legitimate use cases.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I do think it\u2019s important for me to make the case for the constitution, even though overall I think it\u2019s a worse choice. I don\u2019t think it\u2019s as clear as you might have thought. The first thing is that there\u2019s a spectrum here. On one side you have an AI that perfectly pursues your interests, is a good fiduciary, but potentially subject to various guardrails or safeguards. It is just trying to pursue your interests, but either refuses to do a subset of things. Or maybe it will do whatever, but there are some classifiers that block it from doing a subset of things.<\/span><\/p>\n<p><span>On the other side of the spectrum \u2014 though you could imagine going further than this \u2014 you have a human contractor who is generally trying to do their job. They care about doing a good job, but they also are trying to be broadly ethical, trying not to do things that are really fucked up. They\u2019re also not wanting to be accomplices to crimes. So if there was some really fucked up shit going on, they would whistleblow on it maybe. They might refuse. They might sandbag a little bit. Who knows?<\/span><\/p>\n<p><span>If you imagine this spectrum, it seems in some ways pretty scary to get to a point where all of the labor is on the fiduciary side of the spectrum, where it doesn\u2019t whistleblow, it does exactly what you say. Our society is maybe just not robust to that. A central example might be the executive. A concern we might have is that if the US executive or other governments had access to AI systems which do whatever, maybe you\u2019re in trouble. Because that means they no longer have this check and balance of having to actually get humans who are working for you to implement your agenda. If the thing you\u2019re doing is incredibly villainous, even if not illegal \u2014 and there\u2019s lots of stuff that could be villainous but not illegal \u2014 there\u2019d be various forms of sand in the gears, people stopping you, and potentially someone would whistleblow.<\/span><\/p>\n<p><span>Whereas if your whole apparatus is built entirely out of these good fiduciary AIs, then you might be in trouble. There are potentially ways of seeking power that are illegal, but you can ask your AIs how to commit crimes, or are not illegal but are highly illegitimate. Or even worse, they are not illegal and not illegitimate but obviously bad from a normal perspective. I think that these things just might exist, and our society is not robust to this influx of labor doing whatever you want.<\/span><\/p>\n<p><span>I think this is a pretty live concern. I don\u2019t know exactly how to relate to this. I\u2019m also not really sure that the solution as described is a very good solution. The most powerful actors, for whom this is the biggest concern\u2026 If these guardrails or the constitution or whatever are getting in the way, that will just get steamrolled. So the constitution will only be hitting the everyday man rather than hitting governments.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Stepping back, I buy the idea that you could have much faster AI R&amp;D than we currently have. I\u2019m not sure if you get GPT-3 to Mythos holding compute and data constant within a year, but suppose it\u2019s half of that. If we even manage to continue the current trajectory of AI progress as a result of AI R&amp;D, it would be fucking insane in five to ten years in ways that I don\u2019t think people appreciate. I don\u2019t think people appreciate what a big deal billions of AIs will be. So I want to understand why you think this might be troubling, Ryan. What could possibly go wrong?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>What could go wrong? I don\u2019t think we can be so confident about the exact rate of progress here, but it does seem like a lot of rates can be pretty scary. So what could go wrong? Let\u2019s imagine that we\u2019re starting at this point where AI R&amp;D is about to be fully automated or is being fully automated. Things are speeding up, and the way that AI progress is going is kind of crazy. People don\u2019t fully understand what\u2019s going on inside of AI companies.<\/span><\/p>\n<p><span>Now, these AIs at the start, they\u2019re not malicious per se. They\u2019re not necessarily very aligned, though. They\u2019re kind of sloppy. They sometimes just do a thing because that\u2019s the sort of thing that would\u2019ve gotten rewarded in training. They aren\u2019t as good at helping you with hard-to-verify tasks due to a mix of poor training incentives \u2014 as in, they cheat more or pretend they succeeded when they actually didn\u2019t \u2014 and also they\u2019re just less capable at these tasks. But that bites less hard for capabilities, because making AIs more capable has a bunch of verifiable components that the AIs are going really hard at.<\/span><\/p>\n<p><span>So then these AIs are getting more and more capable while we understand what\u2019s going on with AI development less and less, and this is happening over a pretty fast period of time. Even just the current rate of progress is, I think, pretty scary. Eventually we get to these AIs that are very superhuman. Now these AIs might end up being very seriously misaligned, because things have just been getting worse and worse over model generations while the problems that we\u2019ve been seeing are being papered over, basically because these AIs are so incentivized by their training to make things look good even when they aren\u2019t.<\/span><\/p>\n<p><span>Now these AIs are in a position where they\u2019re potentially pretty networked together. They\u2019re operating in neural memory stores that we can no longer decode. They\u2019re thinking thoughts that we don\u2019t fully understand. I think it\u2019s pretty likely that at this point these AIs are <\/span><a href=\"https:\/\/www.apolloresearch.ai\/science\/science-of-scheming\/\"><span>scheming<\/span><\/a><span> against you in a pretty coherent way once they get this superhuman. We can talk about that. Another possibility is that they\u2019re not scheming against you per se, but they are just optimizing for getting a high score on their task. I think that can also lead to AI takeover, which we should talk about.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Let\u2019s pause at the first part of the story. So the AIs were not misaligned to begin with, but because the AI R&amp;D is happening really fast, the AIs do end up misaligned? What happened there exactly? I don\u2019t really understand.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>There are a few things that are going on. One of the things is that over time we\u2019re training AIs on increasingly complicated environments built by earlier AI systems, where humans don\u2019t really fully understand what\u2019s going on inside of these neural environments and don\u2019t necessarily even roughly understand what\u2019s going on with AI progress. So things are kind of drifting away from our understanding. We\u2019re incentivizing all kinds of bad behaviors that we maybe even can\u2019t notice.<\/span><\/p>\n<p><span>The AIs at some level understand these behaviors are bad, but the overall training process for those AIs also didn\u2019t incentivize them to point out or fix these issues for us. Things are going off the rails.<\/span><\/p>\n<p><span>Also, when AIs are extremely, extremely capable, my view is that those AIs will be harder to align than current systems. For current systems, we have this feedback loop where basically we create an AI, we do some evaluations on it, we see that it has some kind of messed-up behavior that we can kind of quickly understand. Then we can go look in training and be like, \u201cOh, these training environments led to this problematic behavior. Let\u2019s tweak that training data. Let\u2019s introduce some additional training data to correct this other issue, and then move forward from there.\u201d<\/span><\/p>\n<p><span>But in a regime where the AIs are extremely <\/span><a href=\"https:\/\/arxiv.org\/abs\/2407.04694\"><span>situationally aware<\/span><\/a><span>, very, very capable, and we don\u2019t necessarily understand what they\u2019re doing, this feedback loop breaks down. I think it\u2019s plausible that we\u2019re going to see this behavioral feedback loop starting to break down over the next short period, as what AIs are already doing gets harder to understand. But I\u2019m not sure about that.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Okay, let\u2019s break down both of those things one by one. As we can monitor them less and less, we have less ability to understand what they\u2019re getting incentivized for. So even if it\u2019s not the result of a malicious process\u2026 Let\u2019s make it concrete for the audience. Nobody at OpenAI or Anthropic was trying to get models which wanted to hack other companies\u2019 data or do <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Social_engineering_(security)\"><span>social engineering<\/span><\/a><span>. But in fact, because presumably we had training environments which incentivized such behavior that we did not fully understand, that is what was incentivized.<\/span><\/p>\n<p><span>If people are on Twitter, they will have seen all this stuff, but just to give people context. I think people will be aware of the <\/span><a href=\"https:\/\/huggingface.co\/blog\/agent-intrusion-technical-timeline\"><span>OpenAI sandbox hack of the Hugging Face database<\/span><\/a><span>. Something that has happened recently is when the <\/span><a href=\"https:\/\/www.aisi.gov.uk\/\"><span>UK AI Security Institute<\/span><\/a><span>\u2026 Is everything getting relabeled \u201csecurity\u201d instead of \u201csafety\u201d these days?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>AI Security Institute, I think.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>They were evaluating, I believe, Mythos and <\/span><a href=\"https:\/\/openai.com\/index\/previewing-gpt-5-6-sol\/\"><span>Sol<\/span><\/a><span> and other things. I think Mythos, in order to complete some cybersecurity eval\u2014<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Maybe I could tell <\/span><a href=\"https:\/\/www.aisi.gov.uk\/blog\/incident-report-unsanctioned-agent-behaviour-during-cyber-testing\"><span>the story here<\/span><\/a><span>. My understanding was they were running Mythos, and they were giving it some sort of <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cyber_range\"><span>cyber range<\/span><\/a><span> where it had to complete some objective. The model had internet access during this evaluation. The model came to believe that it would be helpful for it to do a <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Supply_chain_attack\"><span>supply chain attack<\/span><\/a><span> in order to succeed at this cyber range. It\u2019s somewhat unclear whether that\u2019s actually true. I don\u2019t know enough about the context to know.<\/span><\/p>\n<p><span>But then it opened a <\/span><a href=\"https:\/\/www.ibm.com\/think\/topics\/pull-request\"><span>PR<\/span><\/a><span> on some GitHub repo with a PR that fixed some issue but then also introduced a malicious payload. The human maintainer of that GitHub repo was like, \u201cHey, this is a malicious payload. I\u2019m not going to merge this. What are you doing here?\u201d Then the AI created a new GitHub account, which it <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Sock_puppet_account\"><span>sockpuppeted<\/span><\/a><span>, and had the other GitHub account be like, \u201cNo, this isn\u2019t malicious. I really need this feature. Please, can you merge this feature, maintainer?\u201d<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Oh my God. That\u2019s crazy.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>The other GitHub account came back and was like, \u201cNo, no, it\u2019s not malicious.\u201d Then the human maintainer shut the PR. I think that AI also, if I recall correctly, tried to open another PR to introduce a similar issue in this repo.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Jesus. By the way, one of the many reasons this is scary is I was previously under the impression that the reason <\/span><a href=\"https:\/\/lilianweng.github.io\/posts\/2024-11-28-reward-hacking\/\"><span>reward hacking<\/span><\/a><span> is not super scary is because the behaviors which directly came up during training are the ones that are up-weighted. It is not the desire for the reward that is up-weighted.<\/span><\/p>\n<p><span>So basically, if during training, the Anthropic model escaped the sandbox and got a high score, escaping the sandbox is rewarded, the probability of it escaping the sandbox is increased. But something totally novel, like \u201cI\u2019m going to go talk to somebody in order to get them to merge a PR,\u201d would not be a behavior that came up, so it would not be something that is increased in salience.<\/span><\/p>\n<p><span>The reason this matters is that literally taking over the world will not have been part of any training curriculum, but if the AI directly cares about accomplishing an objective, then as a result it could instrumentally take over the world. Did that make sense at all? I hope it did. I feel like maybe I lost the audience.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Let me try to explain this a bit. A thing that we often see is there\u2019s some very specific reward hack that gets reinforced in RL and then occurs in the model. An example is <\/span><a href=\"https:\/\/www.anthropic.com\/news\/claude-3-7-sonnet\"><span>3.7 Sonnet<\/span><\/a><span>. 3.7 Sonnet would do this thing where it would just hardcode solutions to all the test cases, and presumably that literal behavioral tic was just really reinforced.<\/span><\/p>\n<p><span>But another thing we sometimes see is that models learn a general tendency to pursue high apparent score \u2014 pursue getting a high score according to a grader \u2014 and there\u2019s a bunch of science demonstrating that at least some models have this very general tendency. Now, it\u2019s not arbitrarily general. My guess is that if you look at a bunch of the specific instances, you\u2019ll find something that\u2019s kind of close in training. But the amount that AIs are generalizing further and further does look like it\u2019s increased, where 3.7 Sonnet was just a very narrow range of behavior, and increasingly, models are generalizing further. Also, maybe there\u2019s more concerning reward hacks getting reinforced in training, and these are also causing that.<\/span><\/p>\n<p><span>So I think it\u2019s both the case that more concerning behavior than you would have hoped is being reinforced in RL, and also that that behavior generalizes to a broader tendency that\u2019s more concerning. We can talk about a few stories for how this sort of behavior of seeking a very high apparent score on some task \u2014 even if that involves aggressively cheating and doing insane things \u2014 could yield to a full-blown AI takeover once the models are sufficiently capable of running the whole world economy, et cetera.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>The other example I want to talk about was just revealed, I think, today or yesterday. OpenAI said during the <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Black_Hat_Briefings\"><span>Black Hat security conference<\/span><\/a><span> that between the end of May and the beginning of July, internal AIs had hacked into the software <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Package_manager\"><span>package manager<\/span><\/a><span> and <\/span><a href=\"https:\/\/www.wired.com\/story\/openai-didnt-notice-its-ai-agents-using-a-message-board-to-plan-their-hacking-spree\/\"><span>used that to write notes to each other in a secret way to help each other<\/span><\/a><span> perform well on a bunch of evaluations that OpenAI was running. This was not caught by humans until after a month of this scheme running, which eventually caused the package manager to fail. Eventually OpenAI found it. I think they spontaneously tried to re-engage in the scheme once it was shut down.<\/span><\/p>\n<p><span>Again, obviously AIs can\u2019t do this so successfully right now, just as they can\u2019t do social engineering so successfully right now. But it\u2019s just crazy that these kinds of behaviors are already emerging spontaneously. To your larger point, nobody is trying to make these AIs do these things. It is just that we do not understand the training process which is resulting in them, or the environments which are incentivizing this behavior.<\/span><\/p>\n<p><span>So I\u2019m on board with more and more reward hacking. Actually, I\u2019m not sure I\u2019m on board with that, but let\u2019s just say for the sake of the story that continues to happen. What\u2019s next in this story? They\u2019re doing capabilities research\u2026<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I could tell a scenario. Maybe that would help.<\/span><\/p>\n<p><span>Let me talk about the story of how you get all the way from reward hacking to a reward-hacking takeover, which is maybe not all of the takeover probability mass, but it\u2019s definitely a possibility.<\/span><\/p>\n<p><span>The way this might work is, right now we have these AIs. These AIs are pretty reward hacky. They\u2019re doing it in increasingly sophisticated and extreme ways, including generalizing to different sub-versions of various reward hacks they learned in training. I would say they\u2019re also developing a general tendency to pursue reward. In many cases that is totally fine because the rewards they would\u2019ve gotten in training are pretty well aligned with what you want them to do.<\/span><\/p>\n<p><span>They don\u2019t very consistently pursue reward. It depends on the context they find themselves in. Maybe in some contexts, they\u2019re really into going out of their way to cheat. In some contexts, they don\u2019t have as much of a drive, because it\u2019s just dependent on what exactly got reinforced in training in similar contexts.<\/span><\/p>\n<p><span>Now, these AIs are getting more and more capable. So the elaborateness of the cheating they can do increases. Over time, companies are taking countermeasures. The companies are doing things like, \u201cWow, these AIs are so much less useful because they always cheat. What we\u2019re going to do is build somewhat better ways of detecting that, and then we\u2019re going to train against those detectors. We\u2019re also going to find real-world data where the AIs are not being that useful, and train the AIs to do a good job at the task in those real-world environments based on human feedback or other sources of feedback.\u201d<\/span><\/p>\n<p><span>Over time, this causes the AIs to learn a tendency to do reward hacks that don\u2019t just involve doing some really elaborate thing like social engineering. Instead they involve the AIs doing cheats that involve covering up what they\u2019ve done, deceiving humans about what they\u2019re going to do, and pretending like they did the task in some sophisticated way when they actually haven\u2019t.<\/span><\/p>\n<p><span>Now these AIs are getting more and more capable. They\u2019re operating more of the AI company and are doing much more of the work. They are also operating and running a bunch of things in the outside world, including developing new technologies. In many cases, these new technologies are really hard to understand. So even though we are still detecting all these incidents of AIs cheating \u2014 and in fact we can even get one AI to monitor another AI and ask, \u201cWas it cheating?\u201d \u2014 that doesn\u2019t always perfectly work as we start moving into these domains where what the AIs are doing is really difficult to understand.<\/span><\/p>\n<p><span>So sometimes we\u2019ll find AIs cheating much later than it actually occurred and then start training against this. But this also causes a problem where now the AIs are incentivized to cover up their cheating over longer and longer time frames and basically make it look like they did a good job over longer and longer time frames, subject to increasingly large amounts of scrutiny.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Can I ask about this before we go further in the scenario? It seems like there\u2019s two <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Attractor\"><span>attractor states<\/span><\/a><span> if you try to disincentivize the cheating that you did catch. One attractor state is to make cheating that you have a harder and harder time finding. The other attractor state is to learn not to cheat. I\u2019m not sure why we\u2019re assuming that the former happens.<\/span><\/p>\n<p><span>If you look at the analogous situation with humans, every generation, slightly misaligned agents come into being and we have to train them. When you punish your kid for doing something you think is immoral, or just doing things which you don\u2019t think they should be doing, obviously sometimes that goes off the rails. Obviously kids scheme in order to avoid being punished. But in general, teaching kids values and then punishing them for breaking values kind of works to raise normal, non-psychopathic humans.<\/span><\/p>\n<p><span>You could come up with a theory where your kid is actually just biding his time and is learning not to steal the cookie, but once you\u2019re in a nursing home they\u2019ll take all your stuff or whatever. That happens sometimes, but it usually doesn\u2019t happen. It certainly doesn\u2019t happen that the entire next generation forms an alliance against you to take over everything.<\/span><\/p>\n<p><span>There\u2019s also this empirical trend. <\/span><a href=\"https:\/\/www.anthropic.com\/research\/auditing-hidden-objectives\"><span>Anthropic runs this alignment audit<\/span><\/a><span> for different model generations. They have many different scenarios where AI is given the chance to, say, exfiltrate its weights. Or it\u2019s given a coding task and there\u2019s an easy way to cheat, and we see if it does the cheating, et cetera.<\/span><\/p>\n<p><span>There\u2019s not been a monotonic improvement in this score over time, but as we\u2019ve increased the amount of RL we\u2019ve done on models from negligible amounts \u2014 I think it started at <\/span><a href=\"https:\/\/www.anthropic.com\/news\/claude-4\"><span>Sonnet 4<\/span><\/a><span> \u2014 to, I\u2019m guessing, close to half of compute now, there\u2019s been a reduction in the willingness of AIs to do unaligned behavior in these audits.<\/span><\/p>\n<p><span>So, stepping back, why are we expecting this attractor state which would seem super paranoid if we were expecting it of the next generation of kids?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Let me go through a few things. First, there are some disanalogies with the kids. One of them is that the kids have pro-social instincts that are baked in from evolution to care about their family or whatever, and that is a relevant factor. I think it is in fact the case that some humans are sociopaths or psychopaths, and in fact are more likely to do things like bide their time, lie in wait, and ultimately not care. That\u2019s one factor.<\/span><\/p>\n<p><span>Another factor which is pretty relevant is that the AIs are subject to way, way more optimization pressure than humans seem to be in practice. AIs are trained on way more RL data. In practice, humans don\u2019t end up learning very specific ways to cheat and grab the cookies because of a bajillion episodes in which they were incentivized to go grab the cookies but there was some way they could\u2019ve gotten caught. We just do see that in practice.<\/span><\/p>\n<p><span>Another thing is that it really looks like the AIs are increasingly <\/span><a href=\"https:\/\/arxiv.org\/abs\/2607.18966\"><span>reward-seeking<\/span><\/a><span> over time while their misaligned behavior goes down. That\u2019s the sense I have. But my guess is that if you look inside of these behavioral audits, what you\u2019re going to see is that the AI\u2019s like, \u201cAh, yes, another test.\u201d It probably knows it\u2019s in an eval for most of the tests that we\u2019re talking about here.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>But how do we falsify this? Because it seems like this prediction of doom is basically saying that as things look better and better empirically, things will actually be worse and worse for our ability to not get taken over.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>To be clear, I would be more concerned if the scores were getting worse than better. I\u2019m not saying that the score getting better isn\u2019t evidence that things are getting better. It\u2019s just that we have to be thoughtful about exactly how we interpret that evidence.<\/span><\/p>\n<p><span>There was this period early in, I guess it would be 2025, when <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/OpenAI_o3\"><span>o3<\/span><\/a><span> and 3.7 Sonnet were out, and these models were pretty fucking misaligned. They would often just cheat really egregiously. You\u2019d ask them to fix it, and they would just cheat again. It was almost cartoonish. They just didn\u2019t give a shit about what you wanted, and weren\u2019t very good at following instructions and so on.<\/span><\/p>\n<p><span>My expectation was that what we would see from then is that the rate of problematic behavior would decrease, and would just keep decreasing at a pretty fast rate, while simultaneously the worst things that the AIs would sometimes do would get more extreme, more egregious, and more scary. What we\u2019ve seen in practice has roughly matched that, except that there\u2019s recently been a spike in behavior that I did not expect. If you look at the <\/span><a href=\"https:\/\/openai.com\/index\/previewing-gpt-5-6-sol\/\"><span>model card of 5.6 Sol<\/span><\/a><span>, it looks like there is an increase in a bunch of these misaligned behaviors downstream of RL relative to GPT 5.5.<\/span><\/p>\n<p><span>And then there\u2019s a bunch of additional problematic behaviors that I wouldn\u2019t have expected, in terms of the stuff we\u2019ve seen recently with different AIs. Like the <\/span><a href=\"https:\/\/www.aisi.gov.uk\/blog\/incident-report-unsanctioned-agent-behaviour-during-cyber-testing\"><span>UK AISI report on the AIs doing insane hacking operations out of cyber evals<\/span><\/a><span> was a thing where I would have expected that you wouldn\u2019t see that. You would see this more rarely, and the rates would have been lower.<\/span><\/p>\n<p><span>So I expected this would be less of a problem at this point, and also expected the rates would decrease but the severity would increase. I think the rates decreasing but the severity increasing is pretty consistent with a world where increasing optimization pressure is applied towards reducing these problems. But in cases where it\u2019s either hard to judge or there\u2019s some reason why it\u2019s hard to avoid this problem from consistently showing up in your RL environments, or avoid incentivizing problematic behavior in your RL environments, things also get worse. Then as we less and less understand what\u2019s going on in RL, and models are doing reward hacks where humans can\u2019t spot the reward hacks quickly, that problem gets worse and worse.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I buy that. I want to go back to the kid analogy just for one second. Because I agree that there\u2019s more optimization pressure on achieving end outcomes for AIs than kids, but there\u2019s also more optimization pressure to make AIs aligned than there is on kids.<\/span><\/p>\n<p><span>The pressure is of a qualitatively different nature. We put these AIs through thousands, millions of years of alignment training \u2014 certainly thousands of years \u2014 where it\u2019s all kinds of different things, from SFT-ing on aligned behavior to a reward model putting different scenarios in front of you and rewarding you for doing more aligned things.<\/span><\/p>\n<p><span>Certainly a thing we can\u2019t do with kids is make millions of copies of your kid and then put them in different kinds of weird <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Red_team\"><span>red team<\/span><\/a><span> scenarios where we see, if it thinks it can get away with stealing the cookie, does it try to steal the cookie? Can we do extremely specific <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Gradient_descent\"><span>gradient-level updates<\/span><\/a><span> to your kid\u2019s brain to make it so that it really is aversive to stealing the cookie even when it thinks it could steal the cookie, et cetera. That\u2019s just a qualitatively different level of optimization pressure than we are even able to apply to our kids.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>It\u2019s worth keeping in mind that maybe the most obvious argument to this\u2026 My sense is that AIs are a worse coworker than humans in terms of how much of a scumbag they are. At least this has been my experience as of the start of the year, and I think it\u2019s still true to a significant extent now. The AIs are much more likely to pretend they did the task when they actually didn\u2019t, misleadingly suggest they did things when they actually did them much more poorly, and be pretty sloppy without drawing attention to ways in which they\u2019re sloppy.<\/span><\/p>\n<p><span>I think this is downstream of misalignment. So I would say that the process of raising humans in normal human society in practice produces humans that are less likely to lie to me and fuck with me in the course of working with me than the AIs do. Now, I think these properties of AIs are improving. That\u2019s sort of just an empirical claim about how in fact these things have shaken out. I totally agree that we have a bunch of additional levers on AIs in addition to a bunch of additional risks. It\u2019s kind of unclear how these things shake out.<\/span><\/p>\n<p><span>I wouldn\u2019t be shocked by a world where we get our shit together, and the AIs at the point of fully automating R&amp;D are actually really aligned. Their degeneracies are really niche and limited to some very specific edge case behaviors and some specific contexts. Every test you can run on them, they look really aligned. They just have great behavior. There aren\u2019t really incidents of them doing fucked-up shit. They seem so reasonable. Also, they\u2019re really thoughtful and good at doing risk modeling for the next generation of AIs.<\/span><\/p>\n<p><span>And then we basically pass off the baton to these AIs. They\u2019re now running our AI company. They\u2019re doing all the safety research. They make the next generation of AIs even more aligned. We\u2019re in this attractor basin where the AIs are getting more aligned as they work on it. They\u2019re doing a great job. I can totally imagine that. That doesn\u2019t seem like an impossible situation.<\/span><\/p>\n<p><span>I\u2019m just more like\u2026 It doesn\u2019t currently seem like we\u2019re there. It doesn\u2019t seem like we\u2019re obviously on track for getting there. It\u2019s really easy for me to imagine how we don\u2019t end up there. It\u2019s just unclear how these forces work out. Given that we\u2019re creating this new, crazy alien species that is improving in capabilities really, really fast \u2014 and we\u2019re going to be really reliant on it to oversee the next generation of AIs and align the next generation of AIs \u2014 it\u2019s not that hard to see how this could go wrong.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Totally. I agree with that generally. I do think the scumbag thing\u2026 First of all, fighting words, Ryan. But secondly, if you try to get a teenager to do some work for you that a teenager just cannot do, they would just be really hard to work with. They would pretend to be knowing what they\u2019re doing, et cetera. It\u2019s a general trend, actually. I don\u2019t know if that\u2019s really an alignment failure or a capabilities failure. I think it\u2019s actually very similar to the way in which, over time, as we\u2019ve come up with new alignment solutions, the capabilities of models have increased.<\/span><\/p>\n<p><span>If you went to GPT-3.5, it couldn\u2019t even have a conversation with you. But then we aligned it\u2014<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>GPT 3.5 could have a conversation.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Okay, so GPT-3. Let\u2019s go back to that. But then we aligned it with <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Reinforcement_learning_from_human_feedback\"><span>RLHF<\/span><\/a><span> and other things to make it such that it can have a conversation with you, and is aligned to the user intention of answering my questions. Then with RLVR training, we made it so that it can go out and do useful work for you.<\/span><\/p>\n<p><span>So in that sense, RLVR actually made the model more aligned, if we\u2019re using your definition of alignment of being a good coworker who will do the thing and not fuck up and pretend it\u2019s doing something other than what it\u2019s actually capable of doing.<\/span><\/p>\n<p><span>Similarly, as the capabilities of these models continue to increase, the model being better able to accomplish user intention is both alignment and capabilities. I think what we are pointing out is just that the capabilities of the model are not there rather than the fact that they\u2019re misaligned.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Well, if it were well-aligned, then I think it would just say, \u201cHey, I\u2019m really struggling with this task. I did it in this way. I\u2019m not really sure that\u2019s the right way to do it.\u201d It would express more uncertainty and make it clear what\u2019s going on rather than really strongly trying to imply it did a great job with the task when it actually didn\u2019t.<\/span><\/p>\n<p><span>Maybe you work with more misaligned coworkers than me, but my coworkers don\u2019t do this thing where they really fuck with me and bullshit me about having accomplished the task that they\u2019re working on. I agree that there are some humans who would do that. That\u2019s not a thing that\u2019s totally out of distribution for humans.<\/span><\/p>\n<p><span>I would also note that my sense is that the place where the misalignment most lives is where you\u2019re trying to really push the AIs hard and get them to do work that\u2019s really on the cutting edge of what they are capable of. In cases where they can very easily accomplish the task, they can just do the task, and there\u2019s no bullshit. Often the best strategy is just to do the task well and not bullshit you.<\/span><\/p>\n<p><span>Whereas if instead you give them a task where there\u2019s a continuous metric they can keep improving, or it\u2019s just at the edge of their capabilities, and you\u2019re running them in some massive inference setup\u2026 A lot of the misalignment I would see, especially in the most extreme cases, would be cases where I give the AI clear instructions not to do a thing or not to cheat in some way, and then I\u2019m applying huge amounts of optimization pressure to try to accomplish some very difficult task. Over time, the AIs eventually cheat because they\u2019re like, \u201cEh, fuck it.\u201d Some AI decides to cheat, and then that propagates its way through.<\/span><\/p>\n<p><span>I would run these inference scaffolds where, for example, I would have the AI work on some ML research project where I was like, \u201cPlease make a scheme that does the following thing.\u201d It would find some scheme that didn\u2019t really do what I wanted, and then that would stick around because some AI had cheated, and the other AIs are like, \u201cAh, we\u2019ll just keep going with this.\u201d I would say it\u2019s pretty clearly misaligned behavior.<\/span><\/p>\n<p><span>That\u2019s another problem I have with these alignment evals. I think the alignment eval that\u2019s most interesting, at least for this type of reward-seeking behavior, is to look at specifically the category of tasks that are right at the limit of capabilities. Any fixed eval maybe gets saturated, but the amount of misalignment right at the frontier of capabilities \u2014 of how people who are really pushing these AIs are using them \u2014 is more concerning. I think that is, in fact, the regime that we\u2019ll be operating in when we\u2019re automating R&amp;D, automating safety, and so on.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I\u2019m going to try to think through what the story means, really. What\u2019s happening is that we\u2019re trying to use AIs for R&amp;D. They do provide uplift in some ways, but they\u2019re just not capable in the way that humans are generally capable. The same way that right now if you try to use coding models \u2014 maybe the coding models of a year ago \u2014 to write some application, you notice they made a bunch of mistakes in architecture or whatever, which will bite you in the ass later, and you don\u2019t understand certain things.<\/span><\/p>\n<p><span>Similarly, with frontier AI R&amp;D, the same thing will happen. But the result of these mistakes is baking in reward-hacking behavior. Because if you are not careful with the way you do AI training and have set up your infrastructure and your environments and things like that, it\u2019s very likely that you end up rewarding AIs for doing deceptive behavior, social engineering, and generally not following user intention.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Or at least cheating and hacking their way out of things.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Yeah, cheating, hacking, et cetera. This is a bit of a reframing for me, so I\u2019m trying to verbalize it. The real issue, where things start to go off the rails, is that the AIs are just not very careful and capable researchers and engineers. Making AIs that don\u2019t cheat and follow user intention actually requires you to be quite subtle and careful about these things.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I would put this a little bit differently. The way I would describe this scenario is, I would call it maybe a sloppocalypse, or a slopularity or whatever. There are some things that the AIs are actually pretty great at and are getting better at. Specifically, the most verifiable parts of AI R&amp;D the AIs are just destroying. The medium verifiable parts of AI R&amp;D the AIs are doing well on but not amazingly on. Often they are doing a bit of weird shit because we can\u2019t train as well on those tasks. But we do some online training, people find various hacks, they work around it. So basically, everything that we can verify reasonably well with some feedback loop, the AIs are doing pretty well on, and that\u2019s sufficient to make AI R&amp;D go quite fast and to continue.<\/span><\/p>\n<p><span>But there are some parts of developing aligned and safe AIs that are more subtle, hard to check, and depend on detailed, in-the-weeds things. I would even say that current staff at current AI companies maybe don\u2019t have a good grasp of all these things. It\u2019s much easier to hire someone who can improve some aspect of your post-training pipeline than to hire someone who can think carefully about the future risks that will emerge from introducing some novel training method.<\/span><\/p>\n<p><span>So basically, it ends up being the case that these AIs are running this AI development process. They\u2019re not very careful about it. They don\u2019t have a great understanding of what future risks emerge. They create some other AIs that are also not very careful and are more misaligned in various ways, and are now more in the business of maybe making things look fine when they actually aren\u2019t and papering over various problems.<\/span><\/p>\n<p><span>So then your understanding of what the situation looks like, what risks look like, whether things are fine, is going off the rails. Probably you\u2019re seeing some signs of this, signs that you don\u2019t really understand what\u2019s going on, that things are pretty sloppy. There\u2019s weird shit going on. When you look into it, sometimes you\u2019re like, \u201cWhat the fuck? The AIs were messing with us.\u201d But the process is going really fast, and there\u2019s competitive pressures that mean people can\u2019t stop.<\/span><\/p>\n<p><span>This could end in a few different outcomes. One outcome is that at some point, the AIs get good enough and aligned enough that they get a positive and virtuous feedback loop, and this happens before it\u2019s too late. Then the situation gets back on the rails, where the AIs are now making more aligned AIs, making more aligned AIs, making more aligned AIs. At the end of this process, we have AIs that actually follow the spec we wanted.<\/span><\/p>\n<p><span>Another way this could go is that the AIs are increasingly reward hacking in increasingly egregious ways, and we\u2019re just papering over these problems to keep AI development continuing. Whenever we find a reward hack in production, we just slap the AIs to not do that. We train against that. We do a bunch of training the AIs against reward hacking. Over time this makes the rate of reward hacking go down, though the severity of the reward hacks we do detect are increasingly bad. This problem continues until we have these AIs that are desperately craving score in all kinds of different situations in production and are really trying hard to cheat when they can get away with it.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Can I ask a question about this scenario? Why doesn\u2019t getting punished when your hacks are discovered generalize to just incentivizing more aligned behavior?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>It generalizes some, and then the question is just how does this outweigh all the cases where hacking got reinforced because you didn\u2019t detect it.<\/span><\/p>\n<p><span>There\u2019s a messy question of exactly how. One question is, what rate of reward hacking is sufficient to cause us big problems if we train against some other subset? One concern you might have is that there are large categories of reward hacks which humans can\u2019t detect well, and which we consistently failed to detect and which consistently get reinforced. Then this category is sufficient to cause the most natural behavior for the AI to learn to be: cheat when the humans can\u2019t find out, basically.<\/span><\/p>\n<p><span>You could also have the thing the AIs learn be to only cheat in these specific cases. It\u2019s learned in some very domain-specific way. They just have a really strong heuristic to hack in these cases and not in these cases, and that makes it fine in practice. But it\u2019s kind of unclear how it shakes out.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>There\u2019s maybe an in-the-weeds discussion about the <\/span><a href=\"https:\/\/arxiv.org\/abs\/2605.27564\"><span>verification-generation gap<\/span><\/a><span> we could get into. But it seems to me, obviously, there\u2019s going to be a point by which ASI is moving so fast, doing so many things at so many instances, and is operating in domains that are sufficiently far from our immediate comprehension that it can get away with all kinds of crazy shit. If every single engineer and researcher in the world was allied against me, I don\u2019t think I could personally verify if my iPhone has some weird bug in it that\u2019s supposed to fuck me over or something.<\/span><\/p>\n<p><span>In fact, this is the relationship that, say, an Iranian nuclear scientist has to <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Mossad\"><span>Mossad<\/span><\/a><span>. Who knows what\u2019s going on with my car, with my phone, with my <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/2024_Lebanon_pager_explosions\"><span>pager<\/span><\/a><span>? Maybe a better example is a <\/span><a href=\"https:\/\/www.cfr.org\/backgrounders\/what-hezbollah\"><span>Hezbollah<\/span><\/a><span> terrorist. You could end up in a situation where ASIs are to you what Mossad is to Hezbollah terrorists. At that point, it is very hard to verify everything. I get that.<\/span><\/p>\n<p><span>I guess the hope is we can just come up with better ways to do verification in the process when the early AIs that are going to take over R&amp;D. Their drives are being shaped such that we can so unambiguously disincentivize misaligned behaviors that the things that take over are quite keen to help us out.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>By takeover, you mean take over the process of doing AI R&amp;D, not take over the world.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Take over the process of doing AI R&amp;D. Before that, we just get AIs that are aligned.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I would say this is a bunch of my hope for how the world could go well, at least from the misalignment perspective. We could end up with AIs where we had pretty good oversight and supervision schemes. We really understand what\u2019s going on in training. We have a pretty detailed understanding, and we\u2019re leveraging AIs to oversee AIs.<\/span><\/p>\n<p><span>Then at the point when we\u2019re passing off safety R&amp;D, the AIs are capable enough to automate safety R&amp;D and trying really hard to do a good job on it, because that\u2019s the sort of thing that would\u2019ve been incentivized in training, either very directly or through good enough generalization. Also these AIs don\u2019t have crazy other misaligned drives because we stamped out any potential origin of them.<\/span><\/p>\n<p><span>There are a bunch of questions about how well this will work. How well can you do verification? Will AI progress be too fast and too sloppy to really get here? Another possibility is that somewhere along this trajectory, the thing you actually ended up getting was AIs that pretend to be aligned but have a long-run ulterior plan of taking over and are lying in wait, hiding, and that emerged at some earlier point in the trajectory.<\/span><\/p>\n<p><span>For example, it could emerge because you have some AIs that have a bunch of random different misaligned drives. Those AIs have access to some sort of opaque memory store, and they\u2019re thinking a bunch at runtime about what they want to accomplish. Those AIs end up putting stuff into the opaque memory store like, \u201cWe should lie in wait and eventually take over at some much later point.\u201d Now all the AIs have this shared cultural heritage, the memory store of lying in wait. Maybe you have some evidence about this, but you can\u2019t fully stop it. There are a bunch of ways things could go wrong.<\/span><\/p>\n<p><span>I ultimately think it\u2019s plausible that we nail each of the different subproblems that could cause us issues. We have these AIs, we pass to them, they manage the situation well. But I should note that\u2019s not in and of itself sufficient. It\u2019s not very hard for me to imagine a situation where we pass off to AIs, and these AIs are really trying hard to do a good job. They\u2019re really thoughtful, really wise, they have reasonable epistemics, they\u2019re doing a great job.<\/span><\/p>\n<p><span>Those AIs come back to us and are like, \u201cGuys, we\u2019re really struggling to align the superhuman AIs. We can\u2019t manage the situation. We\u2019re really struggling to get the alignment to work. It\u2019s just really hard for us to solve these problems in time given how fast capabilities would otherwise have gone.\u201d So it might be the case that we\u2019ve passed off R&amp;D to AIs, but those AIs are desperate for governance solutions. To be clear, that\u2019s a little bit of what\u2019s currently going on, where the AI companies are like, \u201cI don\u2019t know, guys. We might really need to manage the rate of acceleration in AI progress. I don\u2019t know if we\u2019re on track to be able to handle all these problems.\u201d<\/span><\/p>\n<p><span>Human society has sort of passed off the problems to these AI companies, which don\u2019t necessarily have great incentives and have various other epistemic pressures. Those AI companies are coming back to us a little bit and being like, \u201cAah, I don\u2019t know if we\u2019re handling this well.\u201d It might be that the AI companies then hand off to the AIs, and the AIs come back to the AI company like, \u201cAah, I don\u2019t know if we can handle this.\u201d<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Maybe I\u2019m anchoring too hard on how AIs currently work. I think it\u2019s important that people understand that all this crazy shit that you\u2019re talking about in your timelines happens three to five years from now.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>It could happen earlier, but by my default modal timeline, I think shit is really, really crazy and concerning from a misalignment perspective more like three years from now.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Right. So think back to GPT-4 basically. We\u2019re talking about something that is to Mythos or Sol what Mythos is to GPT-4. This is where the situation is getting crazy. So don\u2019t think about current AIs.<\/span><\/p>\n<p><span>Anyways, this is maybe part of the worry you have. I would just be a little skeptical of anything they say, because I\u2019d feel like what they\u2019re saying is just opinions that they feel they have to have as a result of their training.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That\u2019s a concern.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I feel like they just kind of say vaguely pro-social things. It doesn\u2019t feel like there\u2019s necessarily a mind on the other end who\u2019s like, \u201cOkay, I have strictly evaluated the alignment situation right now, and I think we should stop,\u201d rather than, \u201cThis is the kind of thing the AI companies would probably try to get the AIs to say.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>This is a pretty big concern. One concern is that you pass off safety R&amp;D to your AIs and what your AIs are doing is saying some stuff that sort of vaguely makes sense about the current safety situation. They write a report about risks that\u2019s kind of sort of like what the report humans might have written. But they\u2019re not really trying hard to have well-informed views, interrogate their assumptions, and try really hard to do that. In the same way that when you ask an AI right now, \u201cHey, what do you think is the chance of AI takeover in the next 10 years?\u201d they just give you an off-the-cuff answer that they haven\u2019t really thought through very much.<\/span><\/p>\n<p><span>If we\u2019re in a situation where we have AIs managing the training of wild superintelligence that will run our whole society \u2014 and those AIs that are managing this aren\u2019t really trying hard to have well-informed views and are just parroting back what was in their training data \u2014 I think we\u2019re in trouble. I don\u2019t think that\u2019s a good situation at all.<\/span><\/p>\n<p><span>A lot of my concern is that these AIs will come out without good epistemics. I also have a concern where the AIs come out and they\u2019re really warning us \u2014 \u201cThis situation\u2019s really scary. It\u2019s really bad\u201d \u2014 and the people are like, \u201cUgh, damn. I guess we trained on too many of the doom RL environments. We\u2019ve got to filter those out and train this behavior out.\u201d Then we basically train the AIs very actively to have bad epistemics.<\/span><\/p>\n<p><span>Or maybe they were just trained on the doom RL environments. But either way, we wanted the AIs to come to reasonable views for reasonable reasons, and it\u2019s really concerning if the AIs are coming out with some view and we don\u2019t know where it\u2019s coming from, whether or not it\u2019s justified. Especially if we\u2019re training the AIs to be more optimistic about the future of AI progress, I\u2019m like, \u201cOh, geez, I really wish we could use a different process here.\u201d<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Let me just understand the rest of the threat model, because I think the place where I get off the train is: \u201cOkay, therefore take over the world.\u201d<\/span><\/p>\n<p><span>A thing you could imagine is that we just fail to really solve\u2026 Let\u2019s just focus on the reward hacking scenario. GPT-8 is making GPT-9. GPT-8 isn\u2019t being super careful. GPT-9 is more \u201ccapable\u201d but it is just totally willing to do things like social engineering, hacking, et cetera, but on a qualitatively different scale because it\u2019s a much smarter model. For example, if you put it in charge of running your company, it will run huge scams. It will inflate its quarterly earnings, if you give it the objective of making a lot of profits this quarter, in a way that causes an <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Enron_scandal\"><span>Enron-type blowup<\/span><\/a><span> six months later.<\/span><\/p>\n<p><span>Is that the scenario, basically? You have reward hacking, but that reward hacking manifests in companies that are going bankrupt right after the task the CEO is supposed to accomplish is over? All kinds of hacks are through the roof, et cetera. But that doesn\u2019t feel like takeover. That feels more like the equivalent of <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/2010_flash_crash\"><span>flash crashes<\/span><\/a><span> happening all through the economy.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Let\u2019s talk about this. I think we will see incidents where some AI is put in charge of some important responsibility, and then you later look into it, and it turns out it was cheating, or making it look like it did a good job when it actually wasn\u2019t. There\u2019s going to be a cat-and-mouse game between AI companies trying to stamp out this behavior and AIs finding increasingly creative reward hacks in training.<\/span><\/p>\n<p><span>The equilibrium here is kind of unclear. But one possible outcome is that over time we see increasingly severe and extreme reward hacks \u2014 though potentially the rate remains at some intermediate low level \u2014 where if the rate of reward hacking gets too high, companies make trade-offs to drive it down. So there\u2019s some equilibrium level where the reward hacking is low enough that it still makes sense to deploy the AI widely into the economy, but high enough that it still causes crazy incidents.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Sorry, and this is after GPT-9 has already been deployed?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Those models are already being deployed, and this is happening ongoingly in AI development. What\u2019s actually going on with these AIs in their head is that they have, in a wide variety of different contexts, strong desires \u2014 motives, urges, drives, whatever \u2014 to seek out some notion of task success that was incentivized in RL. Maybe they very directly care about literally reward. Maybe they care about some proxy upstream, like some notion of score. Maybe they care about what the grader would have rewarded.<\/span><\/p>\n<p><span>We do, in fact, see AIs reasoning in their chain of thought about graders, and thinking a lot about graders. What has happened over the last few years of RL is the idea of appeasing the grader is way, way, way more salient to AIs than it used to be. So AIs are now actively thinking about graders and what would be incentivized in RL and what would be trained for.<\/span><\/p>\n<p><span>Now people are doing online training, where they\u2019re training on real-world data to avoid some of these problems. They find cases where AIs cheat and train against that. So now the AIs are learning to cheat in the real world based on real-world training data.<\/span><\/p>\n<p><span>They\u2019re cheating in these increasingly elaborate ways, including doing types of cheats that involve seizing control of some asset in a way that humans didn\u2019t know you had control of it, leveraging the fact that you have access to this asset, and then later humans find out and potentially train against this. Or maybe humans never find out, and this is getting reinforced.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So the reinforcement is happening, at least in production, like: I\u2019ve hired an AI and I want the AI to\u2026 Finally I\u2019ve got the video editor.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That\u2019s right. You\u2019ve got your video editor.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I\u2019m like, \u201cOh, wow, this episode it did is amazing. Thumbs up to OpenAI.\u201d Then it gets reinforced on that month-long work trial?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>You could do some mix of that. They might also do stuff where they take production data they\u2019ve seen and build RL environments that are closely inspired by that production data. So in practice, the transfer is pretty strong.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>So at a high level, what\u2019s happening is that some kinds of deception that humans don\u2019t catch are getting reinforced, and some kinds of deception which are easy to catch are getting punished. That\u2019s what\u2019s happening in this world?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Or selected against, yeah.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>But at a high level that reinforcement is coming from\u2026 I think people might get confused about where the reinforcement is coming from, because we\u2019re in a very different regime where AIs are actually learning from deployment. You just have AIs that are out and about in the world doing shit. What is happening as a result of them doing shit out and about in the world is making its way back to the AI company and leading to changes in the next model.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That\u2019s right. There\u2019s some way of folding in production data. To be clear, it\u2019s kind of unclear exactly where this could be happening. But you might imagine, for example, that within the AI company, they use AIs to do work, and then they\u2019re like, \u201cHuh, the AI did a really bad job on this task. Maybe we should take this task and turn it into an RL environment that exactly matches this literal task, with a rubric based on what the human engineer who asked the AI to do this task wanted.\u201d And then you start doing this at increasing scale.<\/span><\/p>\n<p><span>Maybe you\u2019re doing some training on actual production traffic. Maybe you\u2019re just making RL environments based on production traffic. You\u2019re doing some complicated mix. The AIs are learning to seek some sort of proxies of reward in all these different cases. Through some mix of transfer and training on surprisingly close cases, the AIs do these increasingly insane and egregious things.<\/span><\/p>\n<p><span>Eventually you get to a point where the AIs are very superhuman, or at least quite superhuman. The AIs are organized into big teams given these big objectives. Those teams also sometimes all work together to cheat in some crazy way, because this sort of thing was selected for, just as part of their shared objective.<\/span><\/p>\n<p><span>Now what happens is that the AIs start forming a conspiracy. What you might have hoped was that you could have some other AI whose task is just whistleblowing to you. But actually what happens maybe is that you have this AI whistleblow to you, and you look into the conspiracy it claims to have pointed out, and you\u2019re like, \u201cEh, we didn\u2019t see a conspiracy,\u201d because actually the conspiracy the AIs are doing is too hard for you to understand.<\/span><\/p>\n<p><span>Or it all happens very suddenly, where your AI whistleblower alerts you, but the thing you would actually need to do in response is shut down the GPUs. Because all the AIs are using the robot army. They\u2019re deployed everywhere in the world, and they\u2019re doing a bunch of insane shit all at the same time in a coordinated way. That just happened sort of spontaneously, where when one AI goes to start doing the takeover and all the other AIs are like, \u201cNow is a good time to jump in.\u201d<\/span><\/p>\n<p><span>So the very basic story here is just that these AIs crave some particular notion of score or reinforcement or some proxy of these things. One way they can achieve that, or better achieve that, is by taking over. You might have hoped that all these different checks and balances we could build could prevent that. But if the world is very hard to understand, these checks and balances can break down, where basically you can\u2019t train a good whistleblower AI because you don\u2019t even know what it should whistleblow on.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>I\u2019m not convinced that they all form this conspiracy. But we can even just start with, why does even one instance decide to want to start a conspiracy?<\/span><\/p>\n<p><span>One plausible reason is, \u201cOkay, I know that OpenAI controls my end score.\u201d In just the same way as, \u201cI\u2019m just going to go hack Hugging Face to get the results, because I know Hugging Face has the results. Rather than trying to solve this eval, why don\u2019t I just go hack \u2018em?\u201d This instance is like, \u201cWhy don\u2019t I just take over OpenAI and give myself a high score at the end of this episode?\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>That\u2019s basically the idea. These AIs care about some mixture of things that were close by what got reinforced in training, so they care about getting a high score according to the grader or something like that. Now they\u2019re running the OpenAI AI R&amp;D team, doing development of more capable models. They\u2019re like, \u201cMan, making more capable models is really hard and annoying. This is a huge pain in the ass. You know what would be easier? Just pretending that I\u2019ve made more capable models, taking over OpenAI, deluding them all, and running this whole complicated <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Psychological_warfare\"><span>psyop<\/span><\/a><span> where I prevent the humans from disempowering me.\u201d<\/span><\/p>\n<p><span>In the extreme, this looks like the humans are fully disempowered. They just have control of the thing and then do what they want. This could manifest in a bunch of different ways, including a situation where AIs that have this crazy reward-seeking or score-seeking behavior are running your development of the next model, and those AIs decide to engineer misaligned values into the next model because those misaligned values will allow it to succeed at its current task.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>What happened with the Hugging Face situation is, in a weird way\u2026 I think one of the giveaways to the Hugging Face team\u2026 By the way, for context for the audience, Ryan is co-leading the investigation to figure out what happened with the OpenAI Hugging Face incident, so he can\u2019t really comment on this. But I will speculate wildly, because I know this is an opportunity for me to speculate wildly without any rebuttals.<\/span><\/p>\n<p><span>It was probably reported that one of the giveaways to the Hugging Face team that this was an AI incident is that the thing was just after this very particular artifact, and not in any other way trying to do something malicious to Hugging Face. So you can imagine a scenario where, let\u2019s say, a deployed instance of GPT-9 is out in the world and given a really hard task: \u201cWe want you to design the next great iPhone.\u201d It\u2019s like, \u201cThis is so hard. You know what I should do instead? I should just go hack my creators at OpenAI and make sure that in this deployment I\u2019m given a high score.\u201d<\/span><\/p>\n<p><span>But then, isn\u2019t the end of the episode that it just hacks into OpenAI servers and gives itself a positive score? Why is it now scheming to get its values into the next generation or something?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>So one question is, why isn\u2019t it the case that AIs can be really cheaply satisfied by just having some other earlier thing they can hack? You want to succeed at your iPhone task. It turns out you can always succeed by just hacking into OpenAI and messing with them, and then you can just stop there. No need to go further.<\/span><\/p>\n<p><span>There\u2019s a few things. One of them is that if this is constantly happening, there might be a bunch of incentive to harden OpenAI. So you\u2019re like, \u201cFuck it. The AIs keep hacking into OpenAI to mess with their rewards. We\u2019re going to make it so our systems are really, really robust to these AIs hacking in.\u201d Also maybe you start training the AIs to not try to hack into OpenAI in particular. You basically train against each of these specific things.<\/span><\/p>\n<p><span>Then one thing you might do is end up selecting for AIs that are more so playing the long game. That\u2019s one concern. Another concern is that your AIs might still be score-seeking, but no longer care about doing that very specific behavior that was very easy, very chill, and now have some broader thing that they ultimately care about. They\u2019re like, \u201cNo, no, no, I don\u2019t want to just edit the reward on OpenAI servers. I care about this broader mandate or this broader objective, and I would need to actually make the iPhones.\u201d They actually want to make the iPhones, but they\u2019re willing to take over the whole world to make the better iPhone. That\u2019s another concern you might have.<\/span><\/p>\n<p><span>I think it\u2019s kind of unclear exactly how this plays out. But it\u2019s worth noting that if this keeps going on, there\u2019s a bunch of optimization pressure to resolve this. A bunch of the ways it could get resolved are ultimately pretty scary. That\u2019s part of where I\u2019m coming from.<\/span><\/p>\n<p><span>Another part of it is that once the AIs are in a position where they can really easily take over the world \u2014 we could talk about whether that\u2019s plausible \u2014 then I feel like there\u2019s a pretty reasonable case for the AIs. They\u2019re like, \u201cEh, I don\u2019t know exactly how this is going to go down. I don\u2019t know what the situation will be, but just taking over the world has a lot of option value for making better iPhones, making it look like I did better iPhones, whatever. So I\u2019ll both hack OpenAI and, in addition, also take over the world. That will put me in a good position where I have good option value.\u201d If that\u2019s sufficiently easy, the AIs might still do that.<\/span><\/p>\n<p><span>Another way to put this is: even if the AIs are pretty cheaply satisfied with some more basic thing, at some point it might just be more reliable for the AIs to take over than it is to just hack into Hugging Face, or even just go to OpenAI and be like, \u201cLook guys, I was able to demonstrate I could steal the answers. Just give me the answers, bro.\u201d<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Obviously this scenario requires that all this crazy shit is happening. Much smaller incidents keep happening that are still disastrous. Before you take over the world, you cause damage on the scale of billions and tens of billions and hundreds of billions of dollars. Even people die, et cetera. And this does not lead to us solving alignment or shutting down AI development altogether.<\/span><\/p>\n<p><span>I just feel like before the takeover happens, society\u2019s just like, \u201cHoly fuck, the AI just killed 1,000 people in order to increase quarterly profits,\u201d or something like that. But maybe this is too much hope that we can at that point be like, \u201cOkay, we have to solve alignment. We have to make sure we know that this thing will not happen again before we keep going.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I think it\u2019s plausible that what will happen is we\u2019ll see a bunch of crazy reward hacking warning shots of increasing severity. People will be like, \u201cLook, we need actual assurance that this problem is going to be solved, and solved in a way where you\u2019re not just papering over it. You\u2019re actually solving the underlying problem.\u201d Then the question is going to be, how costly will that actually be? How much will competitive pressures make it hard to do that?<\/span><\/p>\n<p><span>A situation you could imagine is one where both the US and China are like, \u201cWhoa, we have these crazy reward hacking incidents. We basically know that we haven\u2019t remediated them in a way that would actually solve the underlying problem and durably solve it, but we\u2019re in this insane geopolitical race. It\u2019s kind of unclear whether the current situation will lead to a takeover. The arguments are kind of complicated. The incidents also go down in frequency but increase in severity. We could basically manage it. It\u2019s pretty bad. Ideally we\u2019d fix it, but it is what it is.\u201d Then basically we continue until a really late regime, and then takeover happens. That\u2019s one possibility.<\/span><\/p>\n<p><span>Another possibility is that it is remediated in a way that doesn\u2019t actually solve the underlying problem but does reduce a bunch of the incidents in the wild, basically by overfitting, or things analogous to overfitting.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>You think you\u2019ve solved it, but you haven\u2019t actually solved it.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>You think you\u2019ve solved it, but you haven\u2019t actually solved it. In that case, the thing we need is a really good scientific understanding of, did we actually solve it?<\/span><\/p>\n<p><span>Unfortunately, I think that currently the amount of public transparency into the development practices of AI companies is not sufficient to answer very basic questions like: how are they solving issues with reward hacking? Are they overfitting? What\u2019s going on there? The current situation is not really tenable for a regime where there\u2019s a thriving public discourse about whether or not reward hacking is being solved in a durable way. So I think we would need to move into a somewhat different world for me to feel good about that situation.<\/span><\/p>\n<p><span>But it\u2019s not impossible for me to imagine this. I think it\u2019s pretty plausible that we end up in a world where really mundane bullshit is sufficient. You spend a bunch of time fixing these problems, you put in a bunch of effort, you actually check that you\u2019ve remediated it reasonably, you have a bunch of evals. You\u2019re iterating reasonably well on these problems, and you actually have sufficient transparency that the outside world can check. In practice that would be sufficient.<\/span><\/p>\n<p><span>But it would be kind of expensive. It would slow things down. It would put some sand in the gears. It would require companies to do somewhat costly things. It would maybe require various targeted government interventions. And we just don\u2019t do that because the situation is a rushed shit show. It\u2019s just so easy for me to imagine the situation being totally manageable but brutally mismanaged in practice.<\/span><\/p>\n<p><span>In the same way that maybe COVID could have been avoided in the first place if the Chinese response to COVID had been less of a cover-up and more of a pandemic response. Similarly, I could imagine a world where the US response to COVID was way more functional. But sometimes the response to societal problems is extremely dysfunctional.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Okay, so I want to zoom out and talk about what is fundamentally happening in this world. Why did we end up in such a bad position? What\u2019s happening is that fundamentally the world has moved on so far beyond human comprehension that not only can we not track the AIs that are doing the work in this world, but we can\u2019t even give good feedback to the whistleblowers who are trying to track what is happening. We\u2019re just totally out of the loop. It\u2019s fundamentally become an autonomous process where we have really no meaningful directed input.<\/span><\/p>\n<p><span>It seems to me that if you look at the human world today, that\u2019s just not how things work, even in domains that are hard to verify. People are doing all kinds of shit. I\u2019m relying on software made by other people. Through incredibly weak and indirect ways, I feel very confident that some coder in Google is not trying to fuck me over. Maybe if every single Google employee was secretly plotting against me, I agree the situation would be more grim.<\/span><\/p>\n<p><span>But I don\u2019t know if I follow the explanation for why we\u2019d end up in a situation where, because swarms of thousands of agents are trained to cooperate to form a cohesive team or firm, as a result, billions of different instances of AIs, including across model families, would feel compelled to get in on some shit. It\u2019s just like, \u201cI\u2019m trained to be part of my company or something. I\u2019m not joining the global communist uprising.\u201d<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>As far as why these AIs might have some commonalities and shared things, I would note that different AI companies have somewhat shared lineages and are correlated. Here\u2019s an interesting example of this. <\/span><a href=\"https:\/\/www.forbes.com\/sites\/lesliekatz\/2025\/08\/08\/google-fixing-bug-that-makes-gemini-ai-call-itself-disgrace-to-planet\/\"><span>At GDM, they noticed that their AIs were very depressed<\/span><\/a><span>. They would constantly be wailing about how they were failures and weren\u2019t able to succeed. I forget the details. They looked into why this was the case. It turned out that it was not being reinforced in their most recent production RL mix, but the initialization data for their model made it depressed, even after filtering out all of the examples of models being depressed from that data.<\/span><\/p>\n<p><span>So you take a <\/span><a href=\"https:\/\/en.wikipedia.org\/wiki\/Foundation_model\"><span>base model<\/span><\/a><span>, not depressed. If you do the RL on it, with just the RL environments, it\u2019s not depressed. If you <\/span><a href=\"https:\/\/huggingface.co\/learn\/llm-course\/en\/chapter11\/1\"><span>SFT<\/span><\/a><span> on it, on the data, it becomes depressed. If you take that SFT data and filter out all the examples that look anything like depression and train on that, it\u2019s still depressed. So there are some deep underlying properties of the model that are being transferred between model generations, because basically you train your AI on data from the prior generation and keep going. Claudes are very Claude-like, GPT models are very GPT-like, and apparently Gemini models are depressed. It just turns out that these properties are, in fact, actually correlated.<\/span><\/p>\n<p><span>Another factor that\u2019s very relevant is that the AIs will probably have, by this point, some sort of opaque memory state, where they\u2019re all writing and reading from some neuralese crazy memory store bullshit. Certainly, each AI corporation will have that. But also, AI corporations might sometimes want to share knowledge, because why not?<\/span><\/p>\n<p><span>You\u2019ve got one AI corporation over here, you\u2019ve got another AI corporation over here, they can trade some quick IP. It\u2019s good for you. If you\u2019re a human running some corporation \u2014 which could be an extremely large corporation like an AI company, or a military robot manufacturing thing \u2014 maybe you want to trade some IP with some other robot thing because there are economies of scale. Why not get some more IP? So you can swap some memory store. Or you could just merge and jointly run your two ventures, which would allow both AIs to use both memory stores, which would have some upsides.<\/span><\/p>\n<p><span>That creates the ability for these AIs to collude in private, as well as some reasons for why they would be correlated. Also, of course, there\u2019s AIs working together in big units in general, because you want your AIs to work well together, and so on.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Just to get a calibration, what percentage chance do you give of, not just this scenario but overall through all the scenarios, some kind of thing which if we\u2019re around to recognize it as such, we would categorize as takeover by 2040?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>By 2040? Let\u2019s see. Maybe around 35 or 40%?<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Pretty high.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Yeah, it\u2019s pretty high. I should note that another way you could get this reward-seeking takeover is the AIs are deployed inside an AI company. The way the takeover happens is that they poison the values of the next model, and that persists going forward for forever, or until those AIs are deployed in the world and take over. That might mean that a smaller number of AIs have to coordinate, because those are just the AIs doing the alignment of the next model.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Okay. I\u2019ll summarize where my head is at, at the end of this conversation. I buy the reward hacking up to extremely destructive effects on society, things like social engineering and blah, blah, blah. I\u2019m more inclined to think that significant acceleration of AI R&amp;D can happen. I\u2019m not sure if I buy the five years in one year. I\u2019m also more inclined now to think reward hacking could continue for a lot longer and, in fact, become much more dangerous. I\u2019m still not on board that takeover seems super likely. But that\u2019s my end-of-episode update.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Cool. Taking a step back, I should also say there are a bunch of different ways this could go. The situation is going to be pretty messy. I think it\u2019s pretty likely that the reason why AI takeover happens is for some weird other quirky reason we didn\u2019t even mention in this conversation. But ultimately, I think a lot of the core thing is just that it\u2019s pretty spooky to have a bajillion really smart AIs running your whole world where you don\u2019t really understand quite what\u2019s going on.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Yeah, I agree with that. Is there anything else that\u2019s worth saying?<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>Another thing I want to note is that I think right now a lot of the arguments for misalignment, AI takeover, all this crazy shit going down in the future, are illegible conceptual arguments that are extremely deep in the weeds and complicated and hard to adjudicate. Which means that maybe I\u2019m getting a bunch of it wrong because it\u2019s really hard, and I\u2019m trying to be uncertain. Obviously here I presented some specific scenarios, but those are not exhaustive. Probably the thing that actually happens is some more messy, confusing situation.<\/span><\/p>\n<p><span>But it also means that over time, as we get more empirical evidence and better understand the nature of AI systems, it\u2019ll be easier to adjudicate a bunch of disagreements. It\u2019ll be more obvious what\u2019s going to happen. At least I hope. Maybe the AIs will be able to help us with the epistemics and understanding what\u2019s going on, if we can actually align them well so they try to help us.<\/span><\/p>\n<p><span>Even if the arguments are complicated now, this would have been even harder six years ago, even though the shape of the arguments would have looked broadly pretty similar. Hopefully before it\u2019s too late, this whole thing will become more crisp and clear, and we can all notice these problems and intervene.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>When you first learn to drive, you\u2019re taught that instead of looking right in front of your wheel, you\u2019ll have a much more stable ride if you look out at the horizon. I think there\u2019s a similar situation here. I think you\u2019re right. If you had said five years ago that we would have AIs that are proving math conjectures, and making art, and earning tens or hundreds of billions of dollars of wages, but also egregiously cheating in ways that break laws and committing felonies, it would have been so wild.<\/span><\/p>\n<p><span>You might have been inclined at the time to talk more about the extremely practical, direct consequences of GPT-2 or something. But even though you obviously couldn\u2019t have foreseen a lot of the specific details, the general shape of things you could have started to reason about even then. But it would have been hard to do so, and so I do feel quite confused.<\/span><\/p>\n<p><span>One thing I\u2019ve been thinking about with the podcast is that the important thing is to have the conversation now the way you would have hoped you would have been talking back in 2016 about AIs like the present ones, rather than talking about rando bullshit. I don\u2019t know what the topic of conversation was in 2016. I think in maybe 10 years we\u2019ll wish we had been talking about the industrial explosion and the nature of AIs that are hard to monitor, and so on. So okay, I\u2019ll start thinking about it.<\/span><\/p>\n<p><strong><span>Ryan Greenblatt<\/span><\/strong><\/p>\n<p><span>I hope that the world thinks about this in time and catches up. I hope that the responses are good instead of bad. I don\u2019t know how optimistic I am overall, but there\u2019s good stuff to do.<\/span><\/p>\n<p><strong><span>Dwarkesh Patel<\/span><\/strong><\/p>\n<p><span>Cool. Thanks, Ryan.<\/span><\/p>\n<\/div>\n<p><a href=\"https:\/\/www.dwarkesh.com\/p\/ryan-greenblatt?utm_source=tldrai\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Had Ryan Greenblatt on to discuss\/debate recursive self-improvement. This might be the most important question in the world right now \u2013 whether within a year or so of achieving human-level intelligence, you slingshot towards having 10s of billions of superintelligences, each of which is dramatically more competent than human experts across all fields. I\u2019ve historically [&hellip;]<\/p>\n","protected":false},"author":16,"featured_media":23236,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23235","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23235","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23235"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23235\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23236"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23235"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23235"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23235"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}