{"id":23652,"date":"2026-09-02T00:06:08","date_gmt":"2026-09-02T00:06:08","guid":{"rendered":"https:\/\/scannn.com\/what-is-agentic-testing-goals-instead-of-recorded-steps\/"},"modified":"2026-09-02T00:06:08","modified_gmt":"2026-09-02T00:06:08","slug":"what-is-agentic-testing-goals-instead-of-recorded-steps","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/what-is-agentic-testing-goals-instead-of-recorded-steps\/","title":{"rendered":"What is Agentic Testing? Goals Instead of Recorded Steps"},"content":{"rendered":"\n<div dir=\"auto\">\n<p>Agentic testing hands an agent the goal instead of the steps, and lets it work out how to get there against whatever interface your system exposes. It finds its own way, invents cases nobody wrote, and survives the renames that break your suite. Let it also judge the answers, and a model decides what ships.<\/p>\n<ul>\n<li>\n<p><strong>Agentic testing means you state the goal and an agent works out the steps.<\/strong><span> You write \u201ca coding agent\u2019s pull request builds and passes its tests\u201d. The agent opens the PR, runs what it finds, and reports whether the claim held.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>A scripted test is a recorded route. An agentic test is a destination.<\/strong><span> The recording breaks the moment somebody moves a turn. The destination survives, because the agent finds a new way there.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>The loop is simple: look, act, look again.<\/strong><span> Read the current state, take the action that moves toward the goal, read the state again, repeat until the goal holds or the agent runs out of moves.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>Meta ran this at scale and only a quarter of the output was worth keeping.<\/strong><span> Of everything TestGen-LLM generated, 75% compiled, 57% passed reliably, 25% raised coverage. It works anyway, because three automatic gates throw the rest away.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>One job never gets handed over: deciding whether the answer was right.<\/strong><span> The agent can find any route to the destination. It cannot tell you the destination was the right one. Hand that call to a model and your release gate answers differently on Tuesday than it did on Monday.<\/span><\/p>\n<\/li>\n<\/ul>\n<p>Let\u2019s get into it.<\/p>\n<div data-attrs=\"{&quot;url&quot;:&quot;https:\/\/theaiengineer.substack.com\/p\/what-is-agentic-testing-fa2?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share&quot;,&quot;text&quot;:&quot;Share&quot;}\" data-component-name=\"CaptionedButtonToDOM\" class=\"captioned-button-wrap\">\n<div class=\"preamble\">\n<p class=\"cta-caption\"><span> <\/span><em>New here? Every Tuesday, one AI engineering concept taken apart like this. Subscribe free.<\/em><\/p>\n<\/div>\n<p data-attrs=\"{&quot;url&quot;:&quot;https:\/\/theaiengineer.substack.com\/p\/what-is-agentic-testing-fa2?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share&quot;,&quot;text&quot;:&quot;Share&quot;}\" data-component-name=\"ButtonCreateButton\" class=\"button-wrapper\"><a href=\"https:\/\/theaiengineer.substack.com\/p\/what-is-agentic-testing-fa2?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share\" class=\"button primary\"><span>Share<\/span><\/a><\/p>\n<\/div>\n<p>Your pipeline goes red at 9:14 on a Tuesday. Forty tests passed. One failed, and it waited thirty full seconds for something that was never going to arrive.<\/p>\n<p>You open the app. The feature works. You open the diff. Somebody renamed a field in a styling PR, and your test was pointing at the old name.<\/p>\n<p>Twenty minutes later you have changed one string, pushed, and gone back to work. Nothing was broken and nothing was fixed. Meanwhile three flows nobody ever wrote a test for shipped last week, and your suite said nothing at all.<\/p>\n<p>You have hit the wall every team with a real suite hits: your tests know the route, and they have no idea where you were going. An agent can be told the destination. What that actually buys you, and what it quietly costs, is the rest of this.<\/p>\n<p>To see why agents matter here, you need to see what a test has always been.<\/p>\n<p>Every test you have ever maintained is a recording: you perform the work once, write down what you did and what you expected back, and the machine repeats it forever. It holds exactly two things, and both freeze the moment you type them.<\/p>\n<p><span>The first is the <\/span><strong>locator<\/strong><span>, the name of the thing to act on: a button\u2019s <\/span><code>data-testid<\/code><span>, a route like <\/span><code>POST \/repos\/{owner}\/{repo}\/pulls<\/code><span>, an import path like <\/span><code>from runner import run_tests<\/code><span>.<\/span><\/p>\n<p><span>The second is the <\/span><strong>oracle<\/strong><span>, the line that decides whether what came back was correct: <\/span><code>assert pr.mergeable is True<\/code><span> is an oracle, and so is <\/span><code>assert response.status == 200<\/code><span>.<\/span><\/p>\n<p>Freezing both is the whole point. A frozen recording runs fast, costs nothing per run, and answers identically every time, which is the only reason your release gate can be a green tick that means anything.<\/p>\n<p><span>Declare the destination instead of recording the route and you need something that can work out the route on the day. That something is an <\/span><strong><a href=\"https:\/\/theaiengineer.substack.com\/p\/what-is-an-ai-agent\">agent<\/a><\/strong><span>.<\/span><\/p>\n<p>That 9:14 failure was not bad luck. It is what the frozen recording costs you, and it costs you in three places.<\/p>\n<p><strong>The specific problems:<\/strong><\/p>\n<ol>\n<li>\n<p><strong>The locator holds a name, and a name is all it holds.<\/strong><span> Rename the attribute, move the route, split the module in two, and the recording breaks while the feature works. Every engineer reading this has paid that tax more than once.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>The oracle only covers what somebody thought of.<\/strong><span> The build that fails only when two flags are set, the request that arrives mid-session-refresh, the diff that quietly touches a second service: none of them fail, because none of them run.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>Neither half knows what the user wanted.<\/strong><span> Nowhere in a recording does anybody write down what the system is supposed to do. The steps are there and the check is there, and the intent between them is nowhere.<\/span><\/p>\n<\/li>\n<\/ol>\n<div class=\"captioned-image-container\">\n<figure><a target=\"_blank\" href=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!FPT4!,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79acf75d-9ae5-4ed5-95aa-38aec35aee1f_1180x620.png\" data-component-name=\"Image2ToDOM\" class=\"image-link image2 is-viewable-img can-restack\"><\/p>\n<div class=\"image2-inset\"><picture><source type=\"image\/webp\" srcset=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!FPT4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79acf75d-9ae5-4ed5-95aa-38aec35aee1f_1180x620.png 424w, https:\/\/substackcdn.com\/image\/fetch\/$s_!FPT4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79acf75d-9ae5-4ed5-95aa-38aec35aee1f_1180x620.png 848w, https:\/\/substackcdn.com\/image\/fetch\/$s_!FPT4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79acf75d-9ae5-4ed5-95aa-38aec35aee1f_1180x620.png 1272w, https:\/\/substackcdn.com\/image\/fetch\/$s_!FPT4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79acf75d-9ae5-4ed5-95aa-38aec35aee1f_1180x620.png 1456w\" sizes=\"100vw\"\/><\/picture><\/div>\n<p><\/a><\/figure>\n<\/div>\n<blockquote>\n<p><strong> Confusion Alert:<\/strong><span> \u201cBut we have self-healing locators.\u201d Your framework probably calls them selectors. Self-healing saves you the fix and buys you a new problem: it repoints a broken locator at whatever element is doing that work now, so a regression that hands the work to a different element looks the same to it. It patches, and your suite goes green over a broken feature.<\/span><\/p>\n<\/blockquote>\n<p>So how do you hand over the destination and let something else work out the route? That is agentic testing.<\/p>\n<p><span>Instead of replaying the recording, you hand the agent the goal and put it in a <\/span><strong><a href=\"https:\/\/theaiengineer.substack.com\/p\/what-is-agent-prompt-engineering\">loop<\/a><\/strong><span>.<\/span><\/p>\n<p><strong>The loop: look, act, look again.<\/strong><span> The agent reads the current state of the system, picks the action that moves toward the goal, takes it, and reads the state again. It exits when the goal holds or when it runs out of moves.<\/span><\/p>\n<p>Walk the pull request through it. The agent looks and finds the build has not started, so it triggers one. It looks again, sees one check red on a flaky integration test, and reruns it. It looks a third time, sees green, and exits. No step in that sequence was written down anywhere.<\/p>\n<p><strong>The three jobs.<\/strong><span> Hand an agent a goal and it has to do three things you have been doing by hand:<\/span><\/p>\n<ol>\n<li>\n<p><strong>Explore.<\/strong><span> It works through the system, follows what it finds, and produces a plan naming the cases worth covering. You get the map before anybody writes test code, and it includes paths nobody on the team thought to list.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>Generate.<\/strong><span> Driving the running system, it watches what responds and writes the cases out as ordinary test files you can read and commit.<\/span><\/p>\n<\/li>\n<li>\n<p><strong>Repair.<\/strong><span> When a test fails, it reruns it, reads the system as it stands now, and repoints the broken step at whatever now does what the old target did.<\/span><\/p>\n<\/li>\n<\/ol>\n<p><span>None of the three jobs works from a picture. On a web app the agent reads the <\/span><strong>accessibility tree<\/strong><span>, the same structured description a screen reader consumes, listing every element with its role, its name and its state. On an API it reads the schema. In a codebase it reads the signatures and the call graph. That is why it can still find the Merge button, or the endpoint that opens a pull request, after somebody rewrites the markup around it.<\/span><\/p>\n<div class=\"captioned-image-container\">\n<figure><a target=\"_blank\" href=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png\" data-component-name=\"Image2ToDOM\" class=\"image-link image2 is-viewable-img can-restack\"><\/p>\n<div class=\"image2-inset\"><picture><source type=\"image\/webp\" srcset=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 424w, https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 848w, https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 1272w, https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 1456w\" sizes=\"100vw\"\/><img decoding=\"async\" src=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png\" width=\"1200\" height=\"570.9677419354839\" data-attrs=\"{&quot;src&quot;:&quot;https:\/\/substack-post-media.s3.amazonaws.com\/public\/images\/cc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:590,&quot;width&quot;:1240,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:123978,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image\/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https:\/\/theaiengineer.substack.com\/i\/212186997?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}\" alt=\"\" title=\"\" srcset=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 424w, https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 848w, https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 1272w, https:\/\/substackcdn.com\/image\/fetch\/$s_!mGtJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc1f5bb5-a6de-4b86-8077-db0ac2ebf0f9_1240x590.png 1456w\" sizes=\"auto, 100vw\" loading=\"lazy\" class=\"sizing-large\"\/><\/picture><\/div>\n<p><\/a><\/figure>\n<\/div>\n<p><span>These are not my names for the jobs. <\/span><strong><a href=\"https:\/\/theaiengineer.substack.com\/p\/ai-coding-tools-what-changed-in-the\">Coding agents<\/a><\/strong><span> already run this loop against a repo, with the terminal and the test runner as their tools, and the test-focused ones split it into one agent per job.<\/span><\/p>\n<p><span>I generated one implementation\u2019s definitions to check. The repair agent\u2019s tool list carries no screenshot tool at all, so the job that re-finds a moved element runs on the structure alone, over <\/span><strong><a href=\"https:\/\/theaiengineer.substack.com\/p\/what-is-mcp\">MCP<\/a><\/strong><span>, the protocol that gives an agent tools in the first place.<\/span><\/p>\n<blockquote>\n<p><strong> Deeper Look:<\/strong><span> Repair agents ship with a documented give-up condition, and it is worth knowing before you run one. When the agent concludes your app broke rather than its own locator, it does not fail the build. It marks that test skipped and leaves a comment at the failing step. Nobody decided to drop that flow from your coverage. The agent did.<\/span><\/p>\n<\/blockquote>\n<p><strong>Now notice what those three jobs leave alone.<\/strong><span> The generator writes the oracle, but it writes it into a file. A person reads that file, and CI runs the line unchanged on every push. The deciding still happens the old way: one frozen check, the same answer every time.<\/span><\/p>\n<p>The other option is to write no check at all. You let the agent look at the result and say whether it was right, which means asking a model the same question on every run and taking whatever answer comes back that time.<\/p>\n<p><strong>So one green run proves nothing.<\/strong><span> Run each test three times and count the passes two ways. If any one of the three runs passes, that counts. Engineers call that score <\/span><strong>pass@k<\/strong><span>. Demand all three and you get <\/span><strong>pass^k<\/strong><span style=\"min-width:0;\" data-state=\"closed\"\/><span>.<\/span><\/p>\n<p>I ran five checks on a pull request three times each, all of them checks on the repo rather than on the agent that opened it. The stub behind them reports status after a delay drawn fresh on every call. Two passed on every run, two failed on every run, and the fifth failed once and passed twice. That single flaky check is the whole gap: the suite scores pass@3 of 0.6 and pass^3 of 0.4. Report the first number and you ship. Report the second and you do not. One test decided it.<\/p>\n<p>Report pass^k. Your users run the flow hundreds of times a day, not once, and a gate that calls a test green after two of its three runs failed is not a gate.<\/p>\n<div class=\"captioned-image-container\">\n<figure><a target=\"_blank\" href=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png\" data-component-name=\"Image2ToDOM\" class=\"image-link image2 is-viewable-img can-restack\"><\/p>\n<div class=\"image2-inset\"><picture><source type=\"image\/webp\" srcset=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 424w, https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 848w, https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 1272w, https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 1456w\" sizes=\"100vw\"\/><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png\" width=\"1200\" height=\"720\" data-attrs=\"{&quot;src&quot;:&quot;https:\/\/substack-post-media.s3.amazonaws.com\/public\/images\/604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:600,&quot;width&quot;:1000,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:79503,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image\/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https:\/\/theaiengineer.substack.com\/i\/212186997?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}\" alt=\"\" title=\"\" srcset=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 424w, https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 848w, https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 1272w, https:\/\/substackcdn.com\/image\/fetch\/$s_!HgiE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F604c9283-ba0b-4d99-a2ed-2424bd45f9ae_1000x600.png 1456w\" sizes=\"auto, 100vw\" loading=\"lazy\" class=\"sizing-large\"\/><\/picture><\/div>\n<p><\/a><\/figure>\n<\/div>\n<div data-attrs=\"{&quot;url&quot;:&quot;https:\/\/theaiengineer.substack.com\/p\/what-is-agentic-testing-fa2?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share&quot;,&quot;text&quot;:&quot;Share&quot;}\" data-component-name=\"CaptionedButtonToDOM\" class=\"captioned-button-wrap\">\n<div class=\"preamble\">\n<p class=\"cta-caption\"> Know someone whose CI is about to start repairing its own tests? Send this.<\/p>\n<\/div>\n<p data-attrs=\"{&quot;url&quot;:&quot;https:\/\/theaiengineer.substack.com\/p\/what-is-agentic-testing-fa2?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share&quot;,&quot;text&quot;:&quot;Share&quot;}\" data-component-name=\"ButtonCreateButton\" class=\"button-wrapper\"><a href=\"https:\/\/theaiengineer.substack.com\/p\/what-is-agentic-testing-fa2?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share\" class=\"button primary\"><span>Share<\/span><\/a><\/p>\n<\/div>\n<p><strong>Meta<\/strong><span> ran TestGen-LLM across Instagram and Facebook, and put every generated test through three checks: (1) does it compile, (2) does it pass reliably across runs, (3) does it raise coverage. Three quarters cleared the first, just over half the second, a quarter the third. Engineers then accepted 73% of what survived all three, and most of its output was thrown away<\/span><span style=\"min-width:0;\" data-state=\"closed\"\/><span>.<\/span><\/p>\n<blockquote>\n<p><strong> Engineering Lesson:<\/strong><span> Build those same three checks into your own pipeline this week. Have the agent open a PR, run all three in CI, and discard the PR automatically when the coverage report does not move.<\/span><\/p>\n<\/blockquote>\n<p><strong>Uber<\/strong><span> runs AutoCover, which now writes about one in nine of all new tests added to their codebase. The same pipeline produces a viable passing test 20% of the time in Java, 40% in Go and 80% in Python. Read that spread as a setup cost: Python needs no build step and mocks anything, Java needs compilation, injected dependencies and a build file edit. Point your first agent wherever your tests are cheapest to run in isolation, then measure that number in your own repo<\/span><span style=\"min-width:0;\" data-state=\"closed\"\/><span>.<\/span><\/p>\n<p><strong>Airbnb<\/strong><span> moved nearly 3,500 test files off Enzyme in six weeks, against an estimate of a year and a half by hand. The pipeline reached 75% in the first four hours and 97% four days later, and retries closed the gap. Every file walks a chain of validation steps, and when a check fails the model gets asked to fix that specific failure and the file goes back through. Most files land inside ten attempts. The long tail took between fifty and a hundred, with prompts growing to 100,000 tokens and up to fifty related files pulled in as context<\/span><span style=\"min-width:0;\" data-state=\"closed\"\/><span>.<\/span><\/p>\n<p><strong>Repair is the part that lies to you.<\/strong><span> It cannot tell a harmless edit from a regression that moved the work somewhere else, so it repoints for both and your test goes green over a broken feature. When it cannot repoint at all it marks the test skipped, and a skipped test reports nothing. Green either way.<\/span><\/p>\n<p><strong>Review is supposed to catch that, and it does not.<\/strong><span> A generator emits tests that look exactly like the ones you write, so a reviewer waves them through as fast as they wave through yours, the way teams do with machine-written <\/span><strong><a href=\"https:\/\/theaiengineer.substack.com\/p\/how-coderabbit-actually-works\">review comments<\/a><\/strong><span>. An assertion that checks the build started, instead of checking the tests inside it passed, looks fine in a diff. Read the assertions first and the steps second.<\/span><\/p>\n<p><strong>And the bill lands per step.<\/strong><span> A recorded suite adds no model cost at all. Let an agent drive CI and 500 tests firing on every commit becomes a line item somebody will ask you about, on the same <\/span><strong><a href=\"https:\/\/theaiengineer.substack.com\/p\/should-you-self-host-inference\">breakeven arithmetic<\/a><\/strong><span> every inference bill has.<\/span><\/p>\n<p>The pitch on the box is that maintenance goes away. It does not. You swap maintenance you can see for maintenance you cannot. A broken locator is boring, cheap, and screams the moment it breaks. An oracle that answers differently on Tuesday than it did on Monday does none of those things.<\/p>\n<p>One version of this earns its place today: the agent at authoring time, the model out of CI. It explores and generates against staging while somebody watches, a reviewer reads the output like any PR, and CI runs committed code with no model in it. That is the shape Meta\u2019s three checks put around a generator nobody trusted.<\/p>\n<div class=\"captioned-image-container\">\n<figure><a target=\"_blank\" href=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png\" data-component-name=\"Image2ToDOM\" class=\"image-link image2 is-viewable-img can-restack\"><\/p>\n<div class=\"image2-inset\"><picture><source type=\"image\/webp\" srcset=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 424w, https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 848w, https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 1272w, https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 1456w\" sizes=\"100vw\"\/><img decoding=\"async\" src=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png\" width=\"1200\" height=\"621.4285714285714\" data-attrs=\"{&quot;src&quot;:&quot;https:\/\/substack-post-media.s3.amazonaws.com\/public\/images\/cab83813-9794-4045-8494-d154fcf80867_1120x580.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:580,&quot;width&quot;:1120,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:109784,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image\/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https:\/\/theaiengineer.substack.com\/i\/212186997?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}\" alt=\"\" title=\"\" srcset=\"https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 424w, https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 848w, https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 1272w, https:\/\/substackcdn.com\/image\/fetch\/$s_!XrxZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcab83813-9794-4045-8494-d154fcf80867_1120x580.png 1456w\" sizes=\"auto, 100vw\" loading=\"lazy\" class=\"sizing-large\"\/><\/picture><\/div>\n<p><\/a><\/figure>\n<\/div>\n<p>Agentic testing does not remove the check. It moves who writes it.<\/p>\n<p>Every argument about this is really an argument about where that move should stop. Stop it at authoring and the agent finds the route while your suite still holds the destination, so green means the same thing on every run. Push it into CI and your gate goes from right by construction to right most of the time.<\/p>\n<p><span> <\/span><em>What broke first when you pointed an agent at your suite? Tell me in the comments.<\/em><\/p>\n<p><strong>Is agentic testing the same as testing an AI agent?<\/strong><\/p>\n<p>No, and the two get filed under the same phrase constantly. Agentic testing points an agent at your software and has it drive the system. Testing an AI agent means measuring a nondeterministic system against saved inputs and checks you wrote in advance, which is an eval. The confusing part is that agentic testing inherits the eval problem, because the moment a model decides pass or fail, your test suite becomes the nondeterministic system.<\/p>\n<p><strong>Does an agentic test need a vision model?<\/strong><\/p>\n<p>Not for the browser case. The agent reads a structured description of the interface. For a web page that is the accessibility tree, listing every element with its role, its name and its state. Coordinate-based clicking exists for the cases the tree cannot reach, and it gives up the tree\u2019s whole advantage. A button keeps its role and its accessible name when someone moves it or renames its test id. Its coordinates change the moment the layout does.<\/p>\n<p><strong>How many times should I run an agentic test before I believe it?<\/strong><\/p>\n<p>At least three, for anything you would block a release on. A single run of a nondeterministic check proves the check passed once. Three runs separate pass@k, where at least one run passed, from pass^k, where all of them did. The tests that clear pass@k and fail pass^k are the ones you cannot ship behind.<\/p>\n<p><strong>Do I still write tests by hand?<\/strong><\/p>\n<p>You still review them, and reviewing is where the assertions get fixed. The generator produces spec files that look like the ones you write, which is exactly why they slide through a diff. The tests you write by hand shrink to the ones where you already know the exact edge case, which is a smaller pile than it used to be.<\/p>\n<\/div>\n<p><a href=\"https:\/\/theaiengineer.substack.com\/p\/what-is-agentic-testing-fa2?utm_source=tldrnewsletter\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Agentic testing hands an agent the goal instead of the steps, and lets it work out how to get there against whatever interface your system exposes. It finds its own way, invents cases nobody wrote, and survives the renames that break your suite. Let it also judge the answers, and a model decides what ships. [&hellip;]<\/p>\n","protected":false},"author":16,"featured_media":23653,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23652","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23652","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23652"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23652\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23653"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23652"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}