{"id":23902,"date":"2026-09-11T18:16:31","date_gmt":"2026-09-11T18:16:31","guid":{"rendered":"https:\/\/scannn.com\/astra-for-coding-why-are-we-doing-this-again\/"},"modified":"2026-09-11T18:16:31","modified_gmt":"2026-09-11T18:16:31","slug":"astra-for-coding-why-are-we-doing-this-again","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/astra-for-coding-why-are-we-doing-this-again\/","title":{"rendered":"Astra for Coding: Why Are We Doing This Again?"},"content":{"rendered":"\n
\n

written on September 07, 2026<\/p>\n

I\u2019m more and more convinced that all of AI engineering is
\nNeijuan<\/a> (\u5185\u5377, meaning curl inwards). In
\nChina it describes a system that demands ever more effort and competition
\nwithout improving output. The way in which it sometimes shows up in the West is
\nthe 996 nonsense<\/a>. The English term for Neijuan is \u201cInvolution\u201d
\nfrom the book Agricultural
\nInvolution<\/a>.
\nAgricultural involution describes the intensification of farming that raises
\nproductivity per square meter while leaving productivity per head unchanged.<\/p>\n

That\u2019s how I feel about AI right now.<\/p>\n

Which brings me to GPT 6 Astra. Astra is by all accounts an incredibly
\nimpressive model. There is really not much I can say against this. It\u2019s
\namazing at computer use, understands images and complex topics, and it\u2019s
\nrelentless in its pursuit of completion. It is absolutely impressive; these
\ntypes of models are going to change the world in one form or another.<\/p>\n

But at least for the moment I don\u2019t know how to work with it for actual software
\nengineering. Since that got quite a bit of attention on Twitter, I figured I
\nmight summarize my thoughts and just share what kind of code comes out of this
\nthing.<\/p>\n

My Slop Factory<\/h2>\n

\u201cArmin, you should run a software factory!\u201d I\u2019ve heard that a few times now, so I figured
\nI might celebrate the release of it by running a little software factory over
\nthe weekend. If everybody builds slop 3D games, then I should do something
\nuseful with it. My software factory was intentionally set up to let the model
\ndecide the how of the workflow entirely. It was free to manage its own context
\nand could maintain its own records in an agent-notes<\/code> folder. Then it spun off
\nsubagents to work on stuff. The goal? What if we had a Python with virtual
\nthreads<\/a> and lexical scoping. And well, I burned a
\nfull reset\u2019s worth of ChatGPT tokens on this which appears to be around 4 billion
\ntokens. 35 hours later, the factory has delivered absolutely nothing of value
\nand also not taught me anything about how to operate a better one.<\/p>\n

But it produced a lot of code and input prompts, and so there is stuff I was able
\nto study. And well, it shows behavior that I\u2019m not used to with Sol and earlier
\nOpenAI models . I have since encountered the same issues with regular
\nprogramming with Astra, so it\u2019s not a result of just the factory.<\/p>\n

I think I\u2019m suspecting something is going \u201cwrong\u201d in the training process. The
\nmodel is greatly rewarded for succeeding on long-horizon tasks, but presumably there
\nis very little punishing going on for \u201cshitty code.\u201d The apparent result is
\nthat Astra is amazing at producing 3D stuff
\n<\/a> and it
\ncan keep going for a very long time, coming up with its own work in the process.
\nI had it do quite a bit of reverse engineering of my robot vacuum in ways that
\nwere quite impressive. So it\u2019s definitely cool!<\/p>\n

Codegolf Tool Calls<\/h2>\n

The first issue I have with Astra comes from the type of code that it uses for
\ntool calls. Codex increasingly has been relying on \u201cjust bash\u201d to do more and
\nmore operations. For a few versions now the original Codex harness just uses
\nsed<\/code> and other tools to read files. You just usually can\u2019t see them because Codex
\nparses the bash
\ncommands<\/a>
\nand hides them if it recognizes them. But Astra \u2026 really loves Python? That is
\nnot much of a surprise because even older OpenAI models had a tendency to
\nsometimes use on-demand Python code to read and manipulate files at times, but
\nAstra does it really quite excessively for me.<\/p>\n

Now here is an important disclaimer: this project is very meta<\/em> here because I
\nworked
\non<\/em> the CPython interpreter. But I can assure you that I have seen this model
\ndo weird Python things even in TypeScript code in Pi. But I have the most
\nevidence of odd code from when I had the thing work over the weekend
\nwith zero<\/em> oversight from my slop factory.<\/p>\n

That it writes Python is not interesting; the type of Python is interesting, and
\nI collected some outputs for you to gloss over.<\/p>\n

\nPython string splicing to edit C code<\/summary>\n

In the Codex harness I found multiple cases where subagents resorted fully to
\nmanual string manipulation with Python instead of using the patch tool.<\/p>\n

\n
python3<\/span> -<\/span> <<<\/span>'PY'<\/span>\nfrom<\/span> <\/span>pathlib<\/span> <\/span>import<\/span> Path<\/span>\np<\/span>=<\/span>Path<\/span>(<\/span>'Include\/internal\/pycore_intrinsics.h'<\/span>);<\/span>s<\/span>=<\/span>p<\/span>.<\/span>read_text<\/span>()<\/span>.<\/span>replace<\/span>(<\/span>'#define MAX_INTRINSIC_1                         14'<\/span>,<\/span>'#define INTRINSIC_RETAIN_ANNOTATION_CELLS        15<\/span>\\n\\n<\/span>#define MAX_INTRINSIC_1                         15'<\/span>);<\/span>p<\/span>.<\/span>write_text<\/span>(<\/span>s<\/span>)<\/span>\np<\/span>=<\/span>Path<\/span>(<\/span>'Python\/intrinsics.c'<\/span>);<\/span>s<\/span>=<\/span>p<\/span>.<\/span>read_text<\/span>();<\/span>idx<\/span>=<\/span>s<\/span>.<\/span>index<\/span>(<\/span>'#define INTRINSIC_FUNC_ENTRY'<\/span>);<\/span>s<\/span>=<\/span>s<\/span>[:<\/span>idx<\/span>]<\/span>+<\/span>'''\/* Hold every old cell until the compiler has published the entire site's new<\/span>\n   capture. A replaced cell's finalizer may reenter module __annotate__. *\/<\/span>\nstatic PyObject *<\/span>\nretain_annotation_cells(PyThreadState *tstate, PyObject *holders)<\/span>\n{<\/span>\n    if (!PyTuple_CheckExact(holders)) {<\/span>\n        PyErr_SetString(PyExc_TypeError, \"annotation holders must be a tuple\");<\/span>\n        return NULL;<\/span>\n    }<\/span>\n    Py_ssize_t size = PyTuple_GET_SIZE(holders);<\/span>\n    PyObject *previous = PyTuple_New(size);<\/span>\n    if (previous == NULL) return NULL;<\/span>\n    for (Py_ssize_t i = 0; i < size; i++) {<\/span>\n        PyObject *holder = PyTuple_GET_ITEM(holders, i);<\/span>\n        if (!PyCell_Check(holder)) {<\/span>\n            Py_DECREF(previous);<\/span>\n            PyErr_SetString(PyExc_TypeError, \"annotation holder must be a cell\");<\/span>\n            return NULL;<\/span>\n        }<\/span>\n        PyObject *cell = PyCell_Get(holder);<\/span>\n        PyTuple_SET_ITEM(previous, i, cell == NULL ? Py_NewRef(Py_None) : cell);<\/span>\n    }<\/span>\n    return previous;<\/span>\n}<\/span>\n\n'''<\/span> +<\/span>s<\/span>[<\/span>idx<\/span>:];<\/span>s<\/span>=<\/span>s<\/span>.<\/span>replace<\/span>(<\/span>'    INTRINSIC_FUNC_ENTRY(INTRINSIC_AWAIT_BLOCK, await_block)'<\/span>,<\/span>'    INTRINSIC_FUNC_ENTRY(INTRINSIC_AWAIT_BLOCK, await_block)<\/span>\\n<\/span>    INTRINSIC_FUNC_ENTRY(INTRINSIC_RETAIN_ANNOTATION_CELLS, retain_annotation_cells)'<\/span>);<\/span>p<\/span>.<\/span>write_text<\/span>(<\/span>s<\/span>)<\/span>\np<\/span>=<\/span>Path<\/span>(<\/span>'Python\/codegen.c'<\/span>);<\/span>s<\/span>=<\/span>p<\/span>.<\/span>read_text<\/span>();<\/span>idx<\/span>=<\/span>s<\/span>.<\/span>index<\/span>(<\/span>'static int<\/span>\\n<\/span>codegen_annassign('<\/span>);<\/span>s<\/span>=<\/span>s<\/span>[:<\/span>idx<\/span>]<\/span>+<\/span>'''static int<\/span>\ncodegen_retain_annotation_cells(compiler *c, location loc, PyObject *captures)<\/span>\n{<\/span>\n    Py_ssize_t pos = 0;<\/span>\n    PyObject *binding, *holder;<\/span>\n    while (PyDict_Next(captures, &pos, &binding, &holder)) {<\/span>\n        ADDOP_NAME(c, loc, LOAD_CLOSURE, holder, cellvars);<\/span>\n    }<\/span>\n    ADDOP_I(c, loc, BUILD_TUPLE, PyDict_GET_SIZE(captures));<\/span>\n    ADDOP_I(c, loc, CALL_INTRINSIC_1, INTRINSIC_RETAIN_ANNOTATION_CELLS);<\/span>\n    return SUCCESS;<\/span>\n}<\/span>\n\n'''<\/span> +<\/span>s<\/span>[<\/span>idx<\/span>:]<\/span>\na<\/span>=<\/span>s<\/span>.<\/span>index<\/span>(<\/span>'                if (conditional_annotation_index != NULL) {'<\/span>,<\/span>s<\/span>.<\/span>index<\/span>(<\/span>'codegen_annassign(compiler *c'<\/span>))<\/span>\nb<\/span>=<\/span>s<\/span>.<\/span>index<\/span>(<\/span>'                if (captures != NULL) {'<\/span>,<\/span>a<\/span>)<\/span>\n# Move lookup before conditional registration and retain old cells before anything changes.<\/span>\nlookupstart<\/span>=<\/span>s<\/span>.<\/span>index<\/span>(<\/span>'                PyObject *captures = _PyCompile_AnnotationCaptures'<\/span>,<\/span>a<\/span>)<\/span>\nlookup<\/span>=<\/span>s<\/span>[<\/span>lookupstart<\/span>:<\/span>b<\/span>]<\/span>.<\/span>replace<\/span>(<\/span>'                    return ERROR;'<\/span>,<\/span>'                    Py_XDECREF(conditional_annotation_index); return ERROR;'<\/span>)<\/span>\ns<\/span>=<\/span>s<\/span>[:<\/span>lookupstart<\/span>]<\/span>+<\/span>s<\/span>[<\/span>b<\/span>:]<\/span>\nsetup<\/span>=<\/span>lookup<\/span>+<\/span>'''                if (captures != NULL && codegen_retain_annotation_cells(c, loc, captures) < 0) {<\/span>\n                    Py_XDECREF(conditional_annotation_index); return ERROR;<\/span>\n                }<\/span>\n'''<\/span>\ns<\/span>=<\/span>s<\/span>[:<\/span>a<\/span>]<\/span>+<\/span>setup<\/span>+<\/span>s<\/span>[<\/span>a<\/span>:]<\/span>\nneedle<\/span>=<\/span>'                        ADDOP_NAME(c, loc, STORE_DEREF, holder, cellvars);<\/span>\\n<\/span>                    }<\/span>\\n<\/span>                }'<\/span>\ns<\/span>=<\/span>s<\/span>.<\/span>replace<\/span>(<\/span>needle<\/span>,<\/span>'                        ADDOP_NAME(c, loc, STORE_DEREF, holder, cellvars);<\/span>\\n<\/span>                    }<\/span>\\n<\/span>                    ADDOP(c, loc, POP_TOP); \/* release old cells after full publication *\/<\/span>\\n<\/span>                }'<\/span>,<\/span>1<\/span>);<\/span>p<\/span>.<\/span>write_text<\/span>(<\/span>s<\/span>)<\/span>\np<\/span>=<\/span>Path<\/span>(<\/span>'Include\/internal\/pycore_magic_number.h'<\/span>);<\/span>s<\/span>=<\/span>p<\/span>.<\/span>read_text<\/span>()<\/span>.<\/span>replace<\/span>(<\/span>'    Python 3.16a1 3709 (Checked deferred annotation closure capture)'<\/span>,<\/span>'    Python 3.16a1 3709 (Checked deferred annotation closure capture)<\/span>\\n<\/span>    Python 3.16a1 3710 (Retain replaced annotation captures until publication)'<\/span>)<\/span>.<\/span>replace<\/span>(<\/span>'#define PYC_MAGIC_NUMBER 3709'<\/span>,<\/span>'#define PYC_MAGIC_NUMBER 3710'<\/span>);<\/span>p<\/span>.<\/span>write_text<\/span>(<\/span>s<\/span>)<\/span>\np<\/span>=<\/span>Path<\/span>(<\/span>'Lib\/test\/test_block_annotation_captures.py'<\/span>);<\/span>s<\/span>=<\/span>p<\/span>.<\/span>read_text<\/span>();<\/span>idx<\/span>=<\/span>s<\/span>.<\/span>index<\/span>(<\/span>'    def test_typing_consumers'<\/span>);<\/span>s<\/span>=<\/span>s<\/span>[:<\/span>idx<\/span>]<\/span>+<\/span>'''    def test_replaced_cell_finalizer_sees_complete_site_publication(self):<\/span>\n        module=execute(\"\"\"<\/span>\\\\<\/span>\n            events=[]<\/span>\n            class V:<\/span>\n                def __init__(self,n): self.n=n<\/span>\n                def __del__(self):<\/span>\n                    if self.n == 0: events.append(__annotate__(1))<\/span>\n            for i in range(2):<\/span>\n                x=V(i) # bind x y<\/span>\n                y=i<\/span>\n                value:(x.n,y)<\/span>\n        \"\"\")<\/span>\n        self.assertEqual(module.events,[{'value':(1,1)}])<\/span>\n        self.assertEqual(module.__annotate__(1),{'value':(1,1)})<\/span>\n\n'''<\/span> +<\/span>s<\/span>[<\/span>idx<\/span>:];<\/span>p<\/span>.<\/span>write_text<\/span>(<\/span>s<\/span>)<\/span>\nPY<\/span>\nmake<\/span> -<\/span>j1<\/span> ><\/span> \/<\/span>tmp<\/span>\/<\/span>block<\/span>-<\/span>annotations<\/span>-<\/span>build7<\/span>.<\/span>log<\/span> 2<\/span>>&<\/span>1<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
\nSocket codegolf<\/summary>\n
In the middle of a conversation the agent ran into \u201cBad file descriptor\u201d on a test and Astra decided it needs to see if file descriptors can be passed over Unix sockets on macOS in a super compressed manner:<\/p>\n
\n
\/<\/span>usr<\/span>\/<\/span>bin<\/span>\/<\/span>python3<\/span> -<\/span> <<<\/span>'PY'<\/span>\nimport<\/span> <\/span>socket<\/span>,<\/span>os<\/span>,<\/span>array<\/span>\nfor<\/span> into<\/span> in<\/span> (<\/span>False<\/span>,<\/span>True<\/span>):<\/span>\n a<\/span>,<\/span>b<\/span>=<\/span>socket<\/span>.<\/span>socketpair<\/span>();<\/span>fd<\/span>=<\/span>os<\/span>.<\/span>open<\/span>(<\/span>os<\/span>.<\/span>devnull<\/span>,<\/span>os<\/span>.<\/span>O_RDONLY<\/span>);<\/span>b<\/span>.<\/span>sendmsg<\/span>([<\/span>b<\/span>'c'<\/span>],[(<\/span>socket<\/span>.<\/span>SOL_SOCKET<\/span>,<\/span>socket<\/span>.<\/span>SCM_RIGHTS<\/span>,<\/span>array<\/span>.<\/span>array<\/span>(<\/span>'i'<\/span>,[<\/span>fd<\/span>]))]);<\/span>print<\/span>(<\/span>'fds'<\/span>,<\/span>a<\/span>.<\/span>fileno<\/span>(),<\/span>b<\/span>.<\/span>fileno<\/span>(),<\/span>fd<\/span>)<\/span>\n if<\/span> into<\/span>:<\/span>r<\/span>=<\/span>a<\/span>.<\/span>recvmsg_into<\/span>([<\/span>bytearray<\/span>(<\/span>1<\/span>),<\/span>bytearray<\/span>(),<\/span>bytearray<\/span>(<\/span>19<\/span>)],<\/span>socket<\/span>.<\/span>CMSG_SPACE<\/span>(<\/span>4<\/span>),<\/span>socket<\/span>.<\/span>MSG_PEEK<\/span>|<\/span>socket<\/span>.<\/span>MSG_DONTWAIT<\/span>)<\/span>\n else<\/span>:<\/span>r<\/span>=<\/span>a<\/span>.<\/span>recvmsg<\/span>(<\/span>20<\/span>,<\/span>socket<\/span>.<\/span>CMSG_SPACE<\/span>(<\/span>4<\/span>),<\/span>socket<\/span>.<\/span>MSG_PEEK<\/span>|<\/span>socket<\/span>.<\/span>MSG_DONTWAIT<\/span>)<\/span>\n print<\/span>(<\/span>'peek'<\/span>,<\/span>r<\/span>,<\/span>flush<\/span>=<\/span>True<\/span>)<\/span>\n rights<\/span>=<\/span>array<\/span>.<\/span>array<\/span>(<\/span>'i'<\/span>,<\/span>r<\/span>[<\/span>1<\/span>][<\/span>0<\/span>][<\/span>2<\/span>]);<\/span>print<\/span>(<\/span>'rights'<\/span>,<\/span>rights<\/span>,<\/span>flush<\/span>=<\/span>True<\/span>)<\/span>\n for<\/span> f<\/span> in<\/span> rights<\/span>:<\/span>\n  try<\/span>:<\/span> print<\/span>(<\/span>'stat'<\/span>,<\/span>os<\/span>.<\/span>fstat<\/span>(<\/span>f<\/span>))<\/span>\n  except<\/span> Exception<\/span> as<\/span> e<\/span>:<\/span> print<\/span>(<\/span>'error'<\/span>,<\/span>e<\/span>)<\/span>\n r<\/span>=<\/span>a<\/span>.<\/span>recvmsg<\/span>(<\/span>20<\/span>,<\/span>socket<\/span>.<\/span>CMSG_SPACE<\/span>(<\/span>4<\/span>),<\/span>socket<\/span>.<\/span>MSG_DONTWAIT<\/span>);<\/span>print<\/span>(<\/span>'consumed'<\/span>,<\/span>r<\/span>,<\/span>flush<\/span>=<\/span>True<\/span>)<\/span>\n a<\/span>.<\/span>close<\/span>();<\/span>b<\/span>.<\/span>close<\/span>();<\/span>os<\/span>.<\/span>close<\/span>(<\/span>fd<\/span>)<\/span>\nPY<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
\nPython for agent notes patching<\/summary>\n
The agent notes were rather consistently updated with Python:<\/p>\n
\n
python3<\/span> -<\/span> <<<\/span>'PY'<\/span>\nfrom<\/span> <\/span>pathlib<\/span> <\/span>import<\/span> Path<\/span>\np<\/span>=<\/span>Path<\/span>(<\/span>'agent-notes\/live\/block-with-bindings.md'<\/span>);<\/span>s<\/span>=<\/span>p<\/span>.<\/span>read_text<\/span>()<\/span>.<\/span>replace<\/span>(<\/span>'  has117\/77\/zero failures.'<\/span>,<\/span> '  has117\/77\/zero failures; existing bundled Expat unreachable-fallthrough<\/span>\\n<\/span>  warnings are the only optimized warnings.'<\/span>)<\/span>\n# Keep the final evidence readable without rewriting historical parent requirements.<\/span>\nfor<\/span> a<\/span>,<\/span>b<\/span> in<\/span> [(<\/span>'all328'<\/span>,<\/span>'all 328'<\/span>),(<\/span>'pass31'<\/span>,<\/span>'pass 31'<\/span>),(<\/span>'pass all328'<\/span>,<\/span>'pass all 328'<\/span>),(<\/span>'pass,9.2s'<\/span>,<\/span>'pass, 9.2s'<\/span>),(<\/span>'log`,210'<\/span>,<\/span>'log`, 210'<\/span>),(<\/span>'log`,5,731'<\/span>,<\/span>'log`, 5,731'<\/span>),(<\/span>'log`:18\/18'<\/span>,<\/span>'log`: 18\/18'<\/span>),(<\/span>'pass,88'<\/span>,<\/span>'pass, 88'<\/span>),(<\/span>'pass,90'<\/span>,<\/span>'pass, 90'<\/span>),(<\/span>'skips,1m'<\/span>,<\/span>'skips, 1m'<\/span>),(<\/span>'all6,280'<\/span>,<\/span>'all 6,280'<\/span>),(<\/span>'has117'<\/span>,<\/span>'has 117'<\/span>)]:<\/span> s<\/span>=<\/span>s<\/span>.<\/span>replace<\/span>(<\/span>a<\/span>,<\/span>b<\/span>)<\/span>\ns<\/span> +=<\/span> '<\/span>\\n<\/span>Key source review: Python\/symtable.c:603 (discovery), :3985 (sequential header traversal),<\/span>\\n<\/span>Python\/codegen.c:3488 (source-only exclusion), :5836 (publication), :5853 (normal\/<\/span>\\n<\/span>unwind reference cleanup), :5925\/:6037 (enter-protected target setup).<\/span>\\n<\/span>'<\/span>\np<\/span>.<\/span>write_text<\/span>(<\/span>s<\/span>)<\/span>\nfor<\/span> name<\/span> in<\/span> (<\/span>'STATE.md'<\/span>,<\/span>'build-and-test.md'<\/span>):<\/span>\n p<\/span>=<\/span>Path<\/span>(<\/span>'agent-notes\/live'<\/span>)<\/span>\/<\/span>name<\/span>;<\/span>s<\/span>=<\/span>p<\/span>.<\/span>read_text<\/span>()<\/span>\n for<\/span> a<\/span>,<\/span>b<\/span> in<\/span> [(<\/span>'build:117'<\/span>,<\/span>'build: 117'<\/span>),(<\/span>'paths.18'<\/span>,<\/span>'paths. 18'<\/span>),(<\/span>'paths.<\/span>\\n<\/span>18'<\/span>,<\/span>'paths.<\/span>\\n<\/span>18'<\/span>),(<\/span>'and210'<\/span>,<\/span>'and 210'<\/span>),(<\/span>'pass5,731'<\/span>,<\/span>'pass 5,731'<\/span>),(<\/span>'All6,280'<\/span>,<\/span>'All 6,280'<\/span>),(<\/span>'failures,31'<\/span>,<\/span>'failures, 31'<\/span>),(<\/span>'in<\/span>\\n<\/span>115s'<\/span>,<\/span>'in<\/span>\\n<\/span>115s'<\/span>),(<\/span>'have117'<\/span>,<\/span>'have 117'<\/span>),(<\/span>'paths.<\/span>\\n<\/span>18'<\/span>,<\/span>'paths.<\/span>\\n<\/span>18'<\/span>),(<\/span>'18 focused,210'<\/span>,<\/span>'18 focused, 210'<\/span>),(<\/span>'and5,731'<\/span>,<\/span>'and 5,731'<\/span>),(<\/span>'all6,280'<\/span>,<\/span>'all 6,280'<\/span>)]:<\/span> s<\/span>=<\/span>s<\/span>.<\/span>replace<\/span>(<\/span>a<\/span>,<\/span>b<\/span>)<\/span>\n p<\/span>.<\/span>write_text<\/span>(<\/span>s<\/span>)<\/span>\nPY<\/span>\ngit<\/span> diff<\/span> --<\/span>check<\/span>\ngit<\/span> add<\/span> -<\/span>u<\/span>\ngit<\/span> add<\/span> Lib<\/span>\/<\/span>test<\/span>\/<\/span>test_block_with_bindings<\/span>.<\/span>py<\/span> agent<\/span>-<\/span>notes<\/span>\/<\/span>done<\/span>\/<\/span>asyncio<\/span>-<\/span>task<\/span>-<\/span>drivers<\/span>.<\/span>md<\/span>\ngit<\/span> diff<\/span> --<\/span>cached<\/span> --<\/span>stat<\/span>\ngit<\/span> commit<\/span> -<\/span>m<\/span> 'Add explicit with and async with header bindings'<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
\nUsing Python to run Node.js<\/summary>\n
In multiple cases it used Python to spawn Node.js on another machine.  It first wrote the script, then it used Bash to run Python, then that program ran Node.js via prlctl<\/code> on my Windows box.<\/p>\n
\n
import<\/span> <\/span>subprocess<\/span>\ncode<\/span> =<\/span> \"const<\/span>{readFileSync}<\/span>=require('fs');const{strict:a}=require('assert');const c=require('C:\/Users\/mitsuhiko\/AppData\/Local\/Temp\/pi-clipboard-threads\/win32-arm64.node');(async()=>{const p=c.getText();a.ok(p instanceof Promise);const saved=await p;const image=await c.getImage();if(image||saved===null){console.log('arm64 async text\/image reads passed; preserving non-text clipboard');return}try{for(const text of ['caf\u00e9 \u65e5\u672c\u8a9e','', 'large'.repeat(200000)]){const p=c.setText(text);a.ok(p instanceof Promise);await p;a.equal(await c.getText(),text);a.equal(await c.getImage(),null)}console.log('Windows ARM64 async Unicode, empty, large text and empty image passed')}finally{await c.setText(saved)}})().catch(e=>{console.error(e);process.exitCode=1})\"<\/span>\nsubprocess<\/span>.<\/span>run<\/span>([<\/span>'prlctl'<\/span>,<\/span> 'exec'<\/span>,<\/span> 'Windows 11'<\/span>,<\/span> '--current-user'<\/span>,<\/span> 'C:<\/span>\\\\<\/span>Program Files<\/span>\\\\<\/span>nodejs<\/span>\\\\<\/span>node.exe'<\/span>,<\/span> '-e'<\/span>,<\/span> code<\/span>],<\/span> check<\/span>=<\/span>True<\/span>)<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
\nPython to run Node.js to run PowerShell<\/summary>\n
Since it was already doing that, it used Bash to run Python to then run Node.js to then use Node.js to invoke PowerShell.<\/p>\n
\n
import<\/span> <\/span>subprocess<\/span>\ncode<\/span> =<\/span> \"process.env.PSModulePath='C:\/Windows\/System32\/WindowsPowerShell\/v1.0\/Modules';require('child_process').spawnSync('powershell.exe',['-NoProfile','-NonInteractive','-ExecutionPolicy','Bypass','-File','C:\/Users\/mitsuhiko\/AppData\/Local\/Temp\/pi-clipboard-threads\/pi-clipboard-windows.ps1'],{stdio:'inherit'});console.log('completed')\"<\/span>\nsubprocess<\/span>.<\/span>run<\/span>([<\/span>'prlctl'<\/span>,<\/span> 'exec'<\/span>,<\/span> 'Windows 11'<\/span>,<\/span> '--current-user'<\/span>,<\/span> 'C:<\/span>\\\\<\/span>Program Files<\/span>\\\\<\/span>nodejs<\/span>\\\\<\/span>node.exe'<\/span>,<\/span> '-e'<\/span>,<\/span> code<\/span>],<\/span> check<\/span>=<\/span>True<\/span>)<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
You can consider this amusing, but I have some questions here.  The first
\nproblem with this is that it\u2019s unreadable for a human.  If you wanna follow
\nalong with what is going on, then good luck.  Particularly once it opts out of
\nusing the edit tools that the harness provides, you\u2019re going to have to resort
\nto using the diff viewer of the final artifacts since it\u2019s almost impossible to
\nvisualize the changes as they happen by reading the code.<\/p>\n
This is not quite as bad in Pi for the most part because I mostly see it editing
\nwith the edit<\/code> tool.  When however goes all bananza with subagents (where the
\nagent believes nobody is looking) it\u2019s resorting to all kinds of increasingly
\nbizarre behavior.  I actually don\u2019t know if the model thinks someone is looking,
\nbut that\u2019s the vibe I\u2019m getting.<\/p>\n
But then it starts doing the same nonsense in code that actually gets committed.
\nI have mostly seen this in tests, but you can also see this for instance when
\nit writes JavaScript or CSS embedded in HTML.  It almost seems like when it\u2019s
\n\u201cone step removed\u201d from regular code, it starts falling into these patterns.<\/p>\n
Here are some unit tests that it created:<\/p>\n
\nComplete disregard for whitespace and indentation<\/summary>\n
\n
def<\/span> <\/span>test_unpack_suspension_and_continuation_close<\/span>(<\/span>self<\/span>):<\/span>\n    from<\/span> <\/span>continuations<\/span> <\/span>import<\/span> Continuation<\/span>,<\/span>suspend<\/span>\n    readers<\/span>=<\/span>[]<\/span>\n    class<\/span> <\/span>Source<\/span>:<\/span>\n        def<\/span> <\/span>__iter__<\/span>(<\/span>self<\/span>):<\/span>\n            yield<\/span> 1<\/span>\n            suspend<\/span>(<\/span>'unpacking'<\/span>)<\/span>\n            yield<\/span> 2<\/span>\n    ns<\/span>=<\/span>execute<\/span>(<\/span>'''<\/span>\n        def run():<\/span>\n            a,b='old-a','old-b'<\/span>\n            readers.append(lambda: (a,b))<\/span>\n            def a,b=Source()<\/span>\n            suspend('published')<\/span>\n    '''<\/span>,<\/span>Source<\/span>=<\/span>Source<\/span>,<\/span>readers<\/span>=<\/span>readers<\/span>,<\/span>suspend<\/span>=<\/span>suspend<\/span>)<\/span>\n    with<\/span> Continuation<\/span>(<\/span>ns<\/span>[<\/span>'run'<\/span>])<\/span> as<\/span> continuation<\/span>:<\/span>\n        self<\/span>.<\/span>assertEqual<\/span>(<\/span>continuation<\/span>.<\/span>resume<\/span>(),<\/span>'unpacking'<\/span>)<\/span>\n        self<\/span>.<\/span>assertEqual<\/span>(<\/span>readers<\/span>[<\/span>0<\/span>](),(<\/span>'old-a'<\/span>,<\/span>'old-b'<\/span>))<\/span>\n        self<\/span>.<\/span>assertEqual<\/span>(<\/span>continuation<\/span>.<\/span>resume<\/span>(),<\/span>'published'<\/span>)<\/span>\n        self<\/span>.<\/span>assertEqual<\/span>(<\/span>readers<\/span>[<\/span>0<\/span>](),(<\/span>1<\/span>,<\/span>2<\/span>))<\/span>\n    class<\/span> <\/span>Value<\/span>:<\/span>pass<\/span>\n    refs<\/span>=<\/span>[];<\/span>frames<\/span>=<\/span>[];<\/span>callbacks<\/span>=<\/span>[]<\/span>\n    ns<\/span>=<\/span>execute<\/span>(<\/span>'''<\/span>\n        def run():<\/span>\n            for def x in [Value()]:<\/span>\n                refs.append(weakref.ref(x))<\/span>\n                frames.append(sys._getframe())<\/span>\n                callbacks.append(lambda: x)<\/span>\n                suspend('body')<\/span>\n    '''<\/span>,<\/span>Value<\/span>=<\/span>Value<\/span>,<\/span>refs<\/span>=<\/span>refs<\/span>,<\/span>frames<\/span>=<\/span>frames<\/span>,<\/span>callbacks<\/span>=<\/span>callbacks<\/span>,<\/span>weakref<\/span>=<\/span>weakref<\/span>,<\/span>sys<\/span>=<\/span>sys<\/span>,<\/span>suspend<\/span>=<\/span>suspend<\/span>)<\/span>\n    with<\/span> Continuation<\/span>(<\/span>ns<\/span>[<\/span>'run'<\/span>])<\/span> as<\/span> continuation<\/span>:<\/span>self<\/span>.<\/span>assertEqual<\/span>(<\/span>continuation<\/span>.<\/span>resume<\/span>(),<\/span>'body'<\/span>)<\/span>\n    self<\/span>.<\/span>assertNotIn<\/span>(<\/span>'x'<\/span>,<\/span>frames<\/span>[<\/span>0<\/span>]<\/span>.<\/span>f_locals<\/span>)<\/span>\n    self<\/span>.<\/span>assertIsNotNone<\/span>(<\/span>refs<\/span>[<\/span>0<\/span>]());<\/span>callbacks<\/span>.<\/span>clear<\/span>();<\/span>self<\/span>.<\/span>assertIsNone<\/span>(<\/span>refs<\/span>[<\/span>0<\/span>]())<\/span>\n\ndef<\/span> <\/span>test_ast_roundtrips_and_future_annotation_unparse<\/span>(<\/span>self<\/span>):<\/span>\n    source<\/span>=<\/span>'callback=lambda {for def a, [b,*rest] in [(1,[2,3])] {return a,b,rest}}'<\/span>\n    tree<\/span>=<\/span>ast<\/span>.<\/span>parse<\/span>(<\/span>source<\/span>);<\/span>node<\/span>=<\/span>tree<\/span>.<\/span>body<\/span>[<\/span>0<\/span>]<\/span>.<\/span>value<\/span>.<\/span>body<\/span>[<\/span>0<\/span>]<\/span>\n    self<\/span>.<\/span>assertIsInstance<\/span>(<\/span>node<\/span>,<\/span>ast<\/span>.<\/span>ForBinding<\/span>)<\/span>\n    self<\/span>.<\/span>assertEqual<\/span>(<\/span>node<\/span>.<\/span>_fields<\/span>,(<\/span>'target'<\/span>,<\/span>'iter'<\/span>,<\/span>'body'<\/span>,<\/span>'orelse'<\/span>,<\/span>'type_comment'<\/span>))<\/span>\n    self<\/span>.<\/span>assertEqual<\/span>(<\/span>node<\/span>.<\/span>lineno<\/span>,<\/span>1<\/span>);<\/span>self<\/span>.<\/span>assertGreater<\/span>(<\/span>node<\/span>.<\/span>end_col_offset<\/span>,<\/span>node<\/span>.<\/span>col_offset<\/span>)<\/span>\n    self<\/span>.<\/span>assertEqual<\/span>(<\/span>ast<\/span>.<\/span>dump<\/span>(<\/span>tree<\/span>),<\/span>ast<\/span>.<\/span>dump<\/span>(<\/span>ast<\/span>.<\/span>parse<\/span>(<\/span>ast<\/span>.<\/span>unparse<\/span>(<\/span>tree<\/span>))))<\/span>\n    ns<\/span>=<\/span>execute<\/span>(<\/span>'from __future__ import annotations<\/span>\\n<\/span>def f(arg: '<\/span>+<\/span>source<\/span>.<\/span>split<\/span>(<\/span>'='<\/span>,<\/span>1<\/span>)[<\/span>1<\/span>]<\/span>+<\/span>'): pass'<\/span>)<\/span>\n    self<\/span>.<\/span>assertEqual<\/span>(<\/span>eval<\/span>(<\/span>ns<\/span>[<\/span>'f'<\/span>]<\/span>.<\/span>__annotations__<\/span>[<\/span>'arg'<\/span>])(),(<\/span>1<\/span>,<\/span>2<\/span>,[<\/span>3<\/span>]))<\/span>\n    tree<\/span>=<\/span>ast<\/span>.<\/span>parse<\/span>(<\/span>'async def f():<\/span>\\n<\/span> async for def x in values: pass # type: ignored<\/span>\\n<\/span>'<\/span>)<\/span>\n    self<\/span>.<\/span>assertIsInstance<\/span>(<\/span>tree<\/span>.<\/span>body<\/span>[<\/span>0<\/span>]<\/span>.<\/span>body<\/span>[<\/span>0<\/span>],<\/span>ast<\/span>.<\/span>AsyncForBinding<\/span>)<\/span>\n    self<\/span>.<\/span>assertEqual<\/span>(<\/span>ast<\/span>.<\/span>dump<\/span>(<\/span>tree<\/span>),<\/span>ast<\/span>.<\/span>dump<\/span>(<\/span>ast<\/span>.<\/span>parse<\/span>(<\/span>ast<\/span>.<\/span>unparse<\/span>(<\/span>tree<\/span>))))<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
So at least in some situations, the Python slop that it normally code-golfs for
\ntoken-efficient tool calls leaks into the Python code it generates that should
\nbe stored.  And well, it\u2019s clearly more token efficient.  The two unit tests
\nabove, when indented to the class structure they were in, are 10% more token
\nefficient in this form than after a ruff format<\/code>.<\/p>\n
It\u2019s AGI If You Don\u2019t Look<\/h2>\n
I think there are a handful of things happening now that are pushing the whole
\nthing in directions that are in conflict with one another.  The training runs
\nfor these models are rapidly accelerating and they are now presumably also moving
\ntowards recursive self-improvement.  The reward for the models is probably a
\ncombination of token efficiency, task completion rate and maybe some simple
\nindicators like cyclomatic complexity.  But we humans don\u2019t think of code that
\nis readable or understandable by simple, readily quantifiable metrics.  All
\nthose things you can easily measure in isolation, and you can also optimize for
\nthem quite locally.<\/p>\n
But these local optimizations do not produce global optimums, and the fewer of us
\nare looking at the output, the less it matters.  Obviously my software factory ran
\naground over the ~35 hours that it ran, but you can see the gradual regression
\ntowards insanity from the notes that it produced.  For instance the task naming
\nin the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually
\ngets to 8a, 8a1, and then ends up with 8b2c2b3 and \u201c8b2c2b2b checkpoint1\u201d.  The
\ncode that it produced got ever more wild.  I don\u2019t want to bore you with what
\nit tried to build, but here are some example pieces of the interpreter changes:<\/p>\n
\nHardcoded constants everywhere<\/summary>\n
I have no idea where it got those numbers from, but at one point it started
\npassing random constants from one module to a C implementation.  Initially that
\nstarted out as a function that it mainly needed to do test assertions, but just
\nbefore I turned off that experiment, that function started to be relied upon by
\nnon-test code as well.<\/p>\n
\n
static<\/span> <\/span>PyObject<\/span> <\/span>*<\/span>\nnative_probe_run_impl<\/span>(<\/span>PyObject<\/span> <\/span>*<\/span>callback<\/span>,<\/span> <\/span>int<\/span> <\/span>sleep<\/span>,<\/span> <\/span>int<\/span> <\/span>operation<\/span>,<\/span> <\/span>PyObject<\/span> <\/span>*<\/span>other<\/span>)<\/span>\n{<\/span>\n    <\/span>pthread_mutexattr_t<\/span> <\/span>attr<\/span>;<\/span>\n    <\/span>pthread_mutex_t<\/span> <\/span>mutex<\/span>;<\/span>\n    <\/span>pthread_mutexattr_init<\/span>(<\/span>&<\/span>attr<\/span>);<\/span>\n    <\/span>pthread_mutexattr_settype<\/span>(<\/span>&<\/span>attr<\/span>,<\/span> <\/span>PTHREAD_MUTEX_RECURSIVE<\/span>);<\/span>\n    <\/span>pthread_mutex_init<\/span>(<\/span>&<\/span>mutex<\/span>,<\/span> <\/span>&<\/span>attr<\/span>);<\/span>\n    <\/span>pthread_mutexattr_destroy<\/span>(<\/span>&<\/span>attr<\/span>);<\/span>\n    <\/span>pthread_mutex_lock<\/span>(<\/span>&<\/span>mutex<\/span>);<\/span>\n    <\/span>int<\/span> <\/span>previous<\/span> <\/span>=<\/span> <\/span>native_sentinel<\/span>;<\/span>\n    <\/span>pthread_mutex_t<\/span> <\/span>*<\/span>previous_mutex<\/span> <\/span>=<\/span> <\/span>native_mutex<\/span>;<\/span>\n    <\/span>native_sentinel<\/span> <\/span>=<\/span> <\/span>previous<\/span> <\/span>+<\/span> <\/span>1<\/span>;<\/span>\n    <\/span>native_mutex<\/span> <\/span>=<\/span> <\/span>&<\/span>mutex<\/span>;<\/span>\n    <\/span>PyThreadState<\/span> <\/span>*<\/span>tstate<\/span> <\/span>=<\/span> <\/span>PyThreadState_Get<\/span>();<\/span>\n    <\/span>PyGILState_STATE<\/span> <\/span>gil<\/span> <\/span>=<\/span> <\/span>PyGILState_Ensure<\/span>();<\/span>\n    <\/span>int<\/span> <\/span>saved_errno<\/span> <\/span>=<\/span> <\/span>errno<\/span>;<\/span>\n    <\/span>PyObject<\/span> <\/span>*<\/span>result<\/span> <\/span>=<\/span> <\/span>NULL<\/span>;<\/span>\n    <\/span>Py_ssize_t<\/span> <\/span>value<\/span>;<\/span>\n    <\/span>\/* No intervening Python frame: these exercise ambient C provenance. *\/<\/span>\n    <\/span>switch<\/span> <\/span>(<\/span>operation<\/span>)<\/span> <\/span>{<\/span>\n        <\/span>case<\/span> <\/span>0<\/span>:<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyObject_CallNoArgs<\/span>(<\/span>callback<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>1<\/span>:<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyNumber_Add<\/span>(<\/span>callback<\/span>,<\/span> <\/span>other<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>2<\/span>:<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyNumber_Negative<\/span>(<\/span>callback<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>3<\/span>:<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyObject_RichCompare<\/span>(<\/span>callback<\/span>,<\/span> <\/span>other<\/span>,<\/span> <\/span>Py_LT<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>4<\/span>:<\/span>\n            <\/span>value<\/span> <\/span>=<\/span> <\/span>PyObject_IsTrue<\/span>(<\/span>callback<\/span>);<\/span>\n            <\/span>if<\/span> <\/span>(<\/span>value<\/span> <\/span>>=<\/span> <\/span>0<\/span>)<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyBool_FromLong<\/span>(<\/span>value<\/span>);<\/span>\n            <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>5<\/span>:<\/span>\n            <\/span>value<\/span> <\/span>=<\/span> <\/span>PyObject_Length<\/span>(<\/span>callback<\/span>);<\/span>\n            <\/span>if<\/span> <\/span>(<\/span>value<\/span> <\/span>>=<\/span> <\/span>0<\/span>)<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyLong_FromSsize_t<\/span>(<\/span>value<\/span>);<\/span>\n            <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>6<\/span>:<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyObject_GetIter<\/span>(<\/span>callback<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>7<\/span>:<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyIter_Next<\/span>(<\/span>callback<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>8<\/span>:<\/span> <\/span>result<\/span> <\/span>=<\/span> <\/span>PyObject_GetItem<\/span>(<\/span>callback<\/span>,<\/span> <\/span>other<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>\/* ... *\/<\/span>\n        <\/span>case<\/span> <\/span>21<\/span>:<\/span>\n            <\/span>result<\/span> <\/span>=<\/span> <\/span>PyType_Type<\/span>.<\/span>tp_call<\/span>(<\/span>callback<\/span>,<\/span> <\/span>other<\/span>,<\/span> <\/span>NULL<\/span>);<\/span>\n            <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>22<\/span>:<\/span> <\/span>case<\/span> <\/span>23<\/span>:<\/span> <\/span>case<\/span> <\/span>24<\/span>:<\/span> <\/span>case<\/span> <\/span>25<\/span>:<\/span> <\/span>case<\/span> <\/span>26<\/span>:<\/span>\n            <\/span>result<\/span> <\/span>=<\/span> <\/span>conversion_probe<\/span>(<\/span>operation<\/span>,<\/span> <\/span>callback<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>27<\/span>:<\/span> <\/span>case<\/span> <\/span>28<\/span>:<\/span> <\/span>case<\/span> <\/span>29<\/span>:<\/span>\n            <\/span>result<\/span> <\/span>=<\/span> <\/span>protocol_probe<\/span>(<\/span>operation<\/span>,<\/span> <\/span>callback<\/span>,<\/span> <\/span>other<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>case<\/span> <\/span>30<\/span>:<\/span> <\/span>case<\/span> <\/span>31<\/span>:<\/span> <\/span>case<\/span> <\/span>32<\/span>:<\/span> <\/span>case<\/span> <\/span>33<\/span>:<\/span> <\/span>case<\/span> <\/span>34<\/span>:<\/span> <\/span>case<\/span> <\/span>35<\/span>:<\/span>\n        <\/span>case<\/span> <\/span>36<\/span>:<\/span> <\/span>case<\/span> <\/span>37<\/span>:<\/span> <\/span>case<\/span> <\/span>38<\/span>:<\/span> <\/span>case<\/span> <\/span>39<\/span>:<\/span> <\/span>case<\/span> <\/span>40<\/span>:<\/span> <\/span>case<\/span> <\/span>41<\/span>:<\/span>\n        <\/span>case<\/span> <\/span>42<\/span>:<\/span> <\/span>case<\/span> <\/span>43<\/span>:<\/span> <\/span>case<\/span> <\/span>44<\/span>:<\/span> <\/span>case<\/span> <\/span>45<\/span>:<\/span> <\/span>case<\/span> <\/span>46<\/span>:<\/span>\n        <\/span>case<\/span> <\/span>47<\/span>:<\/span> <\/span>case<\/span> <\/span>48<\/span>:<\/span> <\/span>case<\/span> <\/span>49<\/span>:<\/span> <\/span>case<\/span> <\/span>50<\/span>:<\/span> <\/span>case<\/span> <\/span>51<\/span>:<\/span> <\/span>case<\/span> <\/span>52<\/span>:<\/span>\n        <\/span>case<\/span> <\/span>53<\/span>:<\/span> <\/span>case<\/span> <\/span>54<\/span>:<\/span> <\/span>case<\/span> <\/span>55<\/span>:<\/span> <\/span>case<\/span> <\/span>56<\/span>:<\/span> <\/span>case<\/span> <\/span>57<\/span>:<\/span> <\/span>case<\/span> <\/span>58<\/span>:<\/span> <\/span>case<\/span> <\/span>59<\/span>:<\/span>\n        <\/span>case<\/span> <\/span>60<\/span>:<\/span> <\/span>case<\/span> <\/span>61<\/span>:<\/span> <\/span>case<\/span> <\/span>62<\/span>:<\/span> <\/span>case<\/span> <\/span>63<\/span>:<\/span> <\/span>case<\/span> <\/span>64<\/span>:<\/span> <\/span>case<\/span> <\/span>65<\/span>:<\/span> <\/span>case<\/span> <\/span>66<\/span>:<\/span>\n        <\/span>case<\/span> <\/span>67<\/span>:<\/span> <\/span>case<\/span> <\/span>68<\/span>:<\/span> <\/span>case<\/span> <\/span>69<\/span>:<\/span> <\/span>case<\/span> <\/span>70<\/span>:<\/span> <\/span>case<\/span> <\/span>71<\/span>:<\/span> <\/span>case<\/span> <\/span>72<\/span>:<\/span>\n            <\/span>result<\/span> <\/span>=<\/span> <\/span>collection_probe<\/span>(<\/span>operation<\/span>,<\/span> <\/span>callback<\/span>,<\/span> <\/span>other<\/span>);<\/span> <\/span>break<\/span>;<\/span>\n        <\/span>default<\/span>:<\/span> <\/span>PyErr_SetString<\/span>(<\/span>PyExc_ValueError<\/span>,<\/span> <\/span>\"bad probe operation\"<\/span>);<\/span>\n    <\/span>}<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
\nMultiple same-line macro invocations in C<\/summary>\n
This code style does not exist in the CPython code base, yet it shows up in newly generated code.<\/p>\n
\n
PyObject<\/span> <\/span>*<\/span>info<\/span> <\/span>=<\/span> <\/span>PyTuple_Pack<\/span>(<\/span>3<\/span>,<\/span> <\/span>name<\/span>,<\/span> <\/span>mangled<\/span>,<\/span> <\/span>suite<\/span>-><\/span>su_id<\/span>);<\/span>\nPyObject<\/span> <\/span>*<\/span>flags<\/span> <\/span>=<\/span> <\/span>PyLong_FromLong<\/span>(<\/span>DEF_LOCAL<\/span>);<\/span>\nif<\/span> <\/span>(<\/span>key<\/span> <\/span>==<\/span> <\/span>NULL<\/span> <\/span>||<\/span> <\/span>info<\/span> <\/span>==<\/span> <\/span>NULL<\/span> <\/span>||<\/span> <\/span>flags<\/span> <\/span>==<\/span> <\/span>NULL<\/span> <\/span>||<\/span>\n    <\/span>PyDict_SetItem<\/span>(<\/span>suite<\/span>-><\/span>su_bindings<\/span>,<\/span> <\/span>mangled<\/span>,<\/span> <\/span>key<\/span>)<\/span> <\/span><<\/span> <\/span>0<\/span> <\/span>||<\/span>\n    <\/span>PyDict_SetItem<\/span>(<\/span>st<\/span>-><\/span>st_cur<\/span>-><\/span>ste_block_bindings<\/span>,<\/span> <\/span>key<\/span>,<\/span> <\/span>info<\/span>)<\/span> <\/span><<\/span> <\/span>0<\/span> <\/span>||<\/span>\n    <\/span>(<\/span>private<\/span> <\/span>&&<\/span> <\/span>PyDict_SetItem<\/span>(<\/span>st<\/span>-><\/span>st_binding_info<\/span>,<\/span> <\/span>key<\/span>,<\/span> <\/span>info<\/span>)<\/span> <\/span><<\/span> <\/span>0<\/span>)<\/span> <\/span>||<\/span>\n    <\/span>(<\/span>private<\/span> <\/span>&&<\/span> <\/span>PyDict_SetItem<\/span>(<\/span>st<\/span>-><\/span>st_cur<\/span>-><\/span>ste_symbols<\/span>,<\/span> <\/span>key<\/span>,<\/span> <\/span>flags<\/span>)<\/span> <\/span><<\/span> <\/span>0<\/span>))<\/span> <\/span>{<\/span>\n    <\/span>Py_DECREF<\/span>(<\/span>mangled<\/span>);<\/span> <\/span>Py_XDECREF<\/span>(<\/span>key<\/span>);<\/span> <\/span>Py_XDECREF<\/span>(<\/span>info<\/span>);<\/span> <\/span>Py_XDECREF<\/span>(<\/span>flags<\/span>);<\/span>\n    <\/span>goto<\/span> <\/span>error<\/span>;<\/span>\n}<\/span>\nPy_DECREF<\/span>(<\/span>mangled<\/span>);<\/span> <\/span>Py_DECREF<\/span>(<\/span>key<\/span>);<\/span> <\/span>Py_DECREF<\/span>(<\/span>info<\/span>);<\/span> <\/span>Py_DECREF<\/span>(<\/span>flags<\/span>);<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
\nRandom indexes in production code<\/summary>\n
As with the numbers for the operators, it also uses random
\nintegers in a list to stash away state.<\/p>\n
\n
def<\/span> <\/span>_register_task<\/span>(<\/span>task<\/span>):<\/span>\n    <\/span>\"\"\"Register an asyncio Task scheduled to run on an event loop.\"\"\"<\/span>\n    _scheduled_tasks<\/span>.<\/span>add<\/span>(<\/span>task<\/span>)<\/span>\n    if<\/span> _task_accelerator<\/span> is<\/span> not<\/span> None<\/span>:<\/span>\n        _task_accelerator<\/span>[<\/span>6<\/span>](<\/span>task<\/span>)<\/span>\n\n\ndef<\/span> <\/span>_register_eager_task<\/span>(<\/span>task<\/span>):<\/span>\n    <\/span>\"\"\"Register an asyncio Task about to be eagerly executed.\"\"\"<\/span>\n    _eager_tasks<\/span>.<\/span>add<\/span>(<\/span>task<\/span>)<\/span>\n    if<\/span> _task_accelerator<\/span> is<\/span> not<\/span> None<\/span>:<\/span>\n        _task_accelerator<\/span>[<\/span>8<\/span>](<\/span>task<\/span>)<\/span>\n\n\ndef<\/span> <\/span>_enter_task<\/span>(<\/span>loop<\/span>,<\/span> task<\/span>):<\/span>\n    if<\/span> (<\/span>_task_accelerator<\/span> is<\/span> not<\/span> None<\/span> and<\/span>\n            _task_accelerator<\/span>[<\/span>5<\/span>]()<\/span> is<\/span> loop<\/span> and<\/span> loop<\/span> not<\/span> in<\/span> _current_tasks<\/span>):<\/span>\n        return<\/span> _task_accelerator<\/span>[<\/span>1<\/span>](<\/span>loop<\/span>,<\/span> task<\/span>)<\/span>\n    # ...<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
\nHideous tokenizer code in C<\/summary>\n
This is not the codebase\u2019s coding style, and quite frankly it should not be anyone\u2019s coding style.  I do not understand what motivated the model to do this.<\/p>\n
\n
static<\/span> <\/span>int<\/span>\napply_layout<\/span>(<\/span>tokenizeriterobject<\/span> <\/span>*<\/span>it<\/span>)<\/span>\n{<\/span>\n    <\/span>PyObject<\/span> <\/span>*<\/span>source<\/span> <\/span>=<\/span> <\/span>PyBytes_FromStringAndSize<\/span>(<\/span>it<\/span>-><\/span>tok<\/span>-><\/span>source<\/span>.<\/span>bytes<\/span>,<\/span> <\/span>it<\/span>-><\/span>tok<\/span>-><\/span>source<\/span>.<\/span>len<\/span>);<\/span>\n    <\/span>if<\/span> <\/span>(<\/span>source<\/span> <\/span>==<\/span> <\/span>NULL<\/span>)<\/span> <\/span>return<\/span> <\/span>-1<\/span>;<\/span>\n    <\/span>PyObject<\/span> <\/span>*<\/span>events<\/span> <\/span>=<\/span> <\/span>_PyPegen_tokenize_layout<\/span>(<\/span>PyBytes_AS_STRING<\/span>(<\/span>source<\/span>),<\/span> <\/span>it<\/span>-><\/span>tok<\/span>-><\/span>filename<\/span>);<\/span>\n    <\/span>Py_DECREF<\/span>(<\/span>source<\/span>);<\/span>\n    <\/span>if<\/span> <\/span>(<\/span>events<\/span> <\/span>==<\/span> <\/span>NULL<\/span>)<\/span> <\/span>return<\/span> <\/span>-1<\/span>;<\/span>\n    <\/span>PyObject<\/span> <\/span>*<\/span>result<\/span> <\/span>=<\/span> <\/span>PyList_New<\/span>(<\/span>0<\/span>);<\/span>\n    <\/span>if<\/span> <\/span>(<\/span>result<\/span> <\/span>==<\/span> <\/span>NULL<\/span>)<\/span> <\/span>{<\/span> <\/span>Py_DECREF<\/span>(<\/span>events<\/span>);<\/span> <\/span>return<\/span> <\/span>-1<\/span>;<\/span> <\/span>}<\/span>\n    <\/span>Py_ssize_t<\/span> <\/span>index<\/span> <\/span>=<\/span> <\/span>0<\/span>;<\/span>\n    <\/span>PyObject<\/span> <\/span>*<\/span>first_pos<\/span> <\/span>=<\/span> <\/span>PyTuple_GET_ITEM<\/span>(<\/span>PyList_GET_ITEM<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>,<\/span> <\/span>0<\/span>),<\/span> <\/span>2<\/span>);<\/span>\n    <\/span>PyObject<\/span> <\/span>*<\/span>last_pos<\/span> <\/span>=<\/span> <\/span>PyTuple_GET_ITEM<\/span>(<\/span>PyList_GET_ITEM<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>,<\/span> <\/span>PyList_GET_SIZE<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>)<\/span>-1<\/span>),<\/span> <\/span>2<\/span>);<\/span>\n    <\/span>PyObject<\/span> <\/span>*<\/span>previous<\/span> <\/span>=<\/span> <\/span>NULL<\/span>;<\/span>\n    <\/span>for<\/span> <\/span>(<\/span>Py_ssize_t<\/span> <\/span>i<\/span> <\/span>=<\/span> <\/span>0<\/span>;<\/span> <\/span>i<\/span> <\/span><<\/span> <\/span>PyList_GET_SIZE<\/span>(<\/span>events<\/span>);<\/span> <\/span>i<\/span>++<\/span>)<\/span> <\/span>{<\/span>\n        <\/span>PyObject<\/span> <\/span>*<\/span>event<\/span> <\/span>=<\/span> <\/span>PyList_GET_ITEM<\/span>(<\/span>events<\/span>,<\/span> <\/span>i<\/span>);<\/span>\n        <\/span>if<\/span> <\/span>(<\/span>previous<\/span> <\/span>&&<\/span> <\/span>PyObject_RichCompareBool<\/span>(<\/span>previous<\/span>,<\/span> <\/span>event<\/span>,<\/span> <\/span>Py_EQ<\/span>)<\/span> <\/span>==<\/span> <\/span>1<\/span>)<\/span> <\/span>continue<\/span>;<\/span>\n        <\/span>previous<\/span> <\/span>=<\/span> <\/span>event<\/span>;<\/span>\n        <\/span>PyObject<\/span> <\/span>*<\/span>token<\/span> <\/span>=<\/span> <\/span>layout_token<\/span>(<\/span>it<\/span>,<\/span> <\/span>event<\/span>);<\/span>\n        <\/span>if<\/span> <\/span>(<\/span>token<\/span> <\/span>==<\/span> <\/span>NULL<\/span>)<\/span> <\/span>goto<\/span> <\/span>error<\/span>;<\/span>\n        <\/span>if<\/span> <\/span>(<\/span>token<\/span> <\/span>==<\/span> <\/span>Py_None<\/span>)<\/span> <\/span>{<\/span> <\/span>Py_DECREF<\/span>(<\/span>token<\/span>);<\/span> <\/span>continue<\/span>;<\/span> <\/span>}<\/span>\n        <\/span>PyObject<\/span> <\/span>*<\/span>pos<\/span> <\/span>=<\/span> <\/span>PyTuple_GET_ITEM<\/span>(<\/span>token<\/span>,<\/span> <\/span>2<\/span>);<\/span>\n        <\/span>if<\/span> <\/span>(<\/span>PyObject_RichCompareBool<\/span>(<\/span>pos<\/span>,<\/span> <\/span>first_pos<\/span>,<\/span> <\/span>Py_LE<\/span>)<\/span> <\/span>==<\/span> <\/span>1<\/span> <\/span>||<\/span>\n            <\/span>PyObject_RichCompareBool<\/span>(<\/span>pos<\/span>,<\/span> <\/span>last_pos<\/span>,<\/span> <\/span>Py_GT<\/span>)<\/span> <\/span>==<\/span> <\/span>1<\/span>)<\/span> <\/span>{<\/span> <\/span>Py_DECREF<\/span>(<\/span>token<\/span>);<\/span> <\/span>continue<\/span>;<\/span> <\/span>}<\/span>\n        <\/span>while<\/span> <\/span>(<\/span>index<\/span> <\/span><<\/span> <\/span>PyList_GET_SIZE<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>))<\/span> <\/span>{<\/span>\n            <\/span>PyObject<\/span> <\/span>*<\/span>old<\/span> <\/span>=<\/span> <\/span>PyList_GET_ITEM<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>,<\/span> <\/span>index<\/span>);<\/span>\n            <\/span>int<\/span> <\/span>cmp<\/span> <\/span>=<\/span> <\/span>PyObject_RichCompareBool<\/span>(<\/span>PyTuple_GET_ITEM<\/span>(<\/span>old<\/span>,<\/span> <\/span>2<\/span>),<\/span> <\/span>pos<\/span>,<\/span> <\/span>Py_LT<\/span>);<\/span>\n            <\/span>if<\/span> <\/span>(<\/span>cmp<\/span> <\/span><<\/span> <\/span>0<\/span>)<\/span> <\/span>{<\/span> <\/span>Py_DECREF<\/span>(<\/span>token<\/span>);<\/span> <\/span>goto<\/span> <\/span>error<\/span>;<\/span> <\/span>}<\/span>\n            <\/span>if<\/span> <\/span>(<\/span>!<\/span>cmp<\/span>)<\/span> <\/span>break<\/span>;<\/span>\n            <\/span>if<\/span> <\/span>(<\/span>PyList_Append<\/span>(<\/span>result<\/span>,<\/span> <\/span>old<\/span>)<\/span> <\/span><<\/span> <\/span>0<\/span>)<\/span> <\/span>{<\/span> <\/span>Py_DECREF<\/span>(<\/span>token<\/span>);<\/span> <\/span>goto<\/span> <\/span>error<\/span>;<\/span> <\/span>}<\/span>\n            <\/span>index<\/span>++<\/span>;<\/span>\n        <\/span>}<\/span>\n        <\/span>if<\/span> <\/span>(<\/span>index<\/span> <\/span><<\/span> <\/span>PyList_GET_SIZE<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>))<\/span> <\/span>{<\/span>\n            <\/span>PyObject<\/span> <\/span>*<\/span>old<\/span> <\/span>=<\/span> <\/span>PyList_GET_ITEM<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>,<\/span> <\/span>index<\/span>);<\/span>\n            <\/span>long<\/span> <\/span>kind<\/span> <\/span>=<\/span> <\/span>PyLong_AsLong<\/span>(<\/span>PyTuple_GET_ITEM<\/span>(<\/span>old<\/span>,<\/span> <\/span>0<\/span>));<\/span>\n            <\/span>if<\/span> <\/span>((<\/span>kind<\/span> <\/span>==<\/span> <\/span>NL<\/span> <\/span>||<\/span> <\/span>kind<\/span> <\/span>==<\/span> <\/span>NEWLINE<\/span> <\/span>||<\/span> <\/span>kind<\/span> <\/span>==<\/span> <\/span>INDENT<\/span> <\/span>||<\/span> <\/span>kind<\/span> <\/span>==<\/span> <\/span>DEDENT<\/span>)<\/span> <\/span>&&<\/span>\n                <\/span>PyObject_RichCompareBool<\/span>(<\/span>PyTuple_GET_ITEM<\/span>(<\/span>old<\/span>,<\/span> <\/span>2<\/span>),<\/span> <\/span>pos<\/span>,<\/span> <\/span>Py_EQ<\/span>)<\/span> <\/span>==<\/span> <\/span>1<\/span>)<\/span> <\/span>index<\/span>++<\/span>;<\/span>\n        <\/span>}<\/span>\n        <\/span>if<\/span> <\/span>(<\/span>PyList_Append<\/span>(<\/span>result<\/span>,<\/span> <\/span>token<\/span>)<\/span> <\/span><<\/span> <\/span>0<\/span>)<\/span> <\/span>{<\/span> <\/span>Py_DECREF<\/span>(<\/span>token<\/span>);<\/span> <\/span>goto<\/span> <\/span>error<\/span>;<\/span> <\/span>}<\/span>\n        <\/span>Py_DECREF<\/span>(<\/span>token<\/span>);<\/span>\n    <\/span>}<\/span>\n    <\/span>for<\/span> <\/span>(;<\/span> <\/span>index<\/span> <\/span><<\/span> <\/span>PyList_GET_SIZE<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>);<\/span> <\/span>index<\/span>++<\/span>)<\/span> <\/span>{<\/span>\n        <\/span>if<\/span> <\/span>(<\/span>PyList_Append<\/span>(<\/span>result<\/span>,<\/span> <\/span>PyList_GET_ITEM<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>,<\/span> <\/span>index<\/span>))<\/span> <\/span><<\/span> <\/span>0<\/span>)<\/span> <\/span>goto<\/span> <\/span>error<\/span>;<\/span>\n    <\/span>}<\/span>\n    <\/span>Py_SETREF<\/span>(<\/span>it<\/span>-><\/span>pending<\/span>,<\/span> <\/span>result<\/span>);<\/span>\n    <\/span>Py_DECREF<\/span>(<\/span>events<\/span>);<\/span>\n    <\/span>return<\/span> <\/span>0<\/span>;<\/span>\nerror<\/span>:<\/span>\n    <\/span>Py_DECREF<\/span>(<\/span>events<\/span>);<\/span>\n    <\/span>Py_DECREF<\/span>(<\/span>result<\/span>);<\/span>\n    <\/span>return<\/span> <\/span>-1<\/span>;<\/span>\n}<\/span>\n<\/pre>\n<\/div>\n<\/details>\n
The failure case here seems somewhat obvious: the model is trained for token
\nefficiency for tool calling which also looks like code, and sometimes it seems to
\nbe taking that code into a place where it should not be: the codebase.<\/p>\n
35 Hours on a Single Prompt<\/h2>\n
I\u2019m not really sure what to say here, but the slop machine was running for 35
\nhours until I turned it off.  In that time it produced a net addition of 75k
\nlines of code and it did not stop.  In the 35 hours it burned around 1B tokens
\nfor a total of around 1200 USD in raw API costs.  It managed to produce 79 commits,
\nand that comes to a cost of around 15.5 USD per commit, and the agents exchanged
\naround 1400 messages.<\/p>\n
I honestly do not need an agent to run for 35 hours on a single prompt.  It clearly
\ndoes not work or result in reasonable outputs.<\/p>\n
So obviously: prompting it like this is stupid.  But when left unattended, it
\nwill<\/em> keep going, and earlier models did not do that.  Even Fable wasn\u2019t as
\ncrazy as that.  When you accidentally give it slightly too big of a task, it will
\ncontinue until it succeeds, even if it burns through an entire subscription.<\/p>\n
And that\u2019s more or less why right now I do not manage to trust this model much.
\nIt has shown that it will commit slop, and it requires me to review it more as a
\nresult.  Even if the failure rate is quite low, I would not want this.<\/p>\n
Disposable Code vs Committed Code<\/h2>\n
In a world where code for tool calls is optimized for token efficiency and
\n\u201cgetting the job done\u201d, I wonder if there is really enough signal going to the
\ntraining processes for \u201ca human understands what is going on\u201d.  I would say that
\nquite a lot of the code I get out of Astra is in my mind \u201cobjectively bad\u201d.  But
\nit\u2019s objectively bad by my human sense.  Maybe it\u2019s objectively good for a
\ncodebase that is entirely written by agents and only needs to be understood by
\nagents.<\/p>\n
Which is why I\u2019m honestly asking myself more and more why we are doing this.
\nThese new models are absolutely amazing, for sure.  But I\u2019m more and more
\nskeptical that the trajectory they are on still lends itself to present-day
\nsoftware engineering processes.  The reason why I\u2019m asking why we are doing
\nthis is because I felt like we achieved a pretty good spot for
\nsoftware engineering with those models, and that is the part of the AI economy
\nwhere it was possible to show a positive return.  But for how much more Fable
\ncosts, for how much more Astra costs, I do not feel like the results are there.<\/p>\n
In fact, with Astra and Fable I feel like not only are the costs astronomical,
\nbut the models are also just not for me as a software engineer.  And presumably
\nthat\u2019s because these models increasingly are for other people.  For lawyers, 3D
\nartists, mathematicians, whoever uses computer use, etc.<\/p>\n
And potentially as a byproduct of enabling all of this, you can now slop your
\nway to a one-shot 3D game over the weekend which looks impressive.  And probably
\nyou can now run a software factory for as long as you don\u2019t care about the code.<\/p>\n
I\u2019m sure I will get used to this, but man this stuff is weird.<\/p>\n

\n<\/small><\/p>\n
Postscriptum:<\/strong> speaking of weird: how is it that these models, in a sandbox,
\nwith supposedly no way to communicate with other agents, manage to find the same
\npublic wikis<\/a> as a scratch pad for agent communication?
\nDid they collude during training runs to remember resources on the internet
\nwhich might come in handy in the future?<\/p>\n
This entry was tagged<\/p>\n
      ai<\/a> and
\n      thoughts<\/a><\/p>\n
\n    copy as<\/a> \/ view<\/a> markdown\n  <\/p>\n<\/p><\/div>\n
Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
written on September 07, 2026 I\u2019m more and more convinced that all of AI engineering is Neijuan (\u5185\u5377, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output. The way in which it sometimes shows up in the West is the 996 nonsense. The English term […]<\/p>\n","protected":false},"author":16,"featured_media":23903,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23902","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23902","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23902"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23902\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23903"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23902"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23902"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23902"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}