{"id":22493,"date":"2026-07-20T21:29:01","date_gmt":"2026-07-20T21:29:01","guid":{"rendered":"https:\/\/scannn.com\/fable-5-vs-gpt-5-6-sol-on-an-np-hard-problem-does-goal-help\/"},"modified":"2026-07-20T21:29:01","modified_gmt":"2026-07-20T21:29:01","slug":"fable-5-vs-gpt-5-6-sol-on-an-np-hard-problem-does-goal-help","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/fable-5-vs-gpt-5-6-sol-on-an-np-hard-problem-does-goal-help\/","title":{"rendered":"Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does \/goal Help?"},"content":{"rendered":"\n
\n

TL;DR:<\/strong> I gave Claude Fable 5 and GPT-5.6 Sol the same unpublished NP-hard
\noptimization problem, with and without their native \/goal<\/code> mode. Fable 5 is an absolute beast;
\n\/goal<\/code> is not a game changer.<\/p>\n

<\/p>\n

Context:<\/strong> This is an operations research problem originally submitted to students at a
\nhackathon. I spent a week years
\nago writing C++ to solve it, so I have a useful human baseline.<\/p>\n

Fable 5 was an absolute beast on this benchmark. It produced the best solution overall, and its
\nconsistency is unlike anything I have seen from a model on this problem. This is pure raw
\nintelligence. Incredible.<\/p>\n

The other result is that \/goal<\/code> is not a generic \u201ctry harder\u201d switch. It changes the control
\nloop and the search path. Sometimes that finds a better basin. Sometimes it gives a bad idea
\nmore time to mature.<\/p>\n

All code, prompts, result tables, exclusions, and trajectory notes are in
\nCLIArena<\/a>. This is a follow-up to my
\nfirst article about this benchmark<\/a>.<\/p>\n\n

The problem<\/h2>\n

KIRO is a fiber-network design problem I worked on as an engineering student in 2018. Given
\ndirected distance matrices for Grenoble, Nice, and Paris, the solver has to connect distribution
\npoints and terminals using loops and short chains while respecting several structural
\nconstraints. The objective is total cable length. Lower is better.<\/p>\n

\"KIRO<\/p>\n

A valid network consists of redundant loops rooted at distribution hubs, with short branches
\nhanging from towers on those loops. Every tower must appear exactly once, and reversing a cable
\nsegment can change its cost.<\/p>\n

How large is the search space?<\/h3>\n

There is no single closed-form count because a solution can use any number of loops, variable
\nloop sizes, and differently anchored and ordered branches. But Paris alone gives a useful lower
\nbound.<\/p>\n

Even if we ignore ordering and branches and only assign each of the 532 terminals to one of
\n11 distribution hubs, there are 11^532<\/code> possible assignments.<\/p>\n

A stronger lower bound comes from one deliberately restricted family of valid solutions:
\nexactly 19 loops of 28 terminals each, with no branches. This covers all 532 terminals because
\n19 x 28 = 532<\/code>, while staying below the 30-terminal limit for a loop. Order the 532 terminals,
\nsplit that ordering into 19 consecutive groups, divide by 19!<\/code> because the set of loops is
\nunordered, and choose one of the 11 hubs for each loop:<\/p>\n

(532! \/ 19!) x 11^19 ~= 10^1223<\/code><\/p>\n

What I tested<\/h2>\n

The primary experiment was intentionally narrow:<\/p>\n\n\n\n\n\n\n\n\n\n\n
Setting<\/th>\nValue<\/th>\n<\/tr>\n<\/thead>\n
Models<\/td>\nClaude Fable 5, Opus 4.8, Sonnet 5; GPT-5.6 Sol, Terra, Luna<\/td>\n<\/tr>\n
Modes<\/td>\nPlain; native \/goal<\/code><\/td>\n<\/tr>\n
Optimization budget<\/td>\n30 minutes<\/td>\n<\/tr>\n
Outer agent timeout<\/td>\n1,900 seconds<\/td>\n<\/tr>\n
Reasoning<\/td>\nMaximum available setting for every model<\/td>\n<\/tr>\n
Execution<\/td>\nHarbor 0.1.43, Docker, subscription authentication<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Results<\/h2>\n

Before concentrating repetitions on the flagship pair, I ran one matched 30-minute no-hint
\npair for every model in the sweep. For Fable and Sol, the chart uses Pair 1 from the replicated
\nheadline set; the other four models have one pair each.<\/p>\n

\"All<\/p>\n

I then repeated the flagship comparison until I had three matched runs for Fable 5 and three
\nfor Sol.<\/p>\n

\"Clean<\/p>\n\n\n\n\n\n\n\n\n\n\n
Model<\/th>\nRun<\/th>\nPlain<\/th>\n\/goal<\/code><\/th>\nGoal minus plain<\/th>\n<\/tr>\n<\/thead>\n
Fable 5<\/td>\n1<\/td>\n32,197<\/td>\n31,934<\/strong><\/td>\n-263<\/td>\n<\/tr>\n
Fable 5<\/td>\n2<\/td>\n32,516<\/td>\n32,324<\/strong><\/td>\n-192<\/td>\n<\/tr>\n
Fable 5<\/td>\n3<\/td>\n32,446<\/strong><\/td>\n35,178<\/td>\n+2,732<\/td>\n<\/tr>\n
GPT-5.6 Sol<\/td>\n1<\/td>\n33,581<\/strong><\/td>\n39,371<\/td>\n+5,790<\/td>\n<\/tr>\n
GPT-5.6 Sol<\/td>\n2<\/td>\n35,539<\/td>\n32,703<\/strong><\/td>\n-2,836<\/td>\n<\/tr>\n
GPT-5.6 Sol<\/td>\n3<\/td>\n33,663<\/td>\n33,313<\/strong><\/td>\n-350<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Negative means \/goal<\/code> was better. Goal won four of six trials, so win rate alone makes the
\nfeature look useful. The means tell the other half:<\/p>\n\n\n\n\n\n\n
Model<\/th>\nPlain mean<\/th>\n\/goal<\/code> mean<\/th>\nMean effect<\/th>\nMedian effect<\/th>\n<\/tr>\n<\/thead>\n
Fable 5<\/td>\n32,386<\/strong><\/td>\n33,145<\/td>\n+759 worse<\/td>\n-192 better<\/td>\n<\/tr>\n
GPT-5.6 Sol<\/td>\n34,261<\/strong><\/td>\n35,129<\/td>\n+868 worse<\/td>\n-350 better<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Both models usually got a small benefit and occasionally suffered a large regression. That is
\nwhy \/goal<\/code> won most runs but made both means worse.<\/p>\n

Fable was also clearly stronger. Its plain mean beat Sol\u2019s by 1,875 points, and its goal mean
\nbeat Sol\u2019s by 1,984. More importantly, Fable plain stayed inside a tiny 319-point range while
\nSol plain spanned 1,958 points. Fable goal produced the best clean score, 31,934; Fable plain
\nwas the safest configuration.<\/p>\n

Deep dive into the goal command<\/h2>\n

The same command hides two different systems<\/h2>\n

Claude Code and Codex both expose \/goal<\/code>, but the implementations are fundamentally different.<\/p>\n

\"Claude<\/p>\n

Claude Code: a separate evaluator<\/h3>\n

Claude Code implements \/goal<\/code> as a session-scoped Stop hook. After each main-model turn, a
\nsmall evaluator model, Haiku by default, reads the condition and conversation. It returns yes
\nor no with a reason. A no starts another turn; a yes clears the goal.<\/p>\n

The evaluator cannot use tools or inspect files. It can only judge evidence that appeared in
\nthe transcript. That can catch an early exit, but it cannot know whether another ten million
\nsolver iterations are worthwhile. Anthropic\u2019s goal documentation<\/a><\/p>\n

Keep in mind that claude code is not open source, so we rely solely on what Anthropic tells us.<\/p>\n

Codex: persisted state and lifecycle tools<\/h3>\n

I also read the source for the benchmarked release,
\nCodex CLI 0.144.4<\/a>. Codex treats a goal as
\npersisted thread state:<\/p>\n

    \n
  1. The TUI saves the objective for the active thread, and SQLite stores its status and budget
    \naccounting.     TUI<\/a>,
    \n    schema<\/a><\/li>\n
  2. The working model receives create_goal<\/code>, get_goal<\/code>, and update_goal<\/code> tools.
    \n    Tool specification<\/a><\/li>\n
  3. If the thread becomes idle while the goal is active, Codex injects a continuation turn with
    \nthe objective and a completion audit.     Runtime<\/a>,
    \n    prompt<\/a><\/li>\n<\/ol>\n

    Claude delegates completion to another model. Codex lets the working model declare completion,
    \nthen resumes it while the persisted goal remains active. Claude\u2019s evaluator is independent but
    \nsees only the transcript; Codex sees the files and tools but effectively grades its own work.<\/p>\n

    Why \/goal<\/code> can win most runs and still be a bad default<\/h2>\n

    On a normal coding task, progress is often legible: another turn can fix a test or complete a
    \nmigration. Optimization is different. Once an agent chooses a solver, extra time can amplify
    \neither a good decision or a bad one.<\/p>\n

    That is exactly what happened here. Goal helped when it sustained Fable\u2019s fast compiled
    \nportfolio or Sol\u2019s successful chain repartition. It hurt when Fable built a slow solver or Sol
    \ncommitted to an exhaustive anchor sweep. The median moved slightly in the right direction; the
    \nbad tail moved much farther in the wrong one.<\/p>\n

    Limitations<\/h2>\n

    This is one unpublished NP-hard task, not a general coding leaderboard. Only Fable and Sol have
    \nthree clean matched pairs. Other comparisons mix prompts, wrapper versions, and time limits,
    \nand the trials ran sequentially through subscription services that may have drifted.<\/p>\n

    The containers exposed eight CPUs despite task metadata declaring one, which favored Fable\u2019s
    \nparallel portfolios. Every scored Fable and Sol output was valid, partly because the wrapper
    \nrequired early checkpoints and final verification. The benchmark measures the complete system:
    \nmodel, CLI, prompt, subscription service, and harness.<\/p>\n

    Reproducing this<\/h2>\n

    The benchmark task, wrappers, analysis scripts, figure generator, and full evidence memo are in
    \n    CLIArena<\/a>. Raw job directories are excluded from Git
    \nbecause of their size, but the memo records every publishable score, city breakdown, elapsed
    \ntime, strategy, exclusion, and run ID.<\/p>\n

    The primary commands are:<\/p>\n

    RUN_ID<\/span>=<\/span>article-kiro-YYYYMMDD-clean<\/span> \\<\/span><\/span>\nPHASE=nohint-all <\/span>\\<\/span><\/span>\n.\/scripts\/run_subscription_article_matrix.sh<\/span><\/span>\n\nuv<\/span> run<\/span> python<\/span> scripts\/summarize_subscription_article_results.py<\/span> RUN_ID...<\/span><\/span>\nuv<\/span> run<\/span> python<\/span> scripts\/analyze_subscription_article_results.py<\/span> RUN_ID...<\/span><\/span><\/code><\/pre>\n
    The result I would put in the headline is not that goal helps or hurts. It is that a persistence
    \nfeature can win most individual trials while making observed average performance worse. On a hard
    \noptimization problem, the quality of the loop matters less than the quality of what the loop
    \nkeeps doing.<\/p>\n<\/p><\/div>\n
    TL;DR: I gave Claude Fable 5 and GPT-5.6 Sol the same unpublished NP-hard optimization problem, with and without their native \/goal mode. Fable 5 is an absolute beast; \/goal is not a game changer. Context: This is an operations research problem originally submitted to students at a hackathon. I spent a week years ago writing […]<\/p>\n","protected":false},"author":16,"featured_media":22494,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-22493","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22493","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=22493"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22493\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/22494"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=22493"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=22493"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=22493"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
    Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"