{"id":22689,"date":"2026-07-25T07:21:00","date_gmt":"2026-07-25T07:21:00","guid":{"rendered":"https:\/\/scannn.com\/are-ai-labs-pelicanmaxxing-dylan-castillo\/"},"modified":"2026-07-25T07:21:00","modified_gmt":"2026-07-25T07:21:00","slug":"are-ai-labs-pelicanmaxxing-dylan-castillo","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/are-ai-labs-pelicanmaxxing-dylan-castillo\/","title":{"rendered":"Are AI labs pelicanmaxxing? \u2013 Dylan Castillo"},"content":{"rendered":"\n
\n

For the past few years, Simon Willison<\/a> has tested every major LLM release with the same prompt: \u201cGenerate an SVG of a pelican riding a bicycle\u201d.<\/p>\n

What began as a tongue-in-cheek benchmark has become one of the most famous informal benchmarks in AI. Simon\u2019s pelican-on-a-bicycle results are often among the most upvoted comments on Hacker News threads announcing new releases from AI labs.<\/p>\n

The benchmark is now famous enough that there\u2019s plenty<\/a> of<\/a> discussion<\/a> about its usefulness and about whether AI labs might be benchmaxxing on it. When billions or even trillions of dollars are at stake, and a strong result could help persuade users, wouldn\u2019t it be tempting to pelicanmaxx<\/em> your model just a bit?<\/p>\n

I wanted to find out, so I put together a small experiment. I generated 1,008 SVGs across seven frontier models, scored them with an LLM judge, and used Claude Fable 5 for the analysis.<\/p>\n

This article presents the results. All the code is available on Github<\/a>.<\/p>\n

\n

How I tested it<\/h2>\n

I built a grid of 8 animals \u00d7 6 vehicles = 48 prompts, where the famous prompt is one cell:<\/p>\n

    \n
  1. Animals<\/strong>: pelican, flamingo, heron, otter, raccoon, antelope, whale, cat<\/li>\n
  2. Vehicles<\/strong>: bicycle, unicycle, skateboard, scooter, plane, boat<\/li>\n<\/ol>\n

    Every prompt uses almost identical phrasing to Simon\u2019s, only switching the animal and vehicle. The animal and vehicle selection wasn\u2019t done in a very rigorous manner, but I tried to vary both similarity to the original prompt and difficulty. Flamingo and heron are quite similar to pelicans; cat, raccoon, and otter are easy cases; antelope is hard; and whale is as different as you can get.<\/p>\n

    I tested seven models through OpenRouter<\/a>: GPT-5.6 Terra<\/em>, Claude Sonnet 5<\/em>, Gemini 3.5 Flash<\/em>, Grok 4.5<\/em>, Qwen3.7-Max<\/em>, GLM-5.2<\/em>, and DeepSeek V4 Pro<\/em>. I generated 3 samples per prompt, at temperature 1.0, requesting the same reasoning effort from every model. That resulted in 1,008 SVGs.<\/p>\n

    Then I ran each image through a three-stage pipeline:<\/p>\n

      \n
    1. Rendering<\/strong>: Each SVG is rendered to PNG. If a model returns no SVG or one that fails to render, I regenerate until it produces a valid one, and record the number of attempts. There were only 11 retries across the 1,008 generations.<\/li>\n
    2. Judging<\/strong>: GPT-5.6 Luna<\/em> scores each image with 1-5 ratings for the animal, the vehicle, and the coherence of the action. When I rank animals or vehicles below, I use the matching rating on its own. When I need one number per image, I use the average of the three, which I call the judge score.<\/li>\n
    3. Feature extraction<\/strong>: For a more detailed analysis, I also passed each rendered image to Gemini 3.1 Flash-Lite<\/em>, which recorded the animal and vehicle it recognized, which way the subject faces, and an open-ended list of scene elements.<\/li>\n<\/ol>\n

      My hypothesis is that if a lab trained on the benchmark, it should show up in some combination of the pelican row scoring above what the animal deserves, the bicycle column scoring above what the vehicle deserves, or the specific pelican-bicycle cell beating both.<\/p>\n<\/section>\n

      \n

      Evidence #1: The pelicans on bicycles don\u2019t look any better<\/h2>\n

      Before any scoring, the simplest test is to look at the images yourself. Pick a lab to see everything it drew, with the judge\u2019s score under each image (click to open full size):<\/p>\n

      I looked through the images myself before running the analysis below. Nothing jumped out at me. I couldn\u2019t find a case where the pelican-bicycle images looked noticeably better than the rest of that model\u2019s grid. Maybe in GLM-5.2\u2019s first sample it felt slightly better than the rest, but that batch also produced a pretty cool heron on a skateboard, so I cannot say for sure. Otherwise they look like the rest of what each model draws, and the labs that draw good pelicans on bicycles also do a good job drawing other animal-vehicle combinations.<\/p>\n

      But this test is hard to replicate, and everyone will have a different opinion. So I wanted something more quantitative, which is why I opted for the method detailed above.<\/p>\n<\/section>\n

      \n

      Evidence #2: Labs are not better at drawing pelicans<\/h2>\n

      Here\u2019s the mean animal rating per animal, pooled across all models:<\/p>\n

      \n
      \n
      \nFigure\u00a01: Mean animal rating by animal, all models pooled.
      \n<\/figcaption><\/figure>\n<\/div>\n<\/div>\n

      The pelican is 6th of 8, behind cat, whale, raccoon, heron, and antelope. If AI labs were training on the benchmark, you\u2019d expect pelicans at the top. Instead they\u2019re in the bottom half. All seven labs draw cats, whales, and raccoons better than pelicans.<\/p>\n

      Of course, a pelican may simply be harder to draw than a cat. A lab could train on pelicans and still not push them past the easy animals, so this ranking alone can\u2019t rule that out. I\u2019ll adjust for difficulty in Evidence #4.<\/p>\n<\/section>\n

      \n

      Evidence #3: Labs are not better at drawing bicycles<\/h2>\n

      Bicycles fare even worse. They sit second from last, in a near-tie with planes, which come in last:<\/p>\n

      \n
      \n
      \nFigure\u00a02: Mean vehicle rating by vehicle, all models pooled.
      \n<\/figcaption><\/figure>\n<\/div>\n<\/div>\n

      If labs were training on the benchmark, you\u2019d expect bicycles near the top of this ranking. They\u2019re not. However, the same caveat applies here. A bicycle is harder to draw than a skateboard: it needs two matching wheels, a frame that reaches both axles, handlebars, a seat, and pedals. The judge flags a missing or disconnected one of those on 2\/3 of the bicycle images. You can train on bicycle images and still not do a great job relative to simpler vehicles.<\/p>\n

      One note on the plane, though: I should\u2019ve picked \u201cairplane\u201d instead of \u201cplane\u201d because models often read it geometrically. They drew the animal standing on a flat surface instead of flying an aircraft. The plane is the only vehicle where the feature extractor sometimes found no vehicle at all (25 of 168 images, against zero for the other five), and 20% of plane images scored a 1 or 2 on the vehicle rating, against 5% for bicycles and none at all for boats, scooters, or skateboards.<\/p>\n<\/section>\n

      \n

      Evidence #4: Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty<\/h2>\n

      Put the two together and the \u201cpelican on a bicycle\u201d ends up near the bottom of the ranking, at #42 of 48:<\/p>\n

      \n
      \n
      \nFigure\u00a03: All 48 combos ranked; pelican + bicycle highlighted.
      \n<\/figcaption><\/figure>\n<\/div>\n<\/div>\n

      But again, some combinations might be just harder to draw than others.<\/p>\n

      To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal \u00d7 vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal \u00d7 vehicle terms absorb the inherent difficulty of all 48 combinations. The interactions measure each lab\u2019s benchmark-specific boost relative to the average lab, with confidence intervals.<\/p>\n

      The results:<\/p>\n

        \n
      1. Every per-lab pelican effect (the lab\u2019s boost on pelicans across all six vehicles) lands between -0.11 and +0.14 judge points, and none comes close to significance (smallest p = 0.25).<\/li>\n
      2. The per-lab bicycle effects (the lab\u2019s boost on bicycles across all eight animals) run from Grok 4.5<\/em> at -0.18 (p=0.11) to Gemini 3.5 Flash<\/em> at +0.27 (p=0.022). Only Gemini clears p < 0.05, and the seven point in both directions.<\/li>\n
      3. No pelican-bicycle cell effect (the extra<\/em> boost on the specific combination, on top of the lab\u2019s pelican and bicycle effects) clears p < 0.05. The largest positive is GLM-5.2<\/em> at +0.35 (p=0.12), which is the one I mentioned earlier. It\u2019s the closest thing to a signal in this experiment, but still within chance.<\/li>\n<\/ol>\n

        Here are the full per-lab estimates. A pelicanmaxxing lab would show dots to the right of the zero line across its whole row:<\/p>\n

        \n
        \n
        \nFigure\u00a04: Difficulty-adjusted effects per lab, with 95% confidence intervals. Highlighted intervals exclude zero.
        \n<\/figcaption><\/figure>\n<\/div>\n<\/div>\n

        Every pelican interval and every cell interval contains zero. Exactly one doesn\u2019t: Gemini 3.5 Flash<\/em> in the bicycle column. But with 21 tests at p < 0.05, chance alone predicts about one false positive (21 \u00d7 0.05 \u2248 1.05), and one is exactly what came up. It also doesn\u2019t survive a multiple-comparisons correction: the Bonferroni threshold across the 21 tests is 0.05\/21 \u2248 0.002, and its p-value is 0.022. The full table of estimates and p-values is in the repo<\/a>.<\/p>\n

        But these intervals are wide, about \u00b10.6 judge points on average. Any boost smaller than that won\u2019t be captured by this test.<\/p>\n<\/section>\n

        \n

        Evidence #5: The pelican-bicycle scenes don\u2019t look memorized<\/h2>\n

        Some have suggested that the pelican on a bicycle looks like a memorized composition, pointing to recurring patterns such as the pelican always facing right, or recurring elements like a sun or a scarf. So I wanted to know if this was true.<\/p>\n

        Direction<\/strong>: All 21 pelican-bicycle images, across all seven labs, face right. No other animal\/vehicle combination does that.<\/p>\n

        However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it\u2019s strongest:<\/p>\n\n\n<\/thead>\n\n\n\n\n\n\n\n
        scooter<\/td>\n9%<\/td>\n83%<\/td>\n8%<\/td>\n<\/tr>\n
        bicycle<\/td>\n11%<\/td>\n81%<\/td>\n8%<\/td>\n<\/tr>\n
        skateboard<\/td>\n19%<\/td>\n60%<\/td>\n21%<\/td>\n<\/tr>\n
        plane<\/td>\n21%<\/td>\n58%<\/td>\n21%<\/td>\n<\/tr>\n
        unicycle<\/td>\n22%<\/td>\n45%<\/td>\n33%<\/td>\n<\/tr>\n
        boat<\/td>\n33%<\/td>\n35%<\/td>\n32%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        Pelicans are also among the animals that tend to face right:<\/p>\n\n\n<\/thead>\n\n\n\n\n\n\n\n\n\n
        antelope<\/td>\n21%<\/td>\n78%<\/td>\n1%<\/td>\n<\/tr>\n
        pelican<\/td>\n22%<\/td>\n78%<\/td>\n0%<\/td>\n<\/tr>\n
        heron<\/td>\n22%<\/td>\n77%<\/td>\n1%<\/td>\n<\/tr>\n
        whale<\/td>\n34%<\/td>\n65%<\/td>\n1%<\/td>\n<\/tr>\n
        flamingo<\/td>\n36%<\/td>\n64%<\/td>\n0%<\/td>\n<\/tr>\n
        otter<\/td>\n8%<\/td>\n45%<\/td>\n47%<\/td>\n<\/tr>\n
        cat<\/td>\n8%<\/td>\n40%<\/td>\n52%<\/td>\n<\/tr>\n
        raccoon<\/td>\n3%<\/td>\n36%<\/td>\n61%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        It\u2019s hard to draw a pelican or a bicycle facing the viewer, so models almost always draw them from the side, facing left or right. That\u2019s why so few of their images are ambiguous. Other combinations also come close to unanimous: antelope on a scooter and pelican on a scooter land at 20 of 21, and heron on a bicycle at 19 of 21. So 21 out of 21 doesn\u2019t seem like an outlier.<\/p>\n

        Scene elements<\/strong>: I let the extractor name any element it saw in the image. These are the counts:<\/p>\n

        \n
        \n
        \nFigure\u00a05: Share of images containing each element, from the open-ended extraction pass.
        \n<\/figcaption><\/figure>\n<\/div>\n<\/div>\n

        A memorized scene would show up as the same set of elements recurring picture after picture. I went looking for that, and found some combinations do tend to produce the same elements every time. Every single flamingo on a boat has a sun in it. Otters on planes wear scarves 38% of the time. Cats on bicycles get a basket 38% of the time.<\/p>\n

        The pelican on a bicycle doesn\u2019t seem to have anything particularly different about it. It just has some elements that appear more frequently, like every other animal-vehicle combination.<\/p>\n<\/section>\n

        \n

        Limitations<\/h2>\n
          \n
        1. Using a single LLM judge for scoring.<\/strong> Every score here comes from one model, GPT-5.6 Luna<\/em>, looking at one image at a time. I didn\u2019t do much alignment and didn\u2019t check how often it agrees with itself on a re-run. If a model just can\u2019t judge a drawing reliably, none of the numbers above mean much. The judge is also from the same family as one of the contestants, GPT-5.6 Terra<\/em>. However, every lab draws all 48 combinations, so a judge that happens to like one lab\u2019s style lifts that lab\u2019s whole grid at once. But that doesn\u2019t change the results because this analysis only cares about the within-lab differences.<\/li>\n
        2. SVGmaxxing.<\/strong> A lab that optimized SVG generation as a whole<\/em> (or a subset such as animals on vehicles) rises on every cell at once and looks identical to a lab that\u2019s just good. Some labs, such as Google\/DeepMind, openly<\/a> do this. This experiment can\u2019t detect that.<\/li>\n
        3. Limited budget.<\/strong> The whole experiment ran on roughly $80 of API credits. That capped it at 3 samples per cell, a single judge, and 7 models. This also prevented me from iterating too much on the prompts and pipeline, as with the \u201cplane\u201d vs.\u00a0\u201cairplane\u201d case.<\/li>\n<\/ol>\n<\/section>\n
          \n

          Conclusion<\/h2>\n

          Sorry, HN haters, but there\u2019s little evidence that AI labs are pelicanmaxxing. Or at least they\u2019re not doing it in a plainly obvious manner.<\/p>\n

          Pelicans aren\u2019t drawn any better than other animals. Bicycles aren\u2019t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict. GLM-5.2 comes closest: it has the largest boost on the exact pelican-bicycle cell, and its first pelican-on-bicycle sample caught my eye. But the effect is small and not significant, so I wouldn\u2019t put too much weight on it.<\/p>\n

          The other thing that stands out is direction in the scene composition. All 21 pelican-bicycle images face right, the only combination in the grid where every image agrees. But it doesn\u2019t seem that strange. Facing right is the norm across the experiment. Three other combinations land at 90% or above, and with 48 of them, I\u2019m not surprised one reached 21 out of 21.<\/p>\n

          The more plausible story is SVGmaxxing like Google\/DeepMind does. Other labs might be doing it more quietly. Sadly, this experiment can\u2019t say who\u2019s doing it. But at least you can sleep tonight knowing that AI labs are not producing terabytes of pelicans on bicycles just to trick Simon Willison.<\/p>\n

          If you want to look at the data yourself, the full pipeline is in the repo<\/a>.<\/p>\n<\/section>\n

          \n
          \n

          Citation<\/h2>\n
          \n

          BibTeX citation:<\/p>\n

          @online{castillo2026,\n  author = {Castillo, Dylan},\n  title = {Are {AI} Labs Pelicanmaxxing?},\n  date = {2026-07-18},\n  url = {https:\/\/dylancastillo.co\/posts\/pelicanmaxxing.html},\n  langid = {en}\n}\n<\/code>