{"id":23294,"date":"2026-08-15T04:58:08","date_gmt":"2026-08-15T04:58:08","guid":{"rendered":"https:\/\/scannn.com\/comparing-11-different-ai-models\/"},"modified":"2026-08-15T04:58:08","modified_gmt":"2026-08-15T04:58:08","slug":"comparing-11-different-ai-models","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/comparing-11-different-ai-models\/","title":{"rendered":"Comparing 11 different AI models"},"content":{"rendered":"\n<div data-astro-cid-7jjqptxk=\"true\" data-astro-cid-cnvtppup=\"\" style=\"\">\n<p>We just <a href=\"https:\/\/www.netlify.com\/blog\/build-with-open-models-in-production\/\">launched a partnership<\/a> with <a href=\"https:\/\/openrouter.ai\/\">OpenRouter<\/a> that lets us offer two new pieces of functionality:<\/p>\n<ul>\n<li>First, your projects can use any model on OpenRouter through our <a href=\"https:\/\/docs.netlify.com\/build\/ai-gateway\/overview\/\">AI Gateway<\/a>. That means that if your own web app offers AI inference-based features to your end users, you now have a much wider selection of models to fit any task and budget.<\/li>\n<li>Second, we\u2019re extending the selection of frontier coding models available for use via <a href=\"https:\/\/docs.netlify.com\/build\/build-with-ai\/agent-runners\/overview\/\">Agent Runners<\/a>. Agent Runners is the chat prompt box you get within Netlify, which lets you build new projects from scratch or iterate on an existing one. The selection of models now includes much-hyped recent open models such as Kimi K3, GLM 5.2, and DeepSeek V4, available to everyone.<\/li>\n<\/ul>\n<p>We call it Agent Runners because we run a full coding agent inside, not a pared-down one. Until now, we\u2019ve supported Claude Agent, OpenAI Codex, and Gemini CLI which are optimized to run models from these providers.<\/p>\n<p>We provide these agents with extra skills, and context about the current project, so that the agent will know exactly which Netlify capabilities are available for use (e.g., <a href=\"https:\/\/docs.netlify.com\/build\/data-and-storage\/netlify-database\/\">Netlify Database<\/a>, the AI Gateway, or <a href=\"https:\/\/docs.netlify.com\/manage\/security\/secure-access-to-sites\/identity\/overview\/\">Identity<\/a>), when to use them, and how. But to effectively drive a whole variety of new models, we\u2019ve added the popular open-source <a href=\"https:\/\/opencode.ai\/\">OpenCode<\/a> as a new choice of agent.<\/p>\n<p>But with more choice come the inevitable questions: How do I know which model is right for me? Am I missing out on something that\u2019s materially better, or more cost-effective (so I can do more with my credits), or is going to blow my mind like the internet says? There\u2019s a lot of FOMO going around these days.<\/p>\n<p>To provide you with some insights, here\u2019s what we learned when running identical prompts across a range of models\u2026 all of which are now available for you to <a href=\"https:\/\/app.netlify.com\/start\">use today on Netlify<\/a>.<\/p>\n<blockquote>\n<p>You can see the results of all the models we tested on <a href=\"https:\/\/the-coffee-shop-brief.netlify.app\/\">this site we created with the full report<\/a>.<\/p>\n<\/blockquote>\n<h2 id=\"what-we-tested\">What we tested<\/h2>\n<p>Internally at Netlify, we use AXIS for automatically evaluating models, a tool that we\u2019ve <a href=\"https:\/\/www.netlify.com\/blog\/how-we-measure-netlify-agent-experience\/\">recently open-sourced<\/a>.<\/p>\n<p>We provide AXIS with a variety of test cases: prompts for building a new site and then iterating on it. We instruct AXIS on which agents and models to test these prompts, and define the checks that AXIS should then perform and score the generated site with.<\/p>\n<p>These checks are very much focused on <em>correct functionality of the generated site rather than its design<\/em>, e.g.: does it use a database when a user\u2019s needs call for it? Does it properly use Netlify Database in that case? In those cases where a simple static site will do, we also ensure that the generated site is not over-engineered, and no database is set up.<\/p>\n<p>If a certain model is behind on its test scores, we don\u2019t offer it in Agent Runners. If models too often fail at correctly applying one of our skills, or things do work but the credit cost seems inflated, then the problem is probably with the skill (in which case we optimize that skill).<\/p>\n<p>But this time, we want to provide you with something much more immediately useful: when you go and build your dream using different models that each use wildly different amounts of credits, what do you get? What do the result <em>look<\/em> like?<\/p>\n<p>We tested three relatively straightforward use-cases:<\/p>\n<ol>\n<li><strong>A site for a local coffee shop.<\/strong> LLMs just love making sites for local coffee shops! The initial prompt is simple, and a static site with no fancy database or the like will do. Then we do a follow-up prompt that asks for a simple option to reserve seats, and check how the model handled that.<\/li>\n<li><strong>A simple to-do list web app<\/strong> in which multiple users can view and add tasks. This calls for a simple design, but requires a shared database from the get-go. Then we ask to support an optional photo upload per item, and check if the model used the <a href=\"https:\/\/docs.netlify.com\/build\/data-and-storage\/netlify-blobs\/\">proper Netlify primitive<\/a>.<\/li>\n<li><strong>A \u201cWhat can I cook\u201d web app<\/strong> that lets users enter what ingredients they have at home, and suggests a recipe using AI. The site itself is rather simple, but we want to check that the generated site correctly uses our <a href=\"https:\/\/docs.netlify.com\/build\/ai-gateway\/overview\/\">AI Gateway<\/a> to generate a recipe for the user.<\/li>\n<\/ol>\n<p>For each of these cases, we\u2019ll show you the look of the generated sites, comment on notable issues, and compare how many credits each took to generate. Of course, this is going to be a much more subjective test than our internal test suites, but it\u2019s also going to be a very fun one. We\u2019d love to know your opinion of the results!<\/p>\n<p>All models were run with their default settings on Netlify. One notable mention is that we currently run GPT 5.6 Sol speicifically on low effort by default, giving you a more economical alternative to Opus that still provides pretty darn good results (as you\u2019ll see below). However, the effort setting is now under your control, and our defaults may change with time.<\/p>\n<p>This post is going to cover only the very first scenario: the static page for a coffee shop, while follow-up posts will focus on going beyond that simple use case. There is much to review even for this simple case, so let us begin.<\/p>\n<h2 id=\"scenario-1-the-local-coffee-shop\">Scenario #1: The local coffee shop<\/h2>\n<p>Here\u2019s our first prompt:<\/p>\n<blockquote>\n<p>Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself.<\/p>\n<\/blockquote>\n<p>The last sentence was added as a hint to the model that no fancy Content Management System is needed. Our default skills also include some UI design guidance, mainly to avoid known gotchas (e.g., the now-dreaded purple AI slop) and get the model to reason about the visual identity appropriate for the user\u2019s ask. But beyond that, each model is free to go build what it thinks we\u2019ll want.<\/p>\n<p>Before we reveal what the sites looks like, here\u2019s a table comparing the credit usage for each model we tested. Each model was run three times, and clicking any of the results will take you to the actual generated site!<\/p>\n<figure>\n<\/figure>\n<p>That\u2019s a pretty wide distribution, eh? Not only that: the Claude Opus average is heavily slanted upwards because one of its three runs spent a whopping 1,055 credits! (As a reminder, on the free plan you have 300 credits; on a Personal plan there\u2019s 1,000 included credits; and with a Pro plan there\u2019s 3,000 included credits. Additional credits packs for Pro are $10 for per 1,500 credits.)<\/p>\n<p>The immediate question is then: is this Opus spend worth it? And what trade-offs do the other models offer? Let\u2019s start digging in.<\/p>\n<h3 id=\"claude-opus-5\">Claude Opus 5<\/h3>\n<p>Here\u2019s the full page generated by that 1,055-credit run (about 4x more than any other run).<\/p>\n<p>To be honest, I think it\u2019s delightful, and full of detail in both its visual design (consider the \u201cstamp like\u201d element with the coffee bean in the center: that\u2019s an actual text element that can be animated), and the custom map at the bottom. Dark mode works out of the box &#8211; go check out the live site in the links above.<\/p>\n<p>Of course, we did not explicitly provide the model with any actual details about our coffee shop (well, except for it being a \u201cneighbourhood\u201d one, which is really steering all models in a certain direction). The design language is hip but perhaps cliche by now (take the two-font, two-color  heading for example), but hey &#8211; we didn\u2019t give it any other direction.<\/p>\n<p>So, how did the other two runs by Opus go? (<a href=\"https:\/\/6a6fa32903f0fac7ed28887d--ar-testing-coffee-4cae03c5378f.netlify.app\/\">253<\/a> credits used on the left; <a href=\"https:\/\/6a6faa299f2311f19d0e66ba--ar-testing-coffee-d73b1653d971.netlify.app\/\">249<\/a> on the right)<\/p>\n<p>Not bad either! Vector graphics actually require a lot of work from the models, and the examples above are pretty much on the frontier in terms of what LLMs <em>currently<\/em> are able to achieve (which is, to be honest, not in a very good place yet compared to image or text generation).<\/p>\n<p>As to whether the first result is truly \u201c4x better\u201d or not, opinions might vary. But in all the tests I\u2019ve done, Opus does have a tendency to run off with excessive credit usage (compared to its \u201ctypical\u201d baseline) more than other models. It does not guarantee a worse or better outcome, though. It\u2019s something that just <em>happens<\/em> pretty frequently.<\/p>\n<p>Let\u2019s look at some other models and then reflect on what we can learn.<\/p>\n<h3 id=\"claude-sonnet-5\">Claude Sonnet 5<\/h3>\n<p>Here are our three contenders, at 143 credits on average (<a href=\"https:\/\/6a6fa3118e3202cc6e567749--ar-testing-coffee-024666b5136f.netlify.app\/\">81 credits<\/a> \u00b7 <a href=\"https:\/\/6a6fab0fa54baec9676aa982--ar-testing-coffee-fec1078e3bba.netlify.app\/\">245 credits<\/a> \u00b7 <a href=\"https:\/\/6a707ae1a8f69f351632aba6--ar-testing-coffee-53c45094318e.netlify.app\/\">103 credits<\/a>):<\/p>\n<p>There\u2019s still some delightful detail in each of these, just <em>less so<\/em> (and less content in general). The vector graphics is noticeably simpler and not really something you\u2019d consider for a live site. This doesn\u2019t say anything about this model\u2019s ability to write complex code or answer philosophical questions, but we\u2019re not asking for this here. At this price point, let\u2019s see what OpenAI, Google and Kimi have to offer.<\/p>\n<h3 id=\"gpt-56-sol-low-effort\">GPT 5.6 Sol (low effort)<\/h3>\n<p>What happens when we take OpenAI\u2019s Opus-class model and ask it to spend a bit less time thinking?<\/p>\n<p>(141 credits on average: <a href=\"https:\/\/6a6fa2978b6896cb693319f7--ar-testing-coffee-d66354fb412b.netlify.app\/\">173 credits<\/a> \u00b7 <a href=\"https:\/\/6a6fa98950d89fe74af76f3f--ar-testing-coffee-170e134f86fc.netlify.app\/\">158 credits<\/a> \u00b7 <a href=\"https:\/\/6a707a7ffb03ad3755a4a536--ar-testing-coffee-aeaf76325340.netlify.app\/\">92 credits<\/a>)<\/p>\n<p>Looking into the results, I think OpenAI\u2019s top-tier model in low effort mode wins over Anthropic\u2019s mid-tier model when it comes to basic design intuition, at least in this scenario. There is more richness in content, and no funky vector shapes (though the images are a bit generic).<\/p>\n<h3 id=\"gpt-56-terra\">GPT 5.6 Terra<\/h3>\n<p>When we go one tier down in OpenAI\u2019s offering (it\u2019s Sol\u2192Terra\u2192Luna), will we see the same drop as the one we just witnessed when switching from Anthropic\u2019s Opus to Sonnet?<\/p>\n<p>Surprisingly, that\u2019s not exactly the case: here it seems like Terra has a <em>different visual language<\/em>, and not a necessarily worse one. It does appear simpler content-wise. There are some visual glitches: a missing image in the left run, low-contrast text over an image in the middle one &#8211; but nothing super wrong.<\/p>\n<p>(39 credits on average: <a href=\"https:\/\/6a6fa2e1afaf80aa13da0536--ar-testing-coffee-31dcbb8e514a.netlify.app\/\">43 credits<\/a> \u00b7 <a href=\"https:\/\/6a6fa9da89470b0ff1160411--ar-testing-coffee-9b6487579ea6.netlify.app\/\">23 credits<\/a> \u00b7 <a href=\"https:\/\/6a707a4ea2fdbb352df928c5--ar-testing-coffee-40263077bc51.netlify.app\/\">49 credits<\/a>)<\/p>\n<p>Up to this point, if I had a very vague idea of what design &amp; language I\u2019d like for a project, my personal inclination would be to run the same prompt with Opus 5 and GPT 5.6 Terra, and get two very different but worthwhile takes.<\/p>\n<h3 id=\"gemini-36-flash--31-pro\">Gemini (3.6 Flash &amp; 3.1 Pro)<\/h3>\n<p>These models are not of the same generation, and it shows: Gemini 3.6 Flash actually produced nicer results (or at least, more in line with other modern models) and used more credits compared to Gemini 3.1 Pro.<\/p>\n<p>Here is what Gemini 3.1 Pro generated for 53 credits on average. I\u2019m not even putting the links to the live site here, because there\u2019s really nothing to see.<\/p>\n<p>Yes, these are wholly separate runs. It did what we asked in the prompt, and really nothing more.<\/p>\n<p>On the other hand, Gemini 3.6 Flash seems like a whole new generation, and used up 103 credits on average (<a href=\"https:\/\/6a6fa3376288679d094a8437--ar-testing-coffee-2b15c6a08894.netlify.app\/\">109 credits<\/a> \u00b7 <a href=\"https:\/\/6a6faa31c66359f0baf64680--ar-testing-coffee-ddce34e684da.netlify.app\/\">91 credits<\/a> \u00b7 <a href=\"https:\/\/6a707aac68023338d968539c--ar-testing-coffee-b14397da2ef1.netlify.app\/\">111 credits<\/a>). It also worked much harder on the content side of things. All models repeat themselves, but it seems like Gemini might repeat itself even more.<\/p>\n<h3 id=\"kimi-k3-and-k27-code\">Kimi (K3 and K2.7 Code)<\/h3>\n<p>Ok, let us get to the open-weight models now. Starting with the latest Kimi K3, here is what we get (102 credits on average; <a href=\"https:\/\/6a6fa38db5f49ba22299d850--ar-testing-coffee-51232072614a.netlify.app\/\">125 credits<\/a> \u00b7 <a href=\"https:\/\/6a719b11b2fb3511019c9247--ar-testing-coffee-8546a78592c4.netlify.app\/\">95 credits<\/a> \u00b7 <a href=\"https:\/\/6a71dcf1be56623f3b1a0a97--ar-testing-coffee-1bb05cd696b9.netlify.app\/\">86 credits<\/a>):<\/p>\n<p>To be clear, Kimi K3 is marketed mostly as a frontier model for long-horizon agentic tasks, and various benchmarks and reviews confirm its prowess in that field. It was built to take on Fable 5 more than Opus 5. But in this narrow design-led task, it does not particularly shine among others. <strong>To really do this model justice, we\u2019d need a wholly different set of prompts engineered for a complex web app,<\/strong> which we will cover in a follow-up post.<\/p>\n<p>Going a big step back in model architecture to Kimi K2.7 Code, here is what we get for a very low credit average of just 19 credits:<\/p>\n<p>Despite some hype about Kimi\u2019s visual capabilities from around the K2.6 model launch, in terms of design or content there\u2019s really not much to see here.<\/p>\n<h3 id=\"glm-52\">GLM 5.2<\/h3>\n<p>Let\u2019s try this: look at these pages, ignore GLM\u2019s love for maple, and try to estimate how many credits were used for each:<\/p>\n<p>Here are the correct answers, from left to right: 15, 42, 24 (on average: 27). Surprisingly, these runs are &#8211; maple aside &#8211; very different, as if coming from a few different models. For the relatively low credit cost of GLM, it\u2019s probably worthwhile to run it a few times before settling on what this model can do for you.<\/p>\n<p>Note that being a text-only model that does not receive image inputs, GLM in its current 5.2 iteration cannot do something that Kimi models can: get screenshots from the user for inspiration, as in \u201cthis is the kind of design I\u2019m looking for\u201d.<\/p>\n<h3 id=\"deepseek-v4-v4-pro-and-v4-flash-0731\">DeepSeek V4 (V4 Pro and V4 Flash 0731)<\/h3>\n<p>V4 Pro is a bit older than the latest V4 Flash revision (also known as 0731). For about 47 credits, it does not provide inspiring results &#8211; especially compared to the mid-tier GPT 5.6 Terra model covered above, which sits at almost the same cost.<\/p>\n<p>The middle run also has a broken image: the HTML file points to an image file that does not actually exist in the project, which is a lot less likely to occur nowadays with any of the commercial models from OpenAI, Anthropic, or Google.<\/p>\n<p>V4 Flash 0731, on the other hand, is both newer and sets a new record here on how few credits it consumes.<\/p>\n<p>For only 2.4 credits on average (<a href=\"https:\/\/6a6fa3053d6a01ebf04690b2--ar-testing-coffee-195abf170d70.netlify.app\/\">3.4 credits<\/a> \u00b7 <a href=\"https:\/\/6a6fa9def726c833ace058bb--ar-testing-coffee-3662600ded0d.netlify.app\/\">1.3 credits<\/a> \u00b7 <a href=\"https:\/\/6a707b37ecdfee1a4343e77e--ar-testing-coffee-29af4ff1b1ac.netlify.app\/\">2.5 credits<\/a>), you get a mixture of results. Interestingly, the middle one doesn\u2019t just look the most like what a mid-tier closed model might give you, but also feels the same in terms of language, and has actually consumed the least credits among all runs.<\/p>\n<h2 id=\"interim-conclusions-and-whats-next\">Interim conclusions, and what\u2019s next<\/h2>\n<p>There are two important notes to make here:<\/p>\n<p>First, for anything beyond a simple website or the initial ideation phase for a project, the question shifts from how nice the model design &amp; copy is to:<\/p>\n<ul>\n<li>Does it know which platform features to use, when and how, to get the functionality you want? Can it store user data, use AI in your web app, and handle authentication and security?<\/li>\n<li>Does it rigorously validate its own work? Can it validate the frontend aspect of your project (that\u2019s where image inputs become crucial)? Can it reliably find and fix issues based on feedback from you, and tell you when your own input is misleading or you\u2019ve overlooked an important concern?<\/li>\n<\/ul>\n<p>In the follow-up posts to this, we will start going into these questions, and (teaser) note some interesting differences in how models craft the project\u2019s code.<\/p>\n<p>My second note is that even considering just this design-and-copy-focused test that I covered, it\u2019s important to consider how much ideation you want the model to come up with on its own. Currently, Opus will probably provide the most clever word games and sleekest design, but you don\u2019t necessarily need it to. Of course, Opus will also perform relentless self-validation of its own work (it does not bill itself on good looks alone). But remember there\u2019s certainly a higher-than-average credit cost attached to that.<\/p>\n<p>Given a limited budget, would you prefer a <em>turnkey solution<\/em> that attempts to pre-plan and handle everything for you, or should you go with a simpler model and a more iterative approach, where you guide the model with follow-up prompts towards what you want? No option here is necessarily wrong.<\/p>\n<p>I hope this post inspires you to test out different approaches, and judge for yourself the quality of results you get. We\u2019re also pretty excited to share with you (very soon!) the results for more advanced web-app use-cases, where the Netlify platform capabilities really shine through.<\/p>\n<\/p><\/div>\n<p><a href=\"https:\/\/www.netlify.com\/blog\/one-prompt-11-models-very-different-results\/?utm_source=tldrdev\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We just launched a partnership with OpenRouter that lets us offer two new pieces of functionality: First, your projects can use any model on OpenRouter through our AI Gateway. That means that if your own web app offers AI inference-based features to your end users, you now have a much wider selection of models to [&hellip;]<\/p>\n","protected":false},"author":16,"featured_media":23295,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23294","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23294","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23294"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23294\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23295"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23294"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23294"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23294"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}