{"id":22904,"date":"2026-07-30T03:27:08","date_gmt":"2026-07-30T03:27:08","guid":{"rendered":"https:\/\/scannn.com\/i-handed-a-ux-review-over-to-ai-heres-what-happened\/"},"modified":"2026-07-30T03:27:08","modified_gmt":"2026-07-30T03:27:08","slug":"i-handed-a-ux-review-over-to-ai-heres-what-happened","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/i-handed-a-ux-review-over-to-ai-heres-what-happened\/","title":{"rendered":"I handed a UX review over to AI. Here\u2019s what happened."},"content":{"rendered":"\n
\n
\n

AI is a powerful partner for UX reviews, not a replacement for the expert, and the accuracy comes from the expertise you feed it, not the model.<\/h2>\n
\n
\n
\n
\n
\n
\n
\n
\n
<\/p>\n
\n
<\/div>\n<\/div>\n

<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n

<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

I got a task that isn\u2019t complicated in itself: I had to write a UX expert review for the search, results, and product detail pages of a portal. I had a set of questions about the target audience, the goals, and the USP, an industry research document, plus access to Google Analytics and Clarity.<\/p>\n

My starting hypothesis (one you read more and more everywhere) was that AI is already more than good enough for this kind of thing. You barely have to work with it; the designer just has to look over the result, and on topics like this it basically replaces the human. Since a task like this happened to come along, I figured I\u2019d check whether that\u2019s really true.<\/p>\n

The test<\/h2>\n

I worked with two models in parallel: Claude Fable 5<\/strong> and ChatGPT-5.6 Sol<\/strong>, both on paid subscriptions, of course. I gave each of them everything I had as input: the brief, the questions and answers, the research material, the GA exports, the Clarity heatmaps, attention maps and scroll maps, plus the necessary URLs. The task was precisely defined: produce a UX expert review document, with a specific goal.<\/p>\n

Importantly, I didn\u2019t work with a single prompt. I broke the task down in a structured, multi-step way, and I asked the models to back up their findings with the GA and Clarity data where possible, rather than just assuming. In other words, I wasn\u2019t testing the \u201cthrow everything into one prompt\u201d approach, but what you can get out of a deliberate workflow.<\/p>\n

And naturally, I also did the analysis by hand in parallel (like an animal), so I\u2019d have something to compare against.<\/p>\n

What I found<\/h2>\n

Let me say this up front. This is an experience report, not a controlled experiment. One project, one expert (myself), and my own analysis isn\u2019t an absolute yardstick either. Read the following with that in mind.<\/p>\n

Both models put together an analysis, but they were full of irrelevant, and even non-existent, problems. There were useful observations too, but even those I could only really use at the level of wording and structure. Claude was noticeably more accurate than ChatGPT, at least in phrasing and structure, though this is my subjective impression, not a controlled measurement. One difference was objective, namely that Claude could produce the result in a single document while ChatGPT only managed it in part, but that\u2019s more a question of output format than of analytical quality.<\/p>\n

I\u2019ll admit it: there were problems I hadn\u2019t spotted myself, and that was useful. But there were more that the AI, in turn, didn\u2019t catch.<\/p>\n

This actually lines up quite well with a 2025 study that tested GPT-4o against Nielsen\u2019s usability heuristics:<\/p>\n

\n

\u201cGPT-4o found roughly a fifth (21%) of the usability problems identified by the experts, while also raising several new, partly false problems. So it\u2019s an excellent first-pass UX review tool, but on its own it doesn\u2019t yet replace expert evaluation.\u201d
<\/em>Can GPT-4o Evaluate Usability Like Human Experts? (2025)<\/a><\/p>\n<\/blockquote>\n

On a project somewhat more complex than these Nielsen-style usability heuristics, I found essentially the same thing.<\/p>\n

How the material finally came together<\/h2>\n

In the end I put the analysis together with Claude. I fed it my own observations, and in an ongoing conversation we shaped what to cut and what to add. It reviewed the text for grammar, consistency and similar aspects, and suggested changes. That part was incredibly useful.<\/p>\n

\n

Get Kolozsi Istv\u00e1n (kolboid)\u2019s stories in\u00a0your\u00a0inbox<\/h2>\n<\/p>\n
\n

Join Medium for free to get updates from\u00a0this\u00a0writer.<\/p>\n<\/div>\n

\n

Remember me for faster sign in<\/p>\n<\/div>\n

<\/label><\/div>\n<\/div>\n

I didn\u2019t measure the time precisely, but by feel: if I\u2019d done everything on my own, it would have taken roughly 15% longer overall.<\/p>\n

So AI is a great companion and helping partner that lets me speed up and sharpen my work, but right now it can\u2019t replace me. It\u2019s still a strong partner, and together we\u2019re better than ever.<\/p>\n

\n
Press enter or click to view image in full size<\/span><\/p>\n
\"A<\/picture><\/div>\n<\/div>\n<\/figure>\n

You might ask whether a browser walk-through would have been better. These days there are ways and tools to have the models walk through the site live in a browser, rather than only working from static data and URLs. I can\u2019t know for sure, since I didn\u2019t run it, but I strongly suspect it wouldn\u2019t have led to better results either, because the core limitation isn\u2019t access, it\u2019s judgment, meaning understanding the context and the goals, weighing severity, separating real problems from noise. A live walk-through changes little about that.<\/p>\n

What I will actually try next is different. I\u2019ll build my own agent skill for this task, one that encodes the established usability heuristics, the severity weighting shaped by professional experience, and the expert lens a UX practitioner brings to an interface, and then refine it project by project. The point isn\u2019t for it to decide instead of me, but to turn my own expert judgment into a reusable, consistent form. That\u2019s what I\u2019ll look at next time.<\/p>\n

This actually lines up with what you see in the research on the topic: Baymard\u2019s 95%-accuracy AI heuristic evaluations<\/a> weren\u2019t achieved through better prompting, but by grounding the model in their own researched UX knowledge base. Granted, in their case this is narrowly about usability heuristics, which is a much simpler story than a complex, context-dependent review like this one, but the principle is the same. The accuracy doesn\u2019t come from the model, it comes from the expert framework you feed into it. That\u2019s exactly why it\u2019s worth encoding your own standard into a skill.<\/p>\n

And in the long run?<\/h2>\n

I\u2019m well aware that sooner or later AI will replace our work, that\u2019s essentially not in question. The only question is when it will happen.j<\/p>\n

I\u2019ve read that there are signs of a kind of slowdown, a plateau. According to a late-2024 Reuters piece<\/a>, several leading AI researchers, including Ilya Sutskever, say that simply scaling up model pretraining has reached its limits. Further progress increasingly comes from new approaches, such as \u201co1\u201d-style models that reason step by step, rather than from sheer size. In other words, development hasn\u2019t stopped, but the emphasis is shifting from size toward smarter methods.<\/p>\n

Of course, all sorts of other developments and research are going on in the background, so there\u2019s no question that AI will gain total ground in the long run.<\/p>\n

I recently stumbled on a page suggesting we still have a few good years left, especially in research and complex design, where we use various AI models everywhere to speed up, sharpen and support the work.<\/p>\n

See when AI will replace your job<\/strong><\/a>!<\/p>\n

And while we\u2019re at it, I switch between models depending on which one I use for what, so don\u2019t forget that every LLM is good at different things. Check out this 2026 LLM Compass<\/strong><\/a>!<\/p>\n

\n
Press enter or click to view image in full size<\/span><\/p>\n
\"A<\/picture><\/div>\n<\/div>\n<\/figure>\n

So what should you do tomorrow?<\/strong><\/h2>\n

Start: use AI as a first-pass partner, feed it your real data, work in steps, and codify your own heuristics and severity into a reusable prompt or skill.<\/p>\n

Stop: shipping its output as-is, expecting one prompt to replace a workflow, and handing over your judgment. Severity, prioritization and business context stay with you.<\/p>\n

And stop panicking that it will take your job tomorrow. It won\u2019t, but the designers who learn to direct it will outpace the ones who don\u2019t.<\/p>\n<\/div>\n

Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

AI is a powerful partner for UX reviews, not a replacement for the expert, and the accuracy comes from the expertise you feed it, not the model. I got a task that isn\u2019t complicated in itself: I had to write a UX expert review for the search, results, and product detail pages of a portal. […]<\/p>\n","protected":false},"author":16,"featured_media":22905,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-22904","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22904","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=22904"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22904\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/22905"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=22904"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=22904"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=22904"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}