> Separate question, separate table. This is our standard latency harness (three short prompts, five reps, 400-token cap), not the build tasks. tok/s is output tokens over wall-clock, uniform for all.
> so their tok/s is a ceiling, not a true decode rate. The clear read: the GPT-5.6 tiers are the snappiest models here on short prompts (Luna answers in about a second), Qwen is absurdly cheap and fast, and DeepSeek and GLM are the slowpokes
You put in a lot of good work, and kudos for that, but man, reading paragraphs like these just puts me off of the entire piece.
Like…how hard would it have been really to type these two sentences by hand, in your own natural voice?
"This isn't objective." Correct, and we are not pretending it is. We are not handing down a scientific verdict.
Actually, you are doing rational investigation in a fuzzy probabilistic new/emergent space, with open sharing to the world. I don’t understand why people downplay themselves and put on a pedestal others supposedly serious sciences.
It's a preemptive defense against methodology cynicism seen often on sites including but not limited to Hacker News. I've been guilty of including such defenses myself over the years because I've gotten annoyed with receiving such cynicism.
because if you don’t put this disclaimer the top comment is always "Acthually this isn't real science because you didn't publish your P value" so you can't win.
also the article itself is clearly LLM generated though
Ultimately advocates exist for models and there are incredible financial incentives for some to be advocates, so authors are guaranteed someone being mad if their horse doesn't perform well.
Given that type of reaction is inevitable, it just saves the conversation.
Because for its entire existence, the top HN comment on articles is typically a contrarian take or pointing out flaws. This goes double for a study, where people just hunt for some aspect of the methodology they dislike. If you don't address the flaws, then it looks like you never considered them, and the top comment will say that your entire methodology is suspect. It's super predictable to the point that you can harness this kind of reaction to get stuff on the frontpage if you really want to.
There's a ton of arenamaxxing going on (especially from facebook), though I don't disagree that it's one of the better actual benchmarks.
Always fun to ask them to recreate classic demoscene effects (sadly they're still pretty bad at generating music, though at least claude seems to create decent synths).
I keep trying to get them to recreate the fluid+particle stuff from Agenda Circling Forth etc., but even giving them the blog posts describing the implementation (and screenshots) they're still pretty bad.
It's really interesting to see the Sol/Terra/Luna apps side-by-side.
I need to add these stats somewhere in the UI, but one interesting take away: Terra took 1/2 as much wall-clock time as Sol, but Luna took more wall-clock time than Sol (by about 23%). It's still much much cheaper, but it seems like Terra is likely a more optimal time/cost balance for most use cases.
The Terra quality is usually nearly as good as Sol, but much faster and cheaper. I do appreciate Sol's design sensibilities (see, for example, the audio sequencer). It's the first model in a while that is clearly distinct on that front. They'd all converged to very similar visuals for a while.
Obviously AI-written, but I'm confused with the results: Muse Spark has the best Rubik's cube by far, the only one properly animating, yet it gets a 2/5
Ah, I missed that, and didn't click through the links. Most of the videos are not showing any animation for me, only Opus / Qwen / Muse, so Grok's attempt looked broken.
A lot of these are visual-heavy tests that often require first person sight to confirm results. Considering GLM isn’t multimodal, that might explain why it did better on the calculator question and not much else.
My concern with most of these visual benchmarks, popular as they are, is that they are likely more indicative of knowledge (i.e. how comprehensive the training data is and how well it can be retrieved from the model) than of reasoning ability. I don't see in particular how a model would construct a CoT that mapped somehow to a representation of the cube geometry and its animations in latent space without a large chunk of that being pre-existing information.
Missing the exact prompts - would love to replicate...but also curious how you prompted these: they could be a big reason why some models failed completely at rendering SVGs (ie. GLM 5.2)
Really nice breakdown, surprised by the results - especially the fact that OSS models were so behind on most task... (lol at the SVG of the moon without any sign of life by GLM-5.2)
Is this how I learn that Bezos now has a beard? Interesting that it is a detail that all of the models chose to include (unless that was in the prompt and just not put in the post).
How much does Grok 4.5 cost on TryAI?
Grok 4.5 is Input: $0.02 / 1M tokens, Output: $0.06 / 1M tokens. There is no subscription — you pay only for what you use.
Those numbers are incorrect unless they got a deal that's 100x cheaper than API pricing which is unlikely. They updated the original post with the correct costs.
It’s interesting how all the model names and versions are like SKUS taking up space on a display shelf. I look forward to whatever Sagittarius A* does!
Interesting tests being done but I can't help but think it limits testing innovation in some way given that the requested apps are essentially all clones of others
I hear this take a lot, but every app I’ve ever built was like 80% similar to every other app out there. The unique/ creative part of an app is not the bulk of it, and LLMs have been pretty good at helping me explore the 20%, too.
Calculator / Rubik's cube / game of life apps should be very close to 100% identical, right? I don't see the point of asking an AI for one of these when there are dozens (hundreds?) of repos that all have exactly what you want.
> We generated a big pile of artifacts, we are publishing all of them, and you can form your own opinion.
My opinion is that two gimmicky "one-shot prompting shootout" marketing pieces in two days smells like desperation. I'm not sure you understand what a turnoff this is for potential customers.
> so their tok/s is a ceiling, not a true decode rate. The clear read: the GPT-5.6 tiers are the snappiest models here on short prompts (Luna answers in about a second), Qwen is absurdly cheap and fast, and DeepSeek and GLM are the slowpokes
You put in a lot of good work, and kudos for that, but man, reading paragraphs like these just puts me off of the entire piece.
Like…how hard would it have been really to type these two sentences by hand, in your own natural voice?
So no matter how good or thoughtful the writing is it gets tiresome.
On the other hand, do we have to complain about every seemingly AI written text?
Look at the top comment on their previous HN submission: https://news.ycombinator.com/item?id=48839886
also the article itself is clearly LLM generated though
Given that type of reaction is inevitable, it just saves the conversation.
Agent: https://arena.ai/leaderboard/agent
Web dev: https://arena.ai/leaderboard/code/webdev
Currently Fable and 5.6 are neck and neck on web dev which is basically the same finding as this.
Always fun to ask them to recreate classic demoscene effects (sadly they're still pretty bad at generating music, though at least claude seems to create decent synths).
I keep trying to get them to recreate the fluid+particle stuff from Agenda Circling Forth etc., but even giving them the blog posts describing the implementation (and screenshots) they're still pretty bad.
https://arena.logic.inc/
It's really interesting to see the Sol/Terra/Luna apps side-by-side.
I need to add these stats somewhere in the UI, but one interesting take away: Terra took 1/2 as much wall-clock time as Sol, but Luna took more wall-clock time than Sol (by about 23%). It's still much much cheaper, but it seems like Terra is likely a more optimal time/cost balance for most use cases.
The Terra quality is usually nearly as good as Sol, but much faster and cheaper. I do appreciate Sol's design sensibilities (see, for example, the audio sequencer). It's the first model in a while that is clearly distinct on that front. They'd all converged to very similar visuals for a while.
(edit: seems to be an issue with inline videos)
https://d1md4c6gq9re9p.cloudfront.net/blog/gpt-5.6-buildoff/...
https://d1md4c6gq9re9p.cloudfront.net/blog/gpt-5.6-buildoff/...
https://d1md4c6gq9re9p.cloudfront.net/blog/gpt-5.6-buildoff/...
https://www.tryai.dev/models/grok-4.5
My opinion is that two gimmicky "one-shot prompting shootout" marketing pieces in two days smells like desperation. I'm not sure you understand what a turnoff this is for potential customers.