Launch HN: Design Arena (YC S25) – Head-to-head AI benchmark for aesthetics
Hi HN, I’m Grace from Design Arena (https://www.designarena.ai/) - we’re building a crowdsourced benchmark for AI-generated visuals (websites, images, video, and more). We put AI models and builder tools in head-to-head comparisons that get voted on by real users from around the world. Think “Hot or Not” for the AI era :)
(Btw, when we say real users we mean real users, so you may get a captcha on the site. Sorry, but we have to use every bot protection available! We only want human ratings, for obvious reasons.)
Here’s a demo video: https://www.youtube.com/watch?v=vPyEQnuVgeI
We didn’t set out to build this - we were actually working on an AI game engine. But we found that models sucked at look-and-feel. Even when the output code was usually functional, most visual aspects lacked the soul that makes great graphics feel alive.
So we built a this-or-that game, just for ourselves, to figure out which generated outputs had the best graphics. To our surprise, that turned out to be more exciting than the original idea—it turns out this is a widespread problem! We did a Show HN a month ago (https://news.ycombinator.com/item?id=44542578) and that was partly what convinced us to make this benchmark thing our actual product.
State-of-the-art models might be winning IMO gold, but they are still putting white text on a white background. There needs to be some measurement of what’s good and what isn’t (yes, there is such a thing as good design!), and it sure isn’t going to come from LLMs.
We come from engineering backgrounds (Apple and Nvidia) with a love for design; we know when we like or dislike something, even when we can’t say why. This-or-that / hot-or-not games are made for domains like this: Design Arena’s goal is to make everything stupidly simple so humans can just do the easy part: like-vs.-dislike. Which also turns out to be the valuable part, because what’s easiest for humans is actually the part that the AIs can’t currently do.
Since our Show HN, we’ve extended our initial set of ~25 LLM models to 54 LLM models, 12 image models, 4 video models, 22 audio models, and 22 vibe-coding tools (like Lovable, Bolt, v0, Firebase Studio, and more). In this last category, we’ve been surprised to find that agentic tools that were not specifically marketed as vibe-coders like Devin performed exceedingly well in the builder category, outperforming dedicated builder tools like Lovable, v0, and Bolt.
Our users are mostly devs who want to spin up a frontend, or designers who want to spin up design variants faster. In both cases, Design Arena provides a quick way to find out which options are better than others. Dev-or-designer needs to make the final calls, because there’s no substitute for good judgment. But this type of formatting can really help.
We plan to make money by offering version testing as a service to companies that need to quantify improvements in their product between builds.
This is the first time we’ve ever worked on something like this! We’d love to learn from you all and look forward to your feedback.
This is actually really needed, current ai design tools are so predictable and formulaic, like every output feels like the same purple gradients with rounded corners and that one specific sans serif font that every model seems obsessed with, it's gotten to the point where you can spot ai-generated designs from a mile away because they all have this weird sterile aesthetic that screams "made by a model"
Exactly - we think the ai design tools are in the equivalent of the 'uncanny valley' territory that a lot of the diffusion models were stuck in just 1-2 months ago; most average diffusion models are still in this local optimum, but the best of the best seem to have escaped it.
I don't think this works right now tbh.
It has the same problem as LMArena (which already had webarena): better aesthetics are so far out of distribution you can't even train on the feedback you get here.
You just get a new form of turbo-slop as some hidden preference takes over. With text output that ended up being extensive markdown and emojis. Here that might be people accidentally associating frosted surfaces with relatively better aesthetics, for example.
The problem is so bad LMArena maintains a seperate ranking where they strip away styling entirely
This is interesting but, speaking frankly, I see many seemingly insurmountable issues. Here are some:
- Contests will often be won not by the entry that best adhered to the prompt, but the best-looking one. This happened in the contest "Input Prompt Build a brutalist website to a typeface maker," which I got as a recent example. The winning entry had megawatt-bright magenta and yellow, which shouldn't appear anywhere near brutalism, and in other design aspects had almost no connection to brutalism either -- but it was the most attractive of the bunch.
- The approach only gets you to a local maximum. Current LLMs aren't very good designers, as you say, so contests will involve picking between mostly middling entries. You'd want a design that's, say, a 9 or a 10 on a 10-point scale -- but some 95% of the entry distribution will probably be between 5.5 and 7.5 or so, and that's what users will get to pick from.
All great points. A limitation with human feedback is that once you start asking for more than binary preferences (e.g. multiple rankings or written feedback), the quality of the feedback does decrease. For instance, many times humans can give a quick answer on preference, but when asked "why" they prefer one thing over the other, they might not be able to full explain it in language. This in general is very much an open area of research on collecting and incorporating the most optimal types of feedback.
I definitely agree with your second point. One idea we're experimenting with is adding a human baseline, in which the models are benchmarked against human generated designs as well.
yes! to the second point, someone in our show HN proposed encouraging human designers to compete in submissions as well - we tried implementing this and found that, at least right now, LLMs are still so bad at design that asking a human to beat them is trivial - our plan right now is to focus more on this once it becomes more of challenges and therefore hopefully more interesting/entertaining
Hmm idk about the focus on aesthetics. GPT image is your top image model, a model which is famously poor on aesthetics (though excellent on prompt adherence). I admit it's a difficult thing to eval, though, as in most side by side comparisons users will always pick the image with better prompt adherence regardless of instructions.
What incentive is there for real users to continuously evaluate models for you for free?
Great concept — definitely needed and will hopefully push the labs to improve design abilities of models!
Yes, exactly. We want to be a forcing function for better design models and agents.
Cool - do you train model that will be the proxy from the votes of persons?
we're not training models or proxying human votes with models
Is this an area that is not yet covered by other user rating benchmark sites like LLMarena?
yes! LMArena recently started pushing "webdev" arena, but there was no explicit emphasis on design or aesthetics, just web-based content
Can you write what you imagine is a good “game dev” prompt?
We keep our system prompts across the board as bare bones as possible: https://www.designarena.ai/system-prompts
As for good game dev prompts, here's one from a user that made a pretty fun game: Make asteroids with 2 computers playing against each other on one screen. There should be asteroids flying and 2 ships being controlled by 2 computers. Pay attention to thoroughly implementing the logic to make the ships avoid asteroids at all costs. Absolutely no user input should be necessary, no click to start, no click to restart. The game starts automatically on load and automatically restarts when either computer is dead. The ships should survive as long as possible. The ships should fly around, avoid asteroids as a priority, but also shoot asteroids and each other. Make ships and asteroids positions random each time. Asteroids should split when shot. The goal is to create a robust algorithm for ships so they can survive as long as possible. The game should be playable at 500x500 screen resolution.
AI is terrible at making nice looking design layout with font selections.
Sure it can make great looking images but nothing can make a nice looking poster or basic page layout.
I’m waiting for someone to solve this. I’m not even sure it takes AI it might just be programmatic.
You've perfectly articulated the gap in the market. The solution isn't just a better image generator. I built a tool called LayoutCraft to solve this exact problem. It focuses entirely on creating a great layout with good font choices automatically. It uses AI to understand the request, but then applies a structured, programmatic 'blueprint' to build the layout. This is how it handles fonts and spacing properly, resulting in a clean design, not a chaotic image.
yes - we're trying to figure out why that is
So this went from a Show HN: to a Launch HN in a month?
(Show HN: https://news.ycombinator.com/item?id=44542578)
yes - the changes since Show HN have been builders (https://www.designarena.ai/builder), audio (https://www.designarena.ai/audio), video (https://www.designarena.ai/diffusion), and compare (https://www.designarena.ai/studio). We also had a feed at some point, but ripped that out because it looked messy
Vibe launch
Curious if you guys got into YC for this idea or something else?
We started out building a platform to one-shot games (single-player and multi-player), but realized that the model you used under the hood really made a difference in functionality and graphics. We started out building the benchmark as an internal tool for ourselves to see which model was the best, but found that benchmarking models on visual "taste" was something that people were generally interested in currently.
Post says they were making an AI game engine, so that's probably what they got in with.