The spread isn't as large as I thought from the headline. It's closer to a toss-up than I would have expected.
The length of the response is a huge factor:
> Students tended to prefer longer responses. The selected answer was 37% longer on average than the alternatives. The longest response won 47.7% of decisive writing comparisons. The shortest still won 25.0%.
So the score is partially a proxy for longest responses.
Makes me wonder how much the reviewers actually read the text. Were lazy evaluators picking the text that looked the longest or most structured without reading it all?
Given that this is increasingly the go-to for a college degree, college needs to rethink its cirricula and place in the world. Or at least get rid of the essay.
Lest it become a place where student and teacher ais go to play pay-to-win social deduction video games.
My professor and teacher friends are painfully aware of how many students submit AI generated essays.
They're adding steps like having the students discuss and defend their essay, which immediately reveals the people who had AI write something and thought they could bluff. This triggers complaints about social anxiety and such, which are unfortunately becoming the go-to defense when unable to discuss the work.
They're also moving toward more in-person writing. Instead of long essays, shorter writing segments as part of the test. Submitting a written essay earns you feedback from the professor and a better understanding of the topic, but that's it.
The homework essay is dead. High schoolers in my home state were just relieved of any take-home assignments because it was pointless now, I guess universities won't be far behind.
So, given that developers these days mix and match models anyway with harnesses like Pi, has anyone tried letting Claude or GPT do the coding, and Gemini do the comments and documentation? Maybe even asking it to "translate" Claude's output messages to better human language for the developer or subagents?
Seems reasonable. You could just switch the model after the code is written, when you need to ask questions. You’ll lose your token cache, but that’s it.
For real, the "Claudish" has become so painful to read that it just takes me out of whatever task I was working on.
When prompted to use simple English without jargon, it's still filled with load bearing honest caveats in every footgun seam it talks about — what I should have led with. <Insert whatever other Claude cliche you prefer>
And I'm not the only one to notice this. Next time it's up for renewal, my team is abandoning it for GH Copilot in order to use literally any other frontier model.
Yes, Claude speaks Claudish but at the end of the day I care about the Ruby, Python, JS it writes. It still does a good job at it even if maybe I prefer the way DeepSeek talks. I did not use other models in an agentic harness.
GPT gets the most out there, but creates interesting nuance that turn an idea into a more creative exercise. Claude is like a more grounded GPT but may miss the nuance. Gemini is my last pick, but does better than the rest for making something clear and understandable. It gets to be exhausting parsing through Claude outputs and breaking it down into something more easily understandable. One way I use to improve this is to ask the LLM to pull upstream ideas from prior work on the topics. This makes me feel better about the possibility of hallucination, gives me alternate places to look, but the model still may pull things out of context or fall over on the interpretation.
Given all that, I can definitely see how Gemini would be preferred. And good for Google, because I would rather my offering be the top choice for the most people rather than better serving a small subset of users.
Makes sense when you consider what type of data Google had available in abundance. Natural written language (e.g., Docs, Gmail, Books) and natural spoken language (e.g., YouTube).
One could argue that Google Colab would supply the training data for better coding performance. I would argue that Colab is mostly used for non-complex (e.g., small number of variables) and self-contained (i.e., runnable in one page) code that can’t train a model for multi-folder and multi-page projects that rely on global connections, which real-life coding would often require.
If that's what Occam's razor would suggest, then Occam presumes a world where actors do what they say. We see more and more the opposite of that in Silicon Valley.
You may well be right about Google not including the said data but your support for this point is weak.
Google also has millions of lines of code in several languages written over decades. Probably access to more non-public (quality) code than most other companies in the world.
I'd love to see the same experiment with responses normalized for length, and with actual essay quality scored separately from how helpful the model's feedback felt
when i use gemini i ask it to stop asking me questions, to stop showing me media and just use text, and be concise in output. with these directions it’s pretty useful
My settings are similar. I also added to be very concise without losing important information. It's output is so slim now that I can ask yes or no questions. Asking for a code snippet gives me just a snippet without an accompanying paragraph of explanation that I would just ignore anyway. Of course I can still ask follow up questions, but it gets right to the point. No filler is a dream.
Having coworkers who use it and thus unfortunately needing to read its output, Claude's "English" is very obviously unnatural-sounding, and extremely distinctive in a bad and irritating way. It's almost like another dialect.
...and of course this article itself has a bit of AI-ish tone to it.
I have the same workflow for about 3 months now (constant). same workflow since opus 4.5 and now with fable (variable). start with a plan about a problem (somewhat similar), write down the plan, use the plan to execute the work. Lately the plans are so verbose and confusing, Im starting to think of skipping the plan stage entirely or tell it to be super concise.
From my own tests, the best model for writing complex, nuanced prose is Opus 4.6. All the next versions are impossible to coax into good writing. Gemini is OK, GPT ok but requires significant prompt tuning.
I use Fable for writing code and some planning. I use GPT for planning/roadmaps and code review, it checks Fable. I use Gemini for stray conversations. Gemini has a better conversational style than GPT, less robotic; it hallucinates more unfortunately, it has plainly fallen quite a bit behind, but it's still highly useful to converse with, research with, etc. And it spares me wasting usage of GPT or Fable. I use Gemini for the 'everything else' category, basically. Gemini never goes anywhere near anything serious.
Fable & Claude Opus 4.x or 5 are terrible to talk to about anything. I gave up on that entirely and just use Anthropic's models for work.
Any less serious technical work I'll use GPT for, as the usage limits are quite fantastic.
The length of the response is a huge factor:
> Students tended to prefer longer responses. The selected answer was 37% longer on average than the alternatives. The longest response won 47.7% of decisive writing comparisons. The shortest still won 25.0%.
So the score is partially a proxy for longest responses.
Makes me wonder how much the reviewers actually read the text. Were lazy evaluators picking the text that looked the longest or most structured without reading it all?
Given that this is increasingly the go-to for a college degree, college needs to rethink its cirricula and place in the world. Or at least get rid of the essay.
Lest it become a place where student and teacher ais go to play pay-to-win social deduction video games.
They're adding steps like having the students discuss and defend their essay, which immediately reveals the people who had AI write something and thought they could bluff. This triggers complaints about social anxiety and such, which are unfortunately becoming the go-to defense when unable to discuss the work.
They're also moving toward more in-person writing. Instead of long essays, shorter writing segments as part of the test. Submitting a written essay earns you feedback from the professor and a better understanding of the topic, but that's it.
When prompted to use simple English without jargon, it's still filled with load bearing honest caveats in every footgun seam it talks about — what I should have led with. <Insert whatever other Claude cliche you prefer>
And I'm not the only one to notice this. Next time it's up for renewal, my team is abandoning it for GH Copilot in order to use literally any other frontier model.
Given all that, I can definitely see how Gemini would be preferred. And good for Google, because I would rather my offering be the top choice for the most people rather than better serving a small subset of users.
One could argue that Google Colab would supply the training data for better coding performance. I would argue that Colab is mostly used for non-complex (e.g., small number of variables) and self-contained (i.e., runnable in one page) code that can’t train a model for multi-folder and multi-page projects that rely on global connections, which real-life coding would often require.
You may well be right about Google not including the said data but your support for this point is weak.
at most, the result could be useful for fellow students
I have no doubt other cohorts would rate differently
it is known (on HN at least) e.g. that SWEs tend to prefer brevity, contrary to these students apparently
...and of course this article itself has a bit of AI-ish tone to it.
Fable & Claude Opus 4.x or 5 are terrible to talk to about anything. I gave up on that entirely and just use Anthropic's models for work.
Any less serious technical work I'll use GPT for, as the usage limits are quite fantastic.