This is such a poor idea. LLM Agents don't have taste (abstraction capabilities). They are kind of like water, they take the shape of the container they're in. They are so much powerful when you build your own framework they can work in.
agents are unlike humans in many ways but they have their own limitations. it's magical thinking to believe that they are able to cope with zero software abstractions, especially as a codebase increases in size and complexity
and today's agents increase codebase size and complexity much faster than humans do via traditional software engineering methods, so this is likely to hit a wall faster than if human software teams were to do this experiment.
it sounds like a bad abstraction. ime llms are really bad at reasoning through efficiency and utilization problems. they fail at scheduling and bin-packing for the same reason. it's actually difficult, multivariate, and not great.
i would be shocked if a well designed framework were less efficient than "bespoke" clankercode for low-level work
This is probably true. Today. But I think we're finding the problems LLMs are solving to be both paradoxically predictable but surprising at the same time. Not saying you're wrong, but scheduling and bin-packing seem like cards that will eventually fall to a bespoke prompt.
I mean, the post was written by a Stanford CS lab, so I'm inclined to believe that they know what they're talking about and are not just bad at creating abstractions.
I know nothing about the PhD candidate and professor who co-authored this, but I don't think this statement makes sense in general.
Typically CS labs at universities contain people who have not spent a significant amount of time exposed to large industry codebases and the corresponding complexity. I think the post would have more credibility coming from e.g. the platforms team at a tech company with a monorepo.
The machine learning community decided to choose Python as the main language, which costs probably tens of millions of dollars in bugs every year due to it not being type checked (astral sh is fixing this though)
I think it's generally fair to assume that you don't become a Stanford CS professor by being bad at writing code and creating abstractions, and that the average professor (let alone one at a prestigious university) is more knowledgable than your average HN commentor
and today's agents increase codebase size and complexity much faster than humans do via traditional software engineering methods, so this is likely to hit a wall faster than if human software teams were to do this experiment.
i would be shocked if a well designed framework were less efficient than "bespoke" clankercode for low-level work
Typically CS labs at universities contain people who have not spent a significant amount of time exposed to large industry codebases and the corresponding complexity. I think the post would have more credibility coming from e.g. the platforms team at a tech company with a monorepo.