I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place??
HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown.
We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.
I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption.
I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.
That used to matter to me back in the days when the best models still only accepted ~32,000 tokens, but these days even the models that run on my laptop are happy with ~100,000 and the hosted models I use take ~200,000 or more.
> Let's say you want to plan a trip to Thailand with your family. You could use the fancy AI to do it for you, or you could build a stupid frontend with minimal natural language understanding.
I don’t disagree with you in principle but there is a difference.
The MD content isn’t intended for human consumption whereas HTML is. So you either have that injection readable to your users, which could be jarring to non-technical readers, or you play the cat-and-mouse game of hoping those pesky crawlers don’t ignore text that’s not human-visible on a rendered page.
Right. And today we have something similar with search engines going on where sites want to offer the full text to the engine for seo. But maybe they want to do the opposite to agent and poison the well. So human agents still get the “real” data and agents the “something plausible but not quite right” data.
Or everything just disappears behind something like cloudflare or a paywall.
HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown.
We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.
I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.
It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now?
There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.
> Why do you need AI then?
(from this HN comment, worth a read in full: https://news.ycombinator.com/item?id=43679585)
It's giving https://news.ycombinator.com/item?id=8863#9224
A cherry.jpg image/jpeg file is shown as an image by the browser
A logo in svg like https://news.ycombinator.com/y18.svg too
So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki
Browsers should implement that asap and we will follow
The open internet deserves a real solution instead of gatekeepers in the form of brightdata, firecrawl, cloudflare and the likes.
The MD content isn’t intended for human consumption whereas HTML is. So you either have that injection readable to your users, which could be jarring to non-technical readers, or you play the cat-and-mouse game of hoping those pesky crawlers don’t ignore text that’s not human-visible on a rendered page.
Or everything just disappears behind something like cloudflare or a paywall.
Maybe tarpit them into reading gigabytes of Markov Chain nonsense, really slowly?