Yea I can believe all these. I've personally have been having issues on these points:
"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.
"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them
I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times.
The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
> to the point I wonder if I should just use the providers directly.
How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful.
Personally I pin a single vetted provider in the interest of minimizing risk.
I use Fireworks for a production app. Much lower volumes though. My only gripe is their serverless offering for reranking has only 1 model which is Qwen 8b and it's quite expensive.
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
Just use a single vendor. Literally nothing wrong with that, and you avoid the complexity of both n-1 of the vendors (leaving you with the highest quality vendor) as well as the issues with the aggregating layer.
Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
I never tested this but always suspected it. When I learned that providers differentiate themselves on how they optimize and host the model (otherwise, why would you choose one over another?), I figured some were less "give me the best possible experience" than others.
Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
The way that OpenRouter frames it with having calls routed to whatever provider is handy encourages you to think that they’re all the same in basic functionality. Really sours me on OpenRouter.
could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
Be sure to spend them! Another great feature of OpenRouter is that they will take your credits after a year. They just delete them. A ToS-legalized theft if you ask me. (Even if you are still actively using those credits.)
As of right now, they only expire credits if you don't use the account at all for a year. This is different from, say, OpenAI, where your credits expire unconditionally. That said, they do deserve the right to expire them regardless, so this could change in the future.
It's good to quantify the extent to which a lot of the stuff on there is just vibe-hosted. But I think it was always pretty apparent that this was the case. You've got numerous providers all running the gamut of:
I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
The article that I read addressed this point specifically. "Trusted" providers do not maintain consistent performance, and do not have the same performance across different models.
Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs.
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
As expected vibecoding bros cannot even read the manual properly.
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.
The article mentioned they pinned the provider and model and the result was bad as well with their own shenanigans. That part was in the end of the article so maybe you missed it.
"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.
"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them
The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful.
Personally I pin a single vetted provider in the interest of minimizing risk.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
https://app.answerhq.co/openrouter-ai/articles/credits/credi...
I feel there is still a lot of progress to be made before we can really trust LLM providers.
* Hardware availability
* Competency
* Scruples
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.