This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, it's bound to work exactly like this.
> I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
This is quite literally the point of OpenRouter. A unified interface, so you can easily switch providers without changing a ton of code which using providers directly would most likely mean, as there are slight differences between them. And the providers all run different weights, so of course quality/performance will differ among them.
I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you, but to actually find the good and usable stuff, you need to focus in on what providers/manufacturers you know are good, and stick with those. Still, the unified interface helps you to shop around and try different ones when you want to.
if you want to control the routing, you'd lock down providers
I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate.
If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models.
This is quite literally the point of OpenRouter.
Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.
> There's not even a way to compare providers, AFAICT
That's not quite true. The only thing they don't show per-provider is benchmark data, cause I don't think they are doing continuous benchmarking of each model from each provider, as I assume they feel that's too expensive. You can see hugely detailed breakdowns for near-time metrics per provider for any model by visiting the page for that model on Openrouter. For example see the page for Qwen 3.8 27B: https://openrouter.ai/qwen/qwen3.8-27b
Some of the killer stats they show per provider:
- Pricing: Effective price accounting for cache hit rate, by provider
- Performance: Throughput in tok/s, latency, E2E latency, tool call error rate, structured output error rate, and more; all per provider.
- Uptime: You have to click on the provider to see their specific uptime, but doing so does show the last-7-days uptime, and you can click to see more.
The thing they don't show is the one we really need, especially because model providers can skimp on quality (run lower quantization, lower kv cache precision, etc) to improve their pricing and performance. I agree that it's probably too expensive to keep running the benchmark, but we need some way to hold the providers to a certain standard, otherwise every user has to discover the problems on their own.
No I know, I was in the GUI as I wrote that lol. As the other person said: if they vary this much in quality, not including that way above updtime and performance is absurd. What would you use a fast, always-up, broken endpoint for?
> I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate.
Why do you care about the public benchmarks at all?
The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it.
Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up.
I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices.
For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out.
If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.
If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity.
I don't think anyone serious is just willy-nilly making individual requests against OpenRouter
Despite your confidence, that is indeed the basis of this massive corporations entire business plan.
If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRoute
If OpenRouter is only for bad accuracy, they should say as much and fade into deserved obscurity.
I think you're focusing only on the general coding agent aspect of LLMs.
> If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity.
That is the way LLMs have to be used for highest reliability. While the term stochastic parrot has been co-opted by unreasonable LLM skeptics, that is indeed what LLMs are. You have to have grounded evals that check outcomes if you want to use them reliably - or a human in the loop works too.
The more general your family of tasks, the less likely you can make automated evals. So for "general" coding agents, you need a human in the loop that can verify and it's not that easy to write an eval.
But if you have specific tasks, then you can spend the time to make a eval, and then you can optimise the way you use the LLM and get extremely good success rates. It's not like it's black magic. Nor does it need large budgets.
> that is indeed the basis of this massive corporations entire business plan.
No. Individual developers using codex (for extremely underspecified general engineering) needs human in the loop, is not amenable to evals but is only a fraction of all LLM usecases.
You'd be surprised. Not enough teams still have their own evals. India[1] and the US[2] is my experience. To get many of them to understand the benefit of putting a couple of people to do data labelling and write a couple of verifiers for just a few days every few months was so difficult. Many folks have understood it all wrong and made allotments like "big model for this task" "small model for this task". Small was sometimes parameters, sometimes brand version number, or sometimes because it has "mini" in its name.
Proper LLM adoption beyond fucking around with claude code or github copilot is so low and is only going to go up as people figure it out. I also think the new cloud agents thing might accelerate adoption among these companies, since it's a bit more plug and play. But they are more likely to be stingy about it, so I'm not sure about the high margin expectations of certain model families.
[1] not witch, but BFSI. Surprisingly parts of witch companies have it down to science already, they embedded openai or anthropic or startups like devrev more than a year back.
1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them.
2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime - but it doesn't work if it's at the cost of shitty performance or failing toolcalls.
> 1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them.
OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable.
> 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
In theory, yes. In practice, no, there are differences. Ollama, llama.cpp, vLLM and SGLang all say "ChatCompletionRequest" compatible, but the devil is in the details, they don't have 100% the same request/response schema across all compatible models.
> 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
Yup, makes sense! If you're unable to find models when you use OpenRouter, it makes zero sense to continue to use OpenRouter.
> You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime
Huh? That's not how it works or does it make sense, nor have I've seen anyone use OpenRouter like that.
> Parent is complaining about the auto-provider chosing, not that all providers are unreliable.
Parent is claiming that the choosing is unreliable. If I rely on a provider to bring me tuna to some spec, but they get it from many different fishermen, it doesn't mean that the tuna doesn't have meet the spec. The complaint is that they're given a bunch of knobs that simply don't work with providers that they could be switched to. That's like saying that I want my tunas to be 20lbs. minimum, and I get switched to a provider that doesn't weigh their tuna at all.
The choosing is all OpenRouter provides. If it doesn't do that, then what is it good for? If I have to permanently pin the one provider who doesn't ignore what I've asked, why shouldn't I just deal with them directly?
edit: it's really supposed to reduce providers to a commodity market. If you're selling e.g. produce to a commodity market, you can't just ship whatever the hell you want. You ship something indistinguishable from others, or more likely the market itself allows you to grade what you're shipping so it's put into a bin with virtually identical stuff. The customer just buys Grade B Wheat.
> I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you
This helps forecast one possible future for OpenRouter: they begin to offer in-house provision, and people begin to migrate their uses off the "marketplace" providers and onto the "fulfilled-by-us" provision.
I treat OpenRouter as a central point to access specific providers. Pinning is a given for that use case.
I’ve tried using un-pinned models and the experience is exactly as you described: Some providers are so unreliable that the majority of requests fail. Some providers do weird things like abruptly end the response (which I get billed for and have to re-submit). Some providers are clearly running heavily quantized versions of the model because their eval performance is terrible. Some providers advertise features on OpenRouter but will reject those requests when submitted to their API.
OpenRouter has the advantage that I can maintain a single balance across model providers so I use it for quick iteration across and then write it into my own router (required anyway since I have many on-prem models running). The portable balance is actually very useful and my credit card is in one place.
If I could pay per request without maintaining a balance or credit card out of a single wallet (using crypto or something maybe) I would happily simply write the integration myself because OR’s caching is often not as good without some hoop jumping.
> to the point I wonder if I should just use the providers directly.
How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful.
Personally I pin a single vetted provider in the interest of minimizing risk.
I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible.
For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.
DigitalOcean (at least through OpenRouter) is pretty reliably bad in my experience. Fireworks can be great, but it depends on the model and on the day.
DigitalOcean is bad not just through OR... Direct accesss through DO gave me such a miniscule context and maximum tokens, it was pretty much unusable. So, accessing it through OpenRouter gave bigger context length, but model behaves like it was seriously damaged - tool calling was producing paths that had / replaced with some other symbols, files were not found etc., and this is just for things that errored out, I have no idea how bad reasoning was. I had to blacklist DO completely on OpenRouter.
I only use OpenRouter to benchmark models. For my production app, I have direct integrations with OpenAI, Anthropic, Google, xAI. Even just input caching is enough of a reason to do that, assuming you are trying to build the fastest and most cost-efficient product possible. I don't understand how people supposedly run production apps using OpenRouter unless those two variables somehow don't matter to them.
This (by definition) the reality of every normalization/proxy layer, standard, etc. There’s always a cost for flexibility/freedom. What’s traded is when and by whom it’s paid.
You know you can define your own provider filters and orderings, right? The filters and such are not _that_ advanced, but it might do what you need if you haven't already tried that.
Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cache. I see performance tank and then I look in the logs and a new provider has been rotated in and every call to them is uncached because they are clearly broken. This has happened a few times now and essentially destroys cost savings (these providers also often have terrible quality). Don't they monitor for simple things like this? Their own logs show how clear this pattern is for some providers. Simple cache % stats would allow them to block providers nearly instantly.
We have made MASSIVE improvements here, and network-wide caching rates have been improving relentlessly. We do publish the cache rates for each endpoint; see the "performance" area of our model pages. E.g.
https://openrouter.ai/deepseek/deepseek-v4-flash-0731#perfor...
You know how there's a router mode to use the cheapest provider? That only takes into account uncached rates, last I checked. Make another one that takes into account effective rates (the ones that include cache).
I think the future -may- be a hosted routing client. Something similar to an auth client app. Why? A few reasons:
- Customers being able to decide their own routing with true logic is a huge feature. Open router provides the seamless switching/api, route switching decisions are available in a client.
- Similarly, providing hooks at this level gives a chance for stats/other things that are hard to plug into prod code elsewhere
- a true middle man hosting for other things like MCP may also turn into a real win once it is implemented.
Just a random thought though. My point about quality/cost being clobbered by bad providers remains. The fact that cache and quality is badly handled makes me doubt that training data choices are being respected. You need a more public trust/certification process for providers with real teeth when they cheat. I'm going to wait a bit to see how things evolve and check back later.
You need to validate what providers are actually serving. Add benchmarks, properly showcase what quantization they are serving on the model and KV cache, etc.
Until that happens, your service is doomed to be shitty.
Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
> Providers probably serve quantized versions without disclosing it
It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers.
In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.
100% agree. The key issue is that users think they are getting results from a specific model configuration and they are clearly not, which is fraudulent.
If OpenRouter wants to succeed as a business, they need to be auditing the providers they connect to (i.e. benchmarking) and removing fraudulent ones from their service.
It's more than just quantization. The middleware the provider is running matters a lot even to the point of exactly which version they are running due to defects being introduced / resolved.
In my coding agent harness I've included 25 open weight providers mainly because I keep having to find new ones when what was previously a great combination of model and provider becomes pretty bad. vllm has defect that causes reasoning to get dropped much of the time for the GLM family of models. sglang has a defect that causes the elements of array args to get dropped for the deepseek family of models. Some providers need some very specific additional config passed through for reasoning to make it back to the model.
I've not tried OpenRouter as adding yet another layer will just make it that much more difficult to get a model and provider combination working well.
I suspect people's bad experiences with open weight models have a lot to do with these headaches. Finding a good model and provider combination is pretty tedious and so far its been a never ending process. I'd really like to host my own models but it isn't economically feasible for one person for the open weight models that work well (i.e. the 300B+ ones).
I mean that is addressed as well later, even when they do declare the quantization, doesn't mean you'll get better performance than the lower quantized one. I'm guessing they're doing something similar to what stadia was doing, saying you are playing games in 4k, because you're getting a 4k stream, but the game itself is running in 1080p
The best part about OpenRouter is 200 OK is probably hardcoded into their responses.
I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.
Yeah, this is pretty accurate. Some providers are basically scams too. I encountered one provider for GLM-5.2 which ran at 200 tps (absurdly high), and was so broken that it would issue 20 full reads of the same file in one turn and quickly jump up to 300~400k context and charge me for the prefill. Went straight on my deny-list, but they got a few dollars out of me first.
Another common annoyance is having a request go to a provider that dribbles out ~1 tps (even for small models like DeepSeek V4 Flash). If you cancel the request, you still get charged for the prefill and the handful of generated tokens. If you don't cancel the request, you might be waiting 10 minutes for the turn to finish.
The overall experience is pretty good, and it's the best way to try new models, but they don't appear to do any real vetting or apply any quality standards to their providers, and occasionally it bites you.
This is very useful information and comes at a perfect time! I use Openrouter for my newly released running tracker (I use it for live coaching and post-run debriefs). I've benchmarked a bunch of models over time to evaluate their aptitude for this specific task, and have noticed that sometimes a model can underperform for seemingly no reason. I'll be sure to include model providers in my benchmarking suite going forward!
If you require consistency for research, I wouldn't recommend relying on hosted inference without validating the provider/inference stack very carefully.
For my contribution to OpenAI's gpt-oss red-teaming competition (published as arXiv:2510.01259), I initially used OpenRouter with DeepInfra and Together AI. The results were much too noisy to draw reliable conclusions from, and markedly different behavior through AWS Bedrock was the last straw. I ended up renting GPUs through vast.ai and running the model myself with vLLM.
Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).
Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:
1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.
2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.
These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.
Some quick thoughts on the article itself:
1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!
2. That is bad and we will fix it. Sorry.
3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".
4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.
5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto
6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...
7. Will investigate.
8. We attempt to heal these, but obviously missed some. Will fix.
9. We do not rate limit by IP. Would love some more information here, as that is very surprising.
10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.
> We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them.
I'm glad to hear that y'all are doing this, as I was unaware that this was something OpenRouter does. I was surprised and disappointed that there are so many problematic providers that it seems like community best practice [1] is to ban somewhere in the realm of 5-6 providers. Would it be possible to provide some way to express an even stronger preference for high quality providers? E.g. "only route to first party for this model" or, "cost, but don't route to providers that more than x% worse than the first party". I'm sure something like that can be done via the API but I haven't found a UI way to do it - and having it in the UI would go a long way towards feeling like OpenRouter is looking out for me/helping solve the problem as opposed to leaving it to me to have to figure out.
toven from openrouter, leading the team working with our providers - very interested in some of the things found in the report, I dug in to the image failures specifically, and in that case we have data showing deepinfra was correctly parsing images when the endpoint went live in july, but today fails those tests. we'll work on testing images and reasoning effort etc running constantly as chris mentions in point 3.
Hmm, let me check. That certainly seems wrong. Can you send me an email w/ your email so I can look into it? im cc at openrouter.ai. Or DM me on X? x.com/cclark
My ticket number is #107156 (opened 9 days ago). The point is that the Discord is full of people not getting responses, and I'm not even the one waiting the longest. The only help they get is from the AI assistant in the server saying don't bump the ticket or you'll be punished by getting moved to the back of the queue. So there's nothing anybody can do but wait who knows how long.
Yeah we are very under-staffed. We are hiring aggressively. I readily admit we have inadequate support staffing right now, but we are going as fast as we can to ramp up.
No stress! My first reaction to this was OMG THIS IS AN INCREDIBLE RESOURCE!! We are 100% grateful for this sort of feedback! Here is a direct quote of what I said at 8:17am this morning when someone sent me the article and I scanned it:
this is amazing!!
[8:19 AM]The first obvious win is routing around providers that arent handling image inputs correctly. That should be straightforward
[8:19 AM]The effort param stuff...I thought we had addressed that, but will dig in. This is incredible feedback
[8:20 AM]We should hire this guy.
I have attempted to report numerous errors in your Chat UI, but you seem to have decided that customer input is undesired, as you have an AI support system that ignores my problem and refuses to escalate to anyone or create a ticket.
To this day, your in-chat “Report an Issue” button still does not work consistently, and I am still billed for empty responses from many image providers.
Pulled some data - a surprisingly large number of requests to models that generate images, where the user seems to want an image, do NOT return an image. In our chatroom it's ~11%! However, in almost all of those cases, the model is instead deciding to return text, and you are being billed (correctly) for that text. We are not billing you for an image that wasn't returned.
Concretely, we pulled data on the last few days of image gen requests in our chatroom (50,893 requests). 5,846 got a text response (which is frustrating, I'm sure) and were billed for text appropriately. The model did not generate an image.
There were 16 requests where a customer was billed, but neither an image or text was returned. Those should not have been charged, and we'll see if we can either fix that issue or ensure that customers aren't charged.
One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close.
This might be the fault of the other providers, but it's probably just something slightly different that the first party does with the model inference program than anybody else, and that's not sure to weights it's due to vLLM twiddling (or whatever) and probably becuase the first part actually uses their own customized inference program rather than the standard methods that all the third party providers use. This isn't nefarious, it's just the challenge of these sorts of stochastic systems.
Having been in science for decades now, and seen benchmarking across many different fields, these results are completely expected for me. LLM serving is not mechanical, it's hard to get right and has lots of unknown footguns. Even something as extreme as scrambling a matrix will still likely get results that are nearly as good as normal, and if there's a bug deep in vLLM or the tensors metadata that results in that, then it's going to be pretty hard to find unless you're an active researcher with knowledge of the particular model you're running inference on. I kind of doubt that's happening here, but maybe!
In the scientific literature, when benchmarking methods, everybody's own method performs best in their own hands. Some attribute it to researchers gaming benchmarking for publication purposes, but I think it's just what we see here: the people who made a method are just the best at using it because they know all the quirks and use it best.
Programmers are not used to thinking with that nuance, and jump to conclusions about lying about quantizations, etc., but this is really just an unavoidable part of AI/ML methods: when things aren't perfect they're still pretty good and it's going to take the model creator to truly debug it. At least until the open weights ecosystem gets a lot better at ensuring reproducibility, and model cards are nowhere detailed enough for that to happen yet.
This matches what I found experimentally with gpt-oss-20b during OpenAI's red-teaming challenge. After moving from hosted inference to running the model myself on rented H100s via vast.ai, I saw the model refuse the same kinds of prompts at noticeably different rates depending on the inference stack — differences of roughly 5–10 percentage points with otherwise identical experimental parameters and seeds.
So I very much agree that this isn't necessarily about providers secretly changing the weights. For reproducible work, the serving stack - engine, version, hardware, configuration, and probably more - really belongs in the methodology alongside the model itself.
I wrote up the results here: “In AI Sweet Harmony” (arXiv:2510.01259).
Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way. (And if this was a false positive, I apologize!)
There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
Openrouter is useful for quickly testing various models with just one API. In development, it's useful. I would not run it in production tho' for all the caveat mentioned. Go to the first party provider directly, it's cheaper usually. And the cost to rewrite to use their API is usually noting (you can even have both and a feature flag), especially if you just vibe code it.
I've been using Merge AI Gateway and it's been useful so far. They tend to add new models quickly, and support has been responsive. https://gateway.merge.dev/
I was looking for an LLM gateway and saw that the most popular one had just had a massive supply chain attack, so I wrote my own. Took about 2 weeks and initially I wrote it as a provider for pi coding agent. I connect to moonshot, qwen, Gemini, zhipu, anthropic, deepseek and OpenAI. I use models.dev to load model and pricing info. Adding new providers is pretty easy because I have a standard internal format and each provider has an adapter that translates between my standard format and that required by the provider.
Fireworks has high variance amongst models and while some are served correctly; many of them are junk / broken and degraded and it seems like they don’t even know; because even running 1k MMLU Pro questions would flag it very quickly.
I've used Requesty (https://www.requesty.ai), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.
Just use a single vendor. Literally nothing wrong with that, and you avoid the complexity of both n-1 of the vendors (leaving you with the highest quality vendor) as well as the issues with the aggregating layer.
Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
They used a closed list of 3 vendors in prioritized order, and got 429ed out of two of them, while the third one stopped serving the mode.
This is less of a problem if you're running an agent locally and routing your problem to OpenRouter - you can pin to one or two models for consistency and just switch models when something goes bad. But the article is specifically about production traffic.
Never before have I heard this sentiment, NGL. Vendor-neutrality has been an OS(/FLOSS) darling for, well, the whole time.
RE:"single vendor", if this post is to believed then you might have picked one that has 100x the tool calling errors for the next SoTA model, if your single vendor serves the next SoTA in the first place. It also completely erases the notion of competition driving down prices -- that would only hurt you in the short term, but obviously would ruin the whole ecosystem long term.
How does it erase the notion of competition driving down prices? Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Don't load 6 months' worth of credit in an account, keep it tight. There's fairly little lock-in.
The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optimize my prompts every time I change models, as they all have their own unique... flavor.
As for tool calling errors, it seems like first party providers are among the best, I got the feeling that's what he was suggesting, though of course that's why you test. You can also go directly to Together.ai or whoever else you please.
Like others have said I think Openrouter seems neat for testing, but even just as a hobbyist I've been drawn to go direct to particular providers due to irritating little issues that I now see just weren't me.
>Endpoints are largely compatible, so the code change required to switch from one to another is trivial.
Wrapping a specific implementation in a neutral function is something you learn to do in year 1 of programming.
This specific issue and argument I see in lots of different aggregator dependencies, Terraform, LiteLLM/OpenRouter.
They promise to save some hypothetical work in the future if your boss asks to change vendors, and it turns out to be very trivial work that is just a regular part of our programming job, changing a couple of lines in order to change vendor.
It's worth noting that there exists a similar set of technologies with a reasonable tradeoff, using a framework that targets different user-platforms makes sense, write-once and deploy at iOS and Android is a reasonable tradeoff, but because you are deploying to those providers simultaneously and it's a user-choice so you don't get to pick one or the other (without losing clients), there's still arguments to chosing just one and losing market share, or doubling the workload and building native for both, but this is a true engineering choice. I feel like stuff like OpenRouter and TerraForm take elements of these frontend abstraction technologies and wastefully apply them to backend tech.
A particularly egregious case is when there's an aggregation layer for aggregation layers, say, a tool that generates TerraForm or Chef configs, or a tool that generates Docker and Podman containers, or a tool that generates LiteLLM/OpenRouter configs. Sounds dumb, but it happens when there's a market share for it. Can even get to 3 layers deep.
At the foundation might be an aversion to making an irreversible choice, which is an innate emergent psychological phenomenon, but is supported by the Bezos Amazon policy of reversible and irreversible doors. But again, even if you want to be light, using some of these aggregating tools isn't necessary, you can just build on top of a tech, and switch later. The only thing you get with an aggregating layer is that the API ends up being the common denominator so you lose out on the competitive advantages of each choice, or are forced to use even more complex API logic like LLM(commonParam1, commonParam2, vendorParams= {"vendor1"=:{"vendorParam1":"blabla"}} or worse, use hard-coded aggregator provided mappings between the aggregator API and the vendor API that may be incomplete and relies on updates from the aggregator dev.
I think OpenRouter’s value proposition is less about avoiding two hour developer tasks and more about having a single place to establish policy controls and dynamic selection based on current pricing and performance data. If you can’t actually do that - for the reasons described in the article you end up pinning - they can’t deliver. But it seems to work for some use cases.
Still the best way to ensure such policy control is to have 1 provider, tops 2 or 3.
Having a router thing that reroutes to 18 different vendors is of course no way to ensure any policy control, you can add all the internal buttons and dials on policy control and ISO and GDPR compliance, but all it will do is (incorrectly) check compliance box and increase compliance risk to the 18 different vendors.
In practice most openrouter users look for the cheapest vendor, and they tend to go for chinese vendors, who love to price dump and don't have the same views on contracts and IP as the west.
Who is this vendor that is consistently providing high quality inference for all families of open weight models at a competitive cost?
That's a serious questions that I really interested in the answer too. I have 25 providers included into my coding agent harness not because I care about vendor neutrality, but because I have to keep adding new ones as inference quality degrades at the providers I was using. Its quite tiresome.
LLM ops is not trivial. The systems for running inference are very complex, and running across multiple GPUs and nodes adds tons more complexity. And when LLMs are run incorrectly, they still work, just not at optimal performance. Even noticing that something is wrong is not trivial, and finding the problem is far far harder.
So I guess I shouldn't be surprised at all to see these benchmarks, but still I am!
There are such huge economies of scale with batched inference that it's clear this sort of service will continue, but it has a lot of growing up to do. Even AWS Bedrock has a Claude that feels different to me, but I haven't had a chance to do actual benchmarks that would show that.
I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times.
The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
exactly. you can specify a provider in your request to openrouter, or, better yet, use the :exacto endpoint to make open router automatically choose one that is good at tool calls.
Very informative article, but I'd say all of this is in favor of OpenRouter, not against it. It gives you full flexibility, not just of models, but of providers. That's the offer!
As long as you are constantly diligent in staying on top of what each provider is doing that may impact you.
It reads to me an argument for self hosting, maybe a less capable model to deal with smaller compute resources, but when starting to use that model and inference software to build a benchmark you can easily rerun when you change things to observe the impact of the change.
Maybe that’s a similar level of diligence but they feel different to me.
Yea I can believe all these. I've personally have been having issues on these points:
"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.
"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them
Open router have one more annoying disadvantage: credits you buy expire in around 1 year +/- ~30 days. They do sent warning email 30 days in advance, but then just make them disappear.
If you for some reason had unusee balance and forget about it; its just gone.
Oh, interesting, I feel like a year so ago I read something posted from openrouter team that they write that credits expire, but that they in actuality don’t expire them. But maybe I’m misremembering, or perhaps that’s changed for the worse in last few months :S
We started actually expiring the credits a ~month ago. If you make any kind of API request, it resets the clock. We try to make it a very generous policy, but we can't keep a monotonically increasing liability on the books. We end up owing (a lot) of taxes on it, but can't actually recognize revenue. We would much rather you spend the credits! Hence the reminder emails, and generous "clock reset" policy.
Hmm? You have the revenue already. I know it's awkward from an accounting point of view, but you already took my money. "Letting" me keep the balance in the account is not generous.
Edit: on re-reading this came out more combative than I intended, sorry. I think what you're doing is reasonable.
I kind a get their reasoning. They cant recognize balance topup as revenue because then they'll owe taxes on full amount.
At the same time majority of money we pay them usually just gonna be an expense paid to actual inference providers. Then they pay taxes on their fee aka actual profits.
Its understandable, but it dont make 1 year expiration any good.
Absolutely no where is that shown to customers, and OpenRouter is going to be in a world of shit when people learn this by HN post and the business asks themselves why its reputation keeps taking hits.
What a fucking ridiculous thing for me to learn by social news.
Former Openrouter user, unhappy to report that Openrouter will expire your tokens with very little warning with absolutely no business justification other than they can. If they were a bank and tokens money, their return is -100% interest after a year. Don't expect your tokens to be there for you after a year of no use....
<Rant>
I have been working on a product for months and for 1 of the specific models have become the number 3 user and very likely soon the number 2.
Its a google model. The edge cases are crazy to deal with and have taken a long time to find. I am also finding that since it has no fallbacks but no published rate limit I am single handedly taking the model down on what I thought were a reasonable amount of request. There is no other fallback that isn't google. I'm worried that with sustained usage from my users on launch in a few days. Clearly google can't be making that much money on it it im like 10% of the usage and the top 2-3 user of it spending several hundred a month in pre launch testing.
So why would they care to help a small time start up? I will have to jump to a cost effective different model and find the footguns all over again at somepoint.
Not sure where im going with this but just wanted to let others know
</rant>
Building a product that depends on a Google product seems like folly at this point. They are notoriously unreliable about keeping products alive. I hope you don’t get burned.
Me too, im a bit nervous because I also have evals proving that flash models are actually getting worse at specific task on newer versions and one of my prime use cases peak around flash 3 and tanks in 3.8. So it would be very very horrible for me if they kill old flash models on open router.
Yay google, love living in fear of a vendor that they will rug pull me at any point because they can.
For a Google model you might be better off directly using their API, which is unfortunately far more painful than using OpenRouter, but it might at least be more reliable for your product.
I think the rate limit issue happens because of OpenRouter sending so many requests to Google.
"OpenRouter runs per-provider benchmarks on the same model: GPQA Diamond and TAU-Bench Airline (a tool-calling task)." why I haven't never seen it? Click on the link brings nothing! Benchmark is only for the model. Per provider is a performance matrix (latency, throughput)
What did I miss?
--
Update: oh, that is the AutoExacto Benchmarks! Now I see it.
I've had the same experience in my personal use. Unreliable output, thinking token leaking into the conversation... Time to first token and tk/s also vary wildly per provider.
The worst case of hallucination I had, was DS v4 flash switching to Italian mid-conversation and impersonating a podcast host for no reason.
The big advantage of OpenRouter IMO is that you don't have to manage keys, URLs, and subscriptions with 10+ providers. You definitely should still pick providers and pin them, at least if you care about repeatable results.
We're very happy with OpenRouter in production, although we use it in a pretty limited fashion. We have only ~10 providers allowlisted (based on their location and ZDR policies) and we use only a few models, often frontier models. Has been very reliable for us in production so far. It's generally been great for us to be able to have a single place to manage spend, policies, view logs, etc. Also really nice in code to have a single-line change for switching models. OpenRouter's console is so much better than what you get from the actual frontier labs, and it's nice not to have to translate numbers and concepts across providers.
I noticed this with DeepSeek Flash and DigitalOcean. They are doing something seriously wrong when serving that model; we need something like an SLA, but for intelligence; it's almost fraudulent behavior.
This is surprising to me. Does anyone have a definitive answer that accounts for this difference between providers?
Do the benchmarks that Open Router runs not account for the stochastic nature of the models? Do the providers lie about quantization or context window sizing? Does Open Router not take into account variations for a given model?
I could understand latency/cost benchmarks varying but not the actual generated token responses of the models given the exact same model parameters.
The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).
If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.
text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case.
For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so you can technically save on KV cache. Video reasoning does require that context length, so it's a different set of deployment parameters that need to be enabled.
All of this to say that the providers that offer these models, are simply using vLLM / SGLang, and mostly cater to the text inference use-case (coding, etc). Vision always seems to be a bit of an afterthought.
Got it, I'll test more deployment params when I benchmark models for vision tasks.
Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?
I was just trying deepseek v4.1 flash on OpenRouter.
After running reasonably well for a while, I return to the window and see that my scrollback is nothing but this repeated over and over again:
```
OK.
Let me write.
Let me go.
OK.
Let me write the script.
Let me go.
```
I'm not sure to what extant this is a model problem, vs some providers being fairly broken.
If I chose a single provider, I could know how to blame and to avoid them. With OpenRouter, I don't know which provider I was on when this happened.
That first diagram is striking: that deepseek's own inference is at least 5% higher on tool calling (TAU Bench) than most other providers. I wonder if they make sure their responses are valid json at the token generation level using a grammar, similar to the feature in llama.cpp.
As a regular user of openrouter I didn't know this info was available. Will have to check it out. I've definitely noticed that a high level of deepseek flash responses were looping endlessly before the 0731 release.
Edit: looks like the diagrams data is from the "Auto
Exacto Benchmarks" section of the performance. Looks like they haven't run the benchmarks of deepseek flash 4.1 on the deepseek provider yet: https://openrouter.ai/deepseek/deepseek-v4.1-flash#performan...
Same experience. I migrated over and am happy but that intro was brutal.
My basics are I have a test suite that:
1. Finds newest models of my versions
2. Inferences every single model and a few providers for each with a short problem
3. Analyze latency and if a model failed the stupid simple questions drop it and the provider
4. Run larger context haystack kinds of problems.
It's cheap and fast less than 5$ so I can do this daily, hourly, whatever depending on how much I care. If it's mission critical I would say you need a two day study running once an hour to know the STD of model variance.
Then lock a top 3 contenders via latency dropping routing.
Is this easy? No.
Is it cheap? Also no.
Is it better than just using a trusted labs api? Also not really.
But it does give you exponentially more flexibility. Being able to run 10 unique models at the flick of a switch on a problem for pareto front analysis is amazing. And giving a dropdown for customers for multiple model options is powerful.
Yes, you hit it on the head. It's worth it, but you have to do your own evaluation and promotion.
Example I forgot to mention: `:nitro` ranking is not the fastest, I do a round robin sampling with representative payloads to find out the fastest providers and reorder my list on the fly.
I considered an automatic promotion path but decided against it I want to actually review the data myself first.
What's your model churn rate like? I was worried about customer experience by same day maybe same work getting a totally different model response (also caching is worse)
> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.
This is actually expected behavior. No content and no tool call is the same as content-only: the agent decided it's done. Anthropic has done this for a while.
> 6. 200 OK, no answer
Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.
A 200 tells you the request was served, not that there's an answer in it. No content and no tool call is a failure, throw and retry.
I ran into this so much I had to modify my harness to handle this.
This is an issue self hosting as well. There’s a lot of footguns that give you slightly bad results.
I wonder what tricks one could use to ensure the model is actually performing on par with the reference api, like matching seeds or running exact benchmarks.
Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
The way that OpenRouter frames it with having calls routed to whatever provider is handy encourages you to think that they’re all the same in basic functionality. Really sours me on OpenRouter.
exactly. all of these people complaining are using it wrong. you cant just expect a layer of abstraction to perfectly fix all the layers below it! plus, open router does a great job of handling a lot of the provider problems, but it literally cannot fix the output of the provider
Never had any issues using it for personal small scale development. I think the premise of OpenRouter as a magical way to fall back on error or select providers based on price/tok-per-sec/uptime isn’t delivered upon. But it’s an excellent way to simplify the experience of hopping between model providers with a single unified billing system. I mostly only use first party model hosts. Theoretically there are alternatives with higher throughput or lower prices, but it’s simplest to not worry about that optimization.
could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.
This isn't necessarily an OpenRouter issue, Google's Gemini will sometimes do it on their own API. Since they summarize reasoning this means you pay the whole cost and can't get anything out of it.
Until yesterday I used openrouter mostly as a portal to the big providers: google, anthropic, openai, without having to maintain keys and accounts and credits for all of them separately.
Yesterday I wanted to do a bit of benchmarking a prompt across multiple models. Small requests. Outside the big providers, the experience became awful. This explains that experience.
I've just been using openrouter to be able to quickly switch models and keep tight control over how much i spend and for that it's great. i also love their chat thing on the site where you can ask multiple models simultaneously
, very good for brainstorming
Under privacy, ignored, I put providers that always suck. Digital Ocean, OpenInference, DeepInfra, AtlasCloud, and Alibaba (that one is different, it doesn’t suck in the same way as the others with a dumb model, but instead it’s heavily censored and doesn’t like being used as RP model in Skyrim).
I'm happy with OpenRouter. They're good at routing requests & spend tracking/management. You should just expect to directly own your relationship with the downstream model provider.
Source: Enterprise customer doing $XXM annual run rate of inference spend on their platform
I still don't get the appeal of OpenRouter. Why not just generate API keys from the providers you want to use, which should not exceed 3-4, I assume, and integrate them into your apps to call them? Are people so lazy, or am I missing something?
If you're making a commercial SaaS, that's probably the way to go. For individual users like me, with a coding harness and some extra BYOK tools, OpenRouter is convenient and the few extra percent don't hurt much. I appreciate being able to try out any new model with just an ID swap, same with inference providers when they prices change. I know I wouldn't enjoy managing 5+ accounts, each with their own balance, in 3+ tools, so this is one thing I'm happy to outsource.
I have multiple commercial products using multiple LLM APIs. My concern is adding an extra 3rd party dependency and markup on top of the API usage. It just freaks me out to build entire apps on 3rd party single-point dependency.
I never tested this but always suspected it. When I learned that providers differentiate themselves on how they optimize and host the model (otherwise, why would you choose one over another?), I figured some were less "give me the best possible experience" than others.
I also believe some providers fallback to another model entirely. I was recently using Kimi K3 and saw that some requests had no reasoning trace whatsoever. Unsurprisingly, those requests were routed to the less reputable providers (Sail Research).
hi, I'm one of the founders of Sail. I'm very sorry that you had a bad experience with us! We are serious about serving models correctly, and always publish a link to the exact HF checkpoint we're using for each model in our docs. If you ever have an issue like this again, please send a note to support@sailresearch.com and we'll make it right with a detailed postmortem.
Hey, thanks for reaching out. Just wanted to share my experience. I really thought there was an issue in your API.
I had quite a few Kimi K3 requests served by Sail. Most of these were fine and had the reasoning traces (I like reading them), however I noticed that some requests were being generated unusually quick and had no reasoning trace, which led me to believe another fallback model was being used, since Kimi K3 always emits reasoning tokens.
I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
The article that I read addressed this point specifically. "Trusted" providers do not maintain consistent performance, and do not have the same performance across different models.
great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
Be sure to spend them! Another great feature of OpenRouter is that they will take your credits after a year. They just delete them. A ToS-legalized theft if you ask me. (Even if you are still actively using those credits.)
As of right now, they only expire credits if you don't use the account at all for a year. This is different from, say, OpenAI, where your credits expire unconditionally. That said, they do deserve the right to expire them regardless, so this could change in the future.
I haven't dove into OpenRouter yet but I see it everywhere now. I'll have to take this into consideration when I finally get around to trying it or something similar.
I use Fireworks for a production app. Much lower volumes though. My only gripe is their serverless offering for reranking has only 1 model which is Qwen 8b and it's quite expensive.
Wait WTF?! I thought performance was, well, performance, not efficacy! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more providers", it just shows a clearly broken modal. And the the next two graphs are even worse. Really all of these graphs are pretty and completely 100% useless.
I feel like it's absolutely insane that some providers serve the same model with much less efficacy. That doesn't make sense to me. What's going on?! I'm suddenly feeling intense shame for having routed all my non-subscription usage through them so far, and honestly some white hot anger that they would blatantly lie about something so important.
We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance
And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto
Oh wow someone from the team! I'm really heartened by this response -- as you can tell, I've been something of a fan, so I felt kinda betrayed by the idea that this was a risk. Thanks for clarifying, and for taking my frustrated hyperbole in stride.
Submitting an application to your product role now. "All of the models are at least okay" is heartening, but there's a whole bunch of fun places to take this.
Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs.
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
Just chiming in that I've never had this experience with them. I set, hit, and depend on those limits regularly. I would be wildly interested in reading what their support analysis had to say about this situation, if such a postmortem was done.
So far it was just "investigating". From the logs, my best guess is they sent all the requests at approximately the same time, and whatever limiting they do doesn't react fast enough.
I get it, the API key was my responsibility, but, setting the dollar limit is exactly the guard they suggest against that.
I think that's only on the enterprise plans? So no real protection for those of us using open router personally? Or did I miss other protections somewhere?
I believe OpenRouter addresses this with sticky routing and keeps a conversation on the same provider and falls back if that provider becomes unavailable via an explicit session_id to keep requests together. You’re right though that upon a failure cache is not portable and incurs additional costs.
provider fallbacks are also expensive since you lost the prompt cache. openrouter have a 'session' concept to help with this, but we ended up going direct to providers for higher rate limits and writing our own fallback logic.
I couldn't agree more. This articles describes how frustrated has been OpenRouter experience. At the end of the day, I ended up configuring to use the owner provider of each model I needed to use.
It's good to quantify the extent to which a lot of the stuff on there is just vibe-hosted. But I think it was always pretty apparent that this was the case. You've got numerous providers all running the gamut of:
I agree that OpenRouter kinda sucks, but it took about 15 seconds to see that a few sentences got reverse-compressed into what is now this article, and it's slop.
As expected vibecoding bros cannot even read the manual properly.
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.
The article mentioned they pinned the provider and model and the result was bad as well with their own shenanigans. That part was in the end of the article so maybe you missed it.
This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, it's bound to work exactly like this.
> I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
This is quite literally the point of OpenRouter. A unified interface, so you can easily switch providers without changing a ton of code which using providers directly would most likely mean, as there are slight differences between them. And the providers all run different weights, so of course quality/performance will differ among them.
I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you, but to actually find the good and usable stuff, you need to focus in on what providers/manufacturers you know are good, and stick with those. Still, the unified interface helps you to shop around and try different ones when you want to.
If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models.
Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.> There's not even a way to compare providers, AFAICT
That's not quite true. The only thing they don't show per-provider is benchmark data, cause I don't think they are doing continuous benchmarking of each model from each provider, as I assume they feel that's too expensive. You can see hugely detailed breakdowns for near-time metrics per provider for any model by visiting the page for that model on Openrouter. For example see the page for Qwen 3.8 27B: https://openrouter.ai/qwen/qwen3.8-27b
Some of the killer stats they show per provider:
- Pricing: Effective price accounting for cache hit rate, by provider
- Performance: Throughput in tok/s, latency, E2E latency, tool call error rate, structured output error rate, and more; all per provider.
- Uptime: You have to click on the provider to see their specific uptime, but doing so does show the last-7-days uptime, and you can click to see more.
We are doing continuous benchmarking of each endpoint, for each provider, and it is very expensive :)
The thing they don't show is the one we really need, especially because model providers can skimp on quality (run lower quantization, lower kv cache precision, etc) to improve their pricing and performance. I agree that it's probably too expensive to keep running the benchmark, but we need some way to hold the providers to a certain standard, otherwise every user has to discover the problems on their own.
No I know, I was in the GUI as I wrote that lol. As the other person said: if they vary this much in quality, not including that way above updtime and performance is absurd. What would you use a fast, always-up, broken endpoint for?
> I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate.
Why do you care about the public benchmarks at all?
The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it.
Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up.
I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices.
For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out.
If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.
If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity.
Despite your confidence, that is indeed the basis of this massive corporations entire business plan. If OpenRouter is only for bad accuracy, they should say as much and fade into deserved obscurity.I think you're focusing only on the general coding agent aspect of LLMs.
> If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity.
That is the way LLMs have to be used for highest reliability. While the term stochastic parrot has been co-opted by unreasonable LLM skeptics, that is indeed what LLMs are. You have to have grounded evals that check outcomes if you want to use them reliably - or a human in the loop works too.
The more general your family of tasks, the less likely you can make automated evals. So for "general" coding agents, you need a human in the loop that can verify and it's not that easy to write an eval.
But if you have specific tasks, then you can spend the time to make a eval, and then you can optimise the way you use the LLM and get extremely good success rates. It's not like it's black magic. Nor does it need large budgets.
> that is indeed the basis of this massive corporations entire business plan.
No. Individual developers using codex (for extremely underspecified general engineering) needs human in the loop, is not amenable to evals but is only a fraction of all LLM usecases.
[flagged]
> Hmm, yeah, good and condense version of what my previous comment said. I'm much impressed by your reading ability.
10month old account with 20k karma. Low value rubbish postings as a professional user. Sad.
[flagged]
You'd be surprised. Not enough teams still have their own evals. India[1] and the US[2] is my experience. To get many of them to understand the benefit of putting a couple of people to do data labelling and write a couple of verifiers for just a few days every few months was so difficult. Many folks have understood it all wrong and made allotments like "big model for this task" "small model for this task". Small was sometimes parameters, sometimes brand version number, or sometimes because it has "mini" in its name.
Proper LLM adoption beyond fucking around with claude code or github copilot is so low and is only going to go up as people figure it out. I also think the new cloud agents thing might accelerate adoption among these companies, since it's a bit more plug and play. But they are more likely to be stingy about it, so I'm not sure about the high margin expectations of certain model families.
[1] not witch, but BFSI. Surprisingly parts of witch companies have it down to science already, they embedded openai or anthropic or startups like devrev more than a year back.
[2] again not high fly SF companies, BFSI.
1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them.
2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime - but it doesn't work if it's at the cost of shitty performance or failing toolcalls.
There’s a surprising amount of variety in between implementations of “OpenAI” endpoints.
> 1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them.
OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable.
> 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
In theory, yes. In practice, no, there are differences. Ollama, llama.cpp, vLLM and SGLang all say "ChatCompletionRequest" compatible, but the devil is in the details, they don't have 100% the same request/response schema across all compatible models.
> 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
Yup, makes sense! If you're unable to find models when you use OpenRouter, it makes zero sense to continue to use OpenRouter.
> You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime
Huh? That's not how it works or does it make sense, nor have I've seen anyone use OpenRouter like that.
> Parent is complaining about the auto-provider chosing, not that all providers are unreliable.
Parent is claiming that the choosing is unreliable. If I rely on a provider to bring me tuna to some spec, but they get it from many different fishermen, it doesn't mean that the tuna doesn't have meet the spec. The complaint is that they're given a bunch of knobs that simply don't work with providers that they could be switched to. That's like saying that I want my tunas to be 20lbs. minimum, and I get switched to a provider that doesn't weigh their tuna at all.
The choosing is all OpenRouter provides. If it doesn't do that, then what is it good for? If I have to permanently pin the one provider who doesn't ignore what I've asked, why shouldn't I just deal with them directly?
edit: it's really supposed to reduce providers to a commodity market. If you're selling e.g. produce to a commodity market, you can't just ship whatever the hell you want. You ship something indistinguishable from others, or more likely the market itself allows you to grade what you're shipping so it's put into a bin with virtually identical stuff. The customer just buys Grade B Wheat.
[flagged]
> I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you
This helps forecast one possible future for OpenRouter: they begin to offer in-house provision, and people begin to migrate their uses off the "marketplace" providers and onto the "fulfilled-by-us" provision.
I mean that should be their job to verify the quality of the providers they have in their platform. Then what is the 5% for?
I treat OpenRouter as a central point to access specific providers. Pinning is a given for that use case.
I’ve tried using un-pinned models and the experience is exactly as you described: Some providers are so unreliable that the majority of requests fail. Some providers do weird things like abruptly end the response (which I get billed for and have to re-submit). Some providers are clearly running heavily quantized versions of the model because their eval performance is terrible. Some providers advertise features on OpenRouter but will reject those requests when submitted to their API.
So pinning is the way to go.
What providers do you find most reliable? And is that general across models?
OpenRouter has the advantage that I can maintain a single balance across model providers so I use it for quick iteration across and then write it into my own router (required anyway since I have many on-prem models running). The portable balance is actually very useful and my credit card is in one place.
If I could pay per request without maintaining a balance or credit card out of a single wallet (using crypto or something maybe) I would happily simply write the integration myself because OR’s caching is often not as good without some hoop jumping.
Yep, I wish more people realized that OpenRouter’s sole value proposition is that they are an LLM wholesaler.
> to the point I wonder if I should just use the providers directly.
How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful.
Personally I pin a single vetted provider in the interest of minimizing risk.
I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible.
For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.
Did you miss the part of this article where the benchmarks specifically call out Fireworks as one of the worst in their tests:
> Fireworks scored 46% on TAU, a 30 point gap
Another surprise was DigitalOcean being bottom of barrel too.
Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.
DigitalOcean (at least through OpenRouter) is pretty reliably bad in my experience. Fireworks can be great, but it depends on the model and on the day.
DigitalOcean is bad not just through OR... Direct accesss through DO gave me such a miniscule context and maximum tokens, it was pretty much unusable. So, accessing it through OpenRouter gave bigger context length, but model behaves like it was seriously damaged - tool calling was producing paths that had / replaced with some other symbols, files were not found etc., and this is just for things that errored out, I have no idea how bad reasoning was. I had to blacklist DO completely on OpenRouter.
It's not always deceptive. Bugs in inference can be very subtle (well, bugs everywhere can be subtle).
The providers should be benchmarking their offerings daily
I only use OpenRouter to benchmark models. For my production app, I have direct integrations with OpenAI, Anthropic, Google, xAI. Even just input caching is enough of a reason to do that, assuming you are trying to build the fastest and most cost-efficient product possible. I don't understand how people supposedly run production apps using OpenRouter unless those two variables somehow don't matter to them.
Not every usecase requires prompt caching, for example prompt enchantment jobs usually can't benefit from caching at all
This (by definition) the reality of every normalization/proxy layer, standard, etc. There’s always a cost for flexibility/freedom. What’s traded is when and by whom it’s paid.
You know you can define your own provider filters and orderings, right? The filters and such are not _that_ advanced, but it might do what you need if you haven't already tried that.
[dead]
Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cache. I see performance tank and then I look in the logs and a new provider has been rotated in and every call to them is uncached because they are clearly broken. This has happened a few times now and essentially destroys cost savings (these providers also often have terrible quality). Don't they monitor for simple things like this? Their own logs show how clear this pattern is for some providers. Simple cache % stats would allow them to block providers nearly instantly.
We have made MASSIVE improvements here, and network-wide caching rates have been improving relentlessly. We do publish the cache rates for each endpoint; see the "performance" area of our model pages. E.g. https://openrouter.ai/deepseek/deepseek-v4-flash-0731#perfor...
Open to feedback on how to make this better.
You know how there's a router mode to use the cheapest provider? That only takes into account uncached rates, last I checked. Make another one that takes into account effective rates (the ones that include cache).
Can I block endpoints whose cache rates are below a certain percentage?
I think the future -may- be a hosted routing client. Something similar to an auth client app. Why? A few reasons:
- Customers being able to decide their own routing with true logic is a huge feature. Open router provides the seamless switching/api, route switching decisions are available in a client.
- Similarly, providing hooks at this level gives a chance for stats/other things that are hard to plug into prod code elsewhere
- a true middle man hosting for other things like MCP may also turn into a real win once it is implemented.
Just a random thought though. My point about quality/cost being clobbered by bad providers remains. The fact that cache and quality is badly handled makes me doubt that training data choices are being respected. You need a more public trust/certification process for providers with real teeth when they cheat. I'm going to wait a bit to see how things evolve and check back later.
You need to validate what providers are actually serving. Add benchmarks, properly showcase what quantization they are serving on the model and KV cache, etc.
Until that happens, your service is doomed to be shitty.
Can I block providers (for model, not in general) that have sightly cheaper input/output tokens but have more that 10x the average cache cost?
Can I block providers (for a model, not in general) that set up cache write cost when the mode is free cache writes?
> The same model will benchmark very differently
Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
> Providers probably serve quantized versions without disclosing it
It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers. In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.
100% agree. The key issue is that users think they are getting results from a specific model configuration and they are clearly not, which is fraudulent.
If OpenRouter wants to succeed as a business, they need to be auditing the providers they connect to (i.e. benchmarking) and removing fraudulent ones from their service.
They already succeed selling for 7 billion. Auditing is up to you and can be a new business for those looking for new opportunties.
> by regularly benchmarking providers and giving you the control to avoid bad providers.
Or by fining them, or getting rid of them altogether.
It's more than just quantization. The middleware the provider is running matters a lot even to the point of exactly which version they are running due to defects being introduced / resolved.
In my coding agent harness I've included 25 open weight providers mainly because I keep having to find new ones when what was previously a great combination of model and provider becomes pretty bad. vllm has defect that causes reasoning to get dropped much of the time for the GLM family of models. sglang has a defect that causes the elements of array args to get dropped for the deepseek family of models. Some providers need some very specific additional config passed through for reasoning to make it back to the model.
I've not tried OpenRouter as adding yet another layer will just make it that much more difficult to get a model and provider combination working well.
I suspect people's bad experiences with open weight models have a lot to do with these headaches. Finding a good model and provider combination is pretty tedious and so far its been a never ending process. I'd really like to host my own models but it isn't economically feasible for one person for the open weight models that work well (i.e. the 300B+ ones).
I mean that is addressed as well later, even when they do declare the quantization, doesn't mean you'll get better performance than the lower quantized one. I'm guessing they're doing something similar to what stadia was doing, saying you are playing games in 4k, because you're getting a 4k stream, but the game itself is running in 1080p
I think it's not even that sophisticated here, the providers are just lying.
The best part about OpenRouter is 200 OK is probably hardcoded into their responses.
I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.
I was just thinking that this kind of behavior is a glimpse of the future!
Yeah, this is pretty accurate. Some providers are basically scams too. I encountered one provider for GLM-5.2 which ran at 200 tps (absurdly high), and was so broken that it would issue 20 full reads of the same file in one turn and quickly jump up to 300~400k context and charge me for the prefill. Went straight on my deny-list, but they got a few dollars out of me first.
Another common annoyance is having a request go to a provider that dribbles out ~1 tps (even for small models like DeepSeek V4 Flash). If you cancel the request, you still get charged for the prefill and the handful of generated tokens. If you don't cancel the request, you might be waiting 10 minutes for the turn to finish.
The overall experience is pretty good, and it's the best way to try new models, but they don't appear to do any real vetting or apply any quality standards to their providers, and occasionally it bites you.
This is very useful information and comes at a perfect time! I use Openrouter for my newly released running tracker (I use it for live coaching and post-run debriefs). I've benchmarked a bunch of models over time to evaluate their aptitude for this specific task, and have noticed that sometimes a model can underperform for seemingly no reason. I'll be sure to include model providers in my benchmarking suite going forward!
Whats the tracker called? I'd like to try it.
It's in the parents profile
Android app called "AudioRun"
https://play.google.com/store/apps/details?id=com.audiorun.a...
https://apkpure.com/audiorun/com.audiorun.app
Pretty damning. We're using OpenRouter for some research tasks and it's making me question everything. I suspect lots of the model providers are running into issues like these: https://forum.level1techs.com/t/why-your-local-llm-feels-dum... (HN discussion: https://news.ycombinator.com/item?id=49402232)
If you require consistency for research, I wouldn't recommend relying on hosted inference without validating the provider/inference stack very carefully.
For my contribution to OpenAI's gpt-oss red-teaming competition (published as arXiv:2510.01259), I initially used OpenRouter with DeepInfra and Together AI. The results were much too noisy to draw reliable conclusions from, and markedly different behavior through AWS Bedrock was the last straw. I ended up renting GPUs through vast.ai and running the model myself with vLLM.
Co-founder and COO of OpenRouter here.
Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).
Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:
1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.
2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.
These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.
Some quick thoughts on the article itself:
1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!
2. That is bad and we will fix it. Sorry.
3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".
4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.
5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto
6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...
7. Will investigate.
8. We attempt to heal these, but obviously missed some. Will fix.
9. We do not rate limit by IP. Would love some more information here, as that is very surprising.
10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.
> We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them.
I'm glad to hear that y'all are doing this, as I was unaware that this was something OpenRouter does. I was surprised and disappointed that there are so many problematic providers that it seems like community best practice [1] is to ban somewhere in the realm of 5-6 providers. Would it be possible to provide some way to express an even stronger preference for high quality providers? E.g. "only route to first party for this model" or, "cost, but don't route to providers that more than x% worse than the first party". I'm sure something like that can be done via the API but I haven't found a UI way to do it - and having it in the UI would go a long way towards feeling like OpenRouter is looking out for me/helping solve the problem as opposed to leaving it to me to have to figure out.
[1] https://www.reddit.com/r/LocalLLaMA/comments/1mk4kt0/be_care...
toven from openrouter, leading the team working with our providers - very interested in some of the things found in the report, I dug in to the image failures specifically, and in that case we have data showing deepinfra was correctly parsing images when the endpoint went live in july, but today fails those tests. we'll work on testing images and reasoning effort etc running constantly as chris mentions in point 3.
(screenshot showing our internal testing of deepinfra image inputs: https://raw.githubusercontent.com/ping-Toven/images/main/ima... )
What are your plans to improve a substantial lack of customer support? https://news.ycombinator.com/item?id=49577159
Hmm, let me check. That certainly seems wrong. Can you send me an email w/ your email so I can look into it? im cc at openrouter.ai. Or DM me on X? x.com/cclark
My ticket number is #107156 (opened 9 days ago). The point is that the Discord is full of people not getting responses, and I'm not even the one waiting the longest. The only help they get is from the AI assistant in the server saying don't bump the ticket or you'll be punished by getting moved to the back of the queue. So there's nothing anybody can do but wait who knows how long.
Yeah we are very under-staffed. We are hiring aggressively. I readily admit we have inadequate support staffing right now, but we are going as fast as we can to ramp up.
[flagged]
Thanks for the response Chris, I couldn't make my business work without OpenRouter in the first place, so kudos
This was meant as more of a technical reference, sorry you had to wake up to a PR drill lol
No stress! My first reaction to this was OMG THIS IS AN INCREDIBLE RESOURCE!! We are 100% grateful for this sort of feedback! Here is a direct quote of what I said at 8:17am this morning when someone sent me the article and I scanned it:
Our goal is to get better, fast!I switched to Opencode because they have guaranteed performance. Maybe you could have an openrouter certified badge or label for this sort of thing.
I have attempted to report numerous errors in your Chat UI, but you seem to have decided that customer input is undesired, as you have an AI support system that ignores my problem and refuses to escalate to anyone or create a ticket.
To this day, your in-chat “Report an Issue” button still does not work consistently, and I am still billed for empty responses from many image providers.
Pulled some data - a surprisingly large number of requests to models that generate images, where the user seems to want an image, do NOT return an image. In our chatroom it's ~11%! However, in almost all of those cases, the model is instead deciding to return text, and you are being billed (correctly) for that text. We are not billing you for an image that wasn't returned.
Concretely, we pulled data on the last few days of image gen requests in our chatroom (50,893 requests). 5,846 got a text response (which is frustrating, I'm sure) and were billed for text appropriately. The model did not generate an image.
There were 16 requests where a customer was billed, but neither an image or text was returned. Those should not have been charged, and we'll see if we can either fix that issue or ensure that customers aren't charged.
Will look into this.
What a refreshing response. I'm a big fan and recommend your service frequently to people locked in to providers. Godspeed!
Thanks for the insight here.
One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close.
This might be the fault of the other providers, but it's probably just something slightly different that the first party does with the model inference program than anybody else, and that's not sure to weights it's due to vLLM twiddling (or whatever) and probably becuase the first part actually uses their own customized inference program rather than the standard methods that all the third party providers use. This isn't nefarious, it's just the challenge of these sorts of stochastic systems.
Having been in science for decades now, and seen benchmarking across many different fields, these results are completely expected for me. LLM serving is not mechanical, it's hard to get right and has lots of unknown footguns. Even something as extreme as scrambling a matrix will still likely get results that are nearly as good as normal, and if there's a bug deep in vLLM or the tensors metadata that results in that, then it's going to be pretty hard to find unless you're an active researcher with knowledge of the particular model you're running inference on. I kind of doubt that's happening here, but maybe!
In the scientific literature, when benchmarking methods, everybody's own method performs best in their own hands. Some attribute it to researchers gaming benchmarking for publication purposes, but I think it's just what we see here: the people who made a method are just the best at using it because they know all the quirks and use it best.
Programmers are not used to thinking with that nuance, and jump to conclusions about lying about quantizations, etc., but this is really just an unavoidable part of AI/ML methods: when things aren't perfect they're still pretty good and it's going to take the model creator to truly debug it. At least until the open weights ecosystem gets a lot better at ensuring reproducibility, and model cards are nowhere detailed enough for that to happen yet.
This matches what I found experimentally with gpt-oss-20b during OpenAI's red-teaming challenge. After moving from hosted inference to running the model myself on rented H100s via vast.ai, I saw the model refuse the same kinds of prompts at noticeably different rates depending on the inference stack — differences of roughly 5–10 percentage points with otherwise identical experimental parameters and seeds.
So I very much agree that this isn't necessarily about providers secretly changing the weights. For reproducible work, the serving stack - engine, version, hardware, configuration, and probably more - really belongs in the methodology alongside the model itself.
I wrote up the results here: “In AI Sweet Harmony” (arXiv:2510.01259).
Can you please not post AI-generated or AI-edited comments to HN? It's not allowed here - see https://news.ycombinator.com/newsguidelines.html#generated and https://news.ycombinator.com/item?id=47340079.
Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way. (And if this was a false positive, I apologize!)
Great concrete experience!
I'd love to see some published token log_probs for given prompts and seeds that accompany a model card. Not sure if that's enough, thigh.
But then it's been so long since I dealt with the api directly that I don't even know if modern models show the top tokens and probabilities any more.
[dead]
[dead]
That looks absolutely horrifying. What are the alternatives ??
There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
Openrouter is useful for quickly testing various models with just one API. In development, it's useful. I would not run it in production tho' for all the caveat mentioned. Go to the first party provider directly, it's cheaper usually. And the cost to rewrite to use their API is usually noting (you can even have both and a feature flag), especially if you just vibe code it.
I've been using Merge AI Gateway and it's been useful so far. They tend to add new models quickly, and support has been responsive. https://gateway.merge.dev/
I was looking for an LLM gateway and saw that the most popular one had just had a massive supply chain attack, so I wrote my own. Took about 2 weeks and initially I wrote it as a provider for pi coding agent. I connect to moonshot, qwen, Gemini, zhipu, anthropic, deepseek and OpenAI. I use models.dev to load model and pricing info. Adding new providers is pretty easy because I have a standard internal format and each provider has an adapter that translates between my standard format and that required by the provider.
Fireworks hosts the available models themselves which probably solves the problem consistency problem that OP had to deal with.
It's been a few months since I looked around at this topic, but Fireworks and Openrouter were the two options I (briefly) tried.
Fireworks has high variance amongst models and while some are served correctly; many of them are junk / broken and degraded and it seems like they don’t even know; because even running 1k MMLU Pro questions would flag it very quickly.
Good info, thanks!
I've used Requesty (https://www.requesty.ai), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.
Openrouter lets you pin or blacklist providers or specify provider per-request as well.
Vercel AI Gateway is one I've used: https://vercel.com/ai-gateway/models
Vercel AI Gateway also route to other providers, so the same issues can occur there as well.
If you only care about open source models, cline and opencode provide usage based access and subscriptions for general API access
Just use a single vendor. Literally nothing wrong with that, and you avoid the complexity of both n-1 of the vendors (leaving you with the highest quality vendor) as well as the issues with the aggregating layer.
Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
OP already answered that one:
They used a closed list of 3 vendors in prioritized order, and got 429ed out of two of them, while the third one stopped serving the mode.
This is less of a problem if you're running an agent locally and routing your problem to OpenRouter - you can pin to one or two models for consistency and just switch models when something goes bad. But the article is specifically about production traffic.
> vendor-neutral slop
Never before have I heard this sentiment, NGL. Vendor-neutrality has been an OS(/FLOSS) darling for, well, the whole time.
RE:"single vendor", if this post is to believed then you might have picked one that has 100x the tool calling errors for the next SoTA model, if your single vendor serves the next SoTA in the first place. It also completely erases the notion of competition driving down prices -- that would only hurt you in the short term, but obviously would ruin the whole ecosystem long term.
I feel like I must be missing something?
How does it erase the notion of competition driving down prices? Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Don't load 6 months' worth of credit in an account, keep it tight. There's fairly little lock-in.
The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optimize my prompts every time I change models, as they all have their own unique... flavor.
As for tool calling errors, it seems like first party providers are among the best, I got the feeling that's what he was suggesting, though of course that's why you test. You can also go directly to Together.ai or whoever else you please.
Like others have said I think Openrouter seems neat for testing, but even just as a hobbyist I've been drawn to go direct to particular providers due to irritating little issues that I now see just weren't me.
>Endpoints are largely compatible, so the code change required to switch from one to another is trivial.
Wrapping a specific implementation in a neutral function is something you learn to do in year 1 of programming.
This specific issue and argument I see in lots of different aggregator dependencies, Terraform, LiteLLM/OpenRouter.
They promise to save some hypothetical work in the future if your boss asks to change vendors, and it turns out to be very trivial work that is just a regular part of our programming job, changing a couple of lines in order to change vendor.
It's worth noting that there exists a similar set of technologies with a reasonable tradeoff, using a framework that targets different user-platforms makes sense, write-once and deploy at iOS and Android is a reasonable tradeoff, but because you are deploying to those providers simultaneously and it's a user-choice so you don't get to pick one or the other (without losing clients), there's still arguments to chosing just one and losing market share, or doubling the workload and building native for both, but this is a true engineering choice. I feel like stuff like OpenRouter and TerraForm take elements of these frontend abstraction technologies and wastefully apply them to backend tech.
A particularly egregious case is when there's an aggregation layer for aggregation layers, say, a tool that generates TerraForm or Chef configs, or a tool that generates Docker and Podman containers, or a tool that generates LiteLLM/OpenRouter configs. Sounds dumb, but it happens when there's a market share for it. Can even get to 3 layers deep.
At the foundation might be an aversion to making an irreversible choice, which is an innate emergent psychological phenomenon, but is supported by the Bezos Amazon policy of reversible and irreversible doors. But again, even if you want to be light, using some of these aggregating tools isn't necessary, you can just build on top of a tech, and switch later. The only thing you get with an aggregating layer is that the API ends up being the common denominator so you lose out on the competitive advantages of each choice, or are forced to use even more complex API logic like LLM(commonParam1, commonParam2, vendorParams= {"vendor1"=:{"vendorParam1":"blabla"}} or worse, use hard-coded aggregator provided mappings between the aggregator API and the vendor API that may be incomplete and relies on updates from the aggregator dev.
Less is more.
I think OpenRouter’s value proposition is less about avoiding two hour developer tasks and more about having a single place to establish policy controls and dynamic selection based on current pricing and performance data. If you can’t actually do that - for the reasons described in the article you end up pinning - they can’t deliver. But it seems to work for some use cases.
I see that makes sense.
Still the best way to ensure such policy control is to have 1 provider, tops 2 or 3.
Having a router thing that reroutes to 18 different vendors is of course no way to ensure any policy control, you can add all the internal buttons and dials on policy control and ISO and GDPR compliance, but all it will do is (incorrectly) check compliance box and increase compliance risk to the 18 different vendors.
In practice most openrouter users look for the cheapest vendor, and they tend to go for chinese vendors, who love to price dump and don't have the same views on contracts and IP as the west.
Who is this vendor that is consistently providing high quality inference for all families of open weight models at a competitive cost?
That's a serious questions that I really interested in the answer too. I have 25 providers included into my coding agent harness not because I care about vendor neutrality, but because I have to keep adding new ones as inference quality degrades at the providers I was using. Its quite tiresome.
LLM ops is not trivial. The systems for running inference are very complex, and running across multiple GPUs and nodes adds tons more complexity. And when LLMs are run incorrectly, they still work, just not at optimal performance. Even noticing that something is wrong is not trivial, and finding the problem is far far harder.
So I guess I shouldn't be surprised at all to see these benchmarks, but still I am!
There are such huge economies of scale with batched inference that it's clear this sort of service will continue, but it has a lot of growing up to do. Even AWS Bedrock has a Claude that feels different to me, but I haven't had a chance to do actual benchmarks that would show that.
Bedrock Claude is absolutely not identical to 1P Claude.
They’re close enough to not matter though.
I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times.
The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
Using the same provider?
exactly. you can specify a provider in your request to openrouter, or, better yet, use the :exacto endpoint to make open router automatically choose one that is good at tool calls.
Very informative article, but I'd say all of this is in favor of OpenRouter, not against it. It gives you full flexibility, not just of models, but of providers. That's the offer!
As long as you are constantly diligent in staying on top of what each provider is doing that may impact you.
It reads to me an argument for self hosting, maybe a less capable model to deal with smaller compute resources, but when starting to use that model and inference software to build a benchmark you can easily rerun when you change things to observe the impact of the change.
Maybe that’s a similar level of diligence but they feel different to me.
Yea I can believe all these. I've personally have been having issues on these points:
"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.
"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them
Open router have one more annoying disadvantage: credits you buy expire in around 1 year +/- ~30 days. They do sent warning email 30 days in advance, but then just make them disappear.
If you for some reason had unusee balance and forget about it; its just gone.
Oh, interesting, I feel like a year so ago I read something posted from openrouter team that they write that credits expire, but that they in actuality don’t expire them. But maybe I’m misremembering, or perhaps that’s changed for the worse in last few months :S
We started actually expiring the credits a ~month ago. If you make any kind of API request, it resets the clock. We try to make it a very generous policy, but we can't keep a monotonically increasing liability on the books. We end up owing (a lot) of taxes on it, but can't actually recognize revenue. We would much rather you spend the credits! Hence the reminder emails, and generous "clock reset" policy.
> but can't actually recognize revenue
Hmm? You have the revenue already. I know it's awkward from an accounting point of view, but you already took my money. "Letting" me keep the balance in the account is not generous.
Edit: on re-reading this came out more combative than I intended, sorry. I think what you're doing is reasonable.
I kind a get their reasoning. They cant recognize balance topup as revenue because then they'll owe taxes on full amount.
At the same time majority of money we pay them usually just gonna be an expense paid to actual inference providers. Then they pay taxes on their fee aka actual profits.
Its understandable, but it dont make 1 year expiration any good.
Is this reset clock policy is actually documented somewhere? Because right now your website it's just say 1 year since credit purchase.
Absolutely no where is that shown to customers, and OpenRouter is going to be in a world of shit when people learn this by HN post and the business asks themselves why its reputation keeps taking hits.
What a fucking ridiculous thing for me to learn by social news.
Former Openrouter user, unhappy to report that Openrouter will expire your tokens with very little warning with absolutely no business justification other than they can. If they were a bank and tokens money, their return is -100% interest after a year. Don't expect your tokens to be there for you after a year of no use....
<Rant> I have been working on a product for months and for 1 of the specific models have become the number 3 user and very likely soon the number 2.
Its a google model. The edge cases are crazy to deal with and have taken a long time to find. I am also finding that since it has no fallbacks but no published rate limit I am single handedly taking the model down on what I thought were a reasonable amount of request. There is no other fallback that isn't google. I'm worried that with sustained usage from my users on launch in a few days. Clearly google can't be making that much money on it it im like 10% of the usage and the top 2-3 user of it spending several hundred a month in pre launch testing.
So why would they care to help a small time start up? I will have to jump to a cost effective different model and find the footguns all over again at somepoint.
Not sure where im going with this but just wanted to let others know </rant>
Building a product that depends on a Google product seems like folly at this point. They are notoriously unreliable about keeping products alive. I hope you don’t get burned.
Me too, im a bit nervous because I also have evals proving that flash models are actually getting worse at specific task on newer versions and one of my prime use cases peak around flash 3 and tanks in 3.8. So it would be very very horrible for me if they kill old flash models on open router.
Yay google, love living in fear of a vendor that they will rug pull me at any point because they can.
For a Google model you might be better off directly using their API, which is unfortunately far more painful than using OpenRouter, but it might at least be more reliable for your product.
I think the rate limit issue happens because of OpenRouter sending so many requests to Google.
You are hitting the google model through openrouter or directly?
Openrouter its just so much easier to use openrouter
"OpenRouter runs per-provider benchmarks on the same model: GPQA Diamond and TAU-Bench Airline (a tool-calling task)." why I haven't never seen it? Click on the link brings nothing! Benchmark is only for the model. Per provider is a performance matrix (latency, throughput) What did I miss?
--
Update: oh, that is the AutoExacto Benchmarks! Now I see it.
I've had the same experience in my personal use. Unreliable output, thinking token leaking into the conversation... Time to first token and tk/s also vary wildly per provider.
The worst case of hallucination I had, was DS v4 flash switching to Italian mid-conversation and impersonating a podcast host for no reason.
The big advantage of OpenRouter IMO is that you don't have to manage keys, URLs, and subscriptions with 10+ providers. You definitely should still pick providers and pin them, at least if you care about repeatable results.
We're very happy with OpenRouter in production, although we use it in a pretty limited fashion. We have only ~10 providers allowlisted (based on their location and ZDR policies) and we use only a few models, often frontier models. Has been very reliable for us in production so far. It's generally been great for us to be able to have a single place to manage spend, policies, view logs, etc. Also really nice in code to have a single-line change for switching models. OpenRouter's console is so much better than what you get from the actual frontier labs, and it's nice not to have to translate numbers and concepts across providers.
I noticed this with DeepSeek Flash and DigitalOcean. They are doing something seriously wrong when serving that model; we need something like an SLA, but for intelligence; it's almost fraudulent behavior.
> The same model will benchmark very differently
This is surprising to me. Does anyone have a definitive answer that accounts for this difference between providers?
Do the benchmarks that Open Router runs not account for the stochastic nature of the models? Do the providers lie about quantization or context window sizing? Does Open Router not take into account variations for a given model?
I could understand latency/cost benchmarks varying but not the actual generated token responses of the models given the exact same model parameters.
As said in the article, all of the providers run different runtimes with proprietary, sometimes untested and weird optimisations
There are multiple types of quantisation: weights and KV Cache. Quantizing kv cache can drastically hurt performance
The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).
If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.
Do you have any advice on this front? I use vllm for a project but only for text tasks at the moment.
text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case.
For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so you can technically save on KV cache. Video reasoning does require that context length, so it's a different set of deployment parameters that need to be enabled.
All of this to say that the providers that offer these models, are simply using vLLM / SGLang, and mostly cater to the text inference use-case (coding, etc). Vision always seems to be a bit of an afterthought.
Got it, I'll test more deployment params when I benchmark models for vision tasks.
Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?
---
Gemma 4 31B NVFP4, vLLM 0.29, single B200 (modal), FlashInfer, fp8 KV, prefix caching, 32k ctx
Workload: ~6k-token shared prefix (~98% cache hit) + short input, ~250 tokens out, ~250 seqs running. Get ~5-6k output tok/s, flat from 128 to 384 concurrency.
Spec decode (DFlash, n-gram) didn't give a good boost which is annoying because I feel like this task should be easy for a draft model to predict.
Individual request latency doesn't matter I just need as much throughput as possible. Its only active for a few hours when I need it.
I was just trying deepseek v4.1 flash on OpenRouter. After running reasonably well for a while, I return to the window and see that my scrollback is nothing but this repeated over and over again:
``` OK.
Let me write.
Let me go.
OK.
Let me write the script.
Let me go. ```
I'm not sure to what extant this is a model problem, vs some providers being fairly broken. If I chose a single provider, I could know how to blame and to avoid them. With OpenRouter, I don't know which provider I was on when this happened.
Looks like someone could make some dollars re-creating openrouter from scratch and proving reliability across evals/models/providers/configs
That first diagram is striking: that deepseek's own inference is at least 5% higher on tool calling (TAU Bench) than most other providers. I wonder if they make sure their responses are valid json at the token generation level using a grammar, similar to the feature in llama.cpp.
As a regular user of openrouter I didn't know this info was available. Will have to check it out. I've definitely noticed that a high level of deepseek flash responses were looping endlessly before the 0731 release.
Edit: looks like the diagrams data is from the "Auto Exacto Benchmarks" section of the performance. Looks like they haven't run the benchmarks of deepseek flash 4.1 on the deepseek provider yet: https://openrouter.ai/deepseek/deepseek-v4.1-flash#performan...
Same experience. I migrated over and am happy but that intro was brutal.
My basics are I have a test suite that: 1. Finds newest models of my versions 2. Inferences every single model and a few providers for each with a short problem 3. Analyze latency and if a model failed the stupid simple questions drop it and the provider 4. Run larger context haystack kinds of problems.
It's cheap and fast less than 5$ so I can do this daily, hourly, whatever depending on how much I care. If it's mission critical I would say you need a two day study running once an hour to know the STD of model variance.
Then lock a top 3 contenders via latency dropping routing.
Is this easy? No. Is it cheap? Also no. Is it better than just using a trusted labs api? Also not really.
But it does give you exponentially more flexibility. Being able to run 10 unique models at the flick of a switch on a problem for pareto front analysis is amazing. And giving a dropdown for customers for multiple model options is powerful.
Yes, you hit it on the head. It's worth it, but you have to do your own evaluation and promotion.
Example I forgot to mention: `:nitro` ranking is not the fastest, I do a round robin sampling with representative payloads to find out the fastest providers and reorder my list on the fly.
Your doing that sampling/promotion automatically?
I considered an automatic promotion path but decided against it I want to actually review the data myself first.
What's your model churn rate like? I was worried about customer experience by same day maybe same work getting a totally different model response (also caching is worse)
> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.
This is actually expected behavior. No content and no tool call is the same as content-only: the agent decided it's done. Anthropic has done this for a while.
> 6. 200 OK, no answer Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user. A 200 tells you the request was served, not that there's an answer in it. No content and no tool call is a failure, throw and retry.
I ran into this so much I had to modify my harness to handle this.
This is why it's not recommended to use providers below the Silver level in Semianalysis' rating index.
https://newsletter.semianalysis.com/p/clustermax-20-the-indu...
This is an issue self hosting as well. There’s a lot of footguns that give you slightly bad results.
I wonder what tricks one could use to ensure the model is actually performing on par with the reference api, like matching seeds or running exact benchmarks.
Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
The way that OpenRouter frames it with having calls routed to whatever provider is handy encourages you to think that they’re all the same in basic functionality. Really sours me on OpenRouter.
I wouldn’t blame openrouter, blame the providers that have lower quality of service than they should
exactly. all of these people complaining are using it wrong. you cant just expect a layer of abstraction to perfectly fix all the layers below it! plus, open router does a great job of handling a lot of the provider problems, but it literally cannot fix the output of the provider
[dead]
Never had any issues using it for personal small scale development. I think the premise of OpenRouter as a magical way to fall back on error or select providers based on price/tok-per-sec/uptime isn’t delivered upon. But it’s an excellent way to simplify the experience of hopping between model providers with a single unified billing system. I mostly only use first party model hosts. Theoretically there are alternatives with higher throughput or lower prices, but it’s simplest to not worry about that optimization.
could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.
This isn't necessarily an OpenRouter issue, Google's Gemini will sometimes do it on their own API. Since they summarize reasoning this means you pay the whole cost and can't get anything out of it.
Until yesterday I used openrouter mostly as a portal to the big providers: google, anthropic, openai, without having to maintain keys and accounts and credits for all of them separately.
Yesterday I wanted to do a bit of benchmarking a prompt across multiple models. Small requests. Outside the big providers, the experience became awful. This explains that experience.
I've just been using openrouter to be able to quickly switch models and keep tight control over how much i spend and for that it's great. i also love their chat thing on the site where you can ask multiple models simultaneously , very good for brainstorming
> The tool call is in the text
Noob question: do good harnesses automatically optimize for bad tool calling behavior automatically?
Under privacy, ignored, I put providers that always suck. Digital Ocean, OpenInference, DeepInfra, AtlasCloud, and Alibaba (that one is different, it doesn’t suck in the same way as the others with a dumb model, but instead it’s heavily censored and doesn’t like being used as RP model in Skyrim).
I'm happy with OpenRouter. They're good at routing requests & spend tracking/management. You should just expect to directly own your relationship with the downstream model provider.
Source: Enterprise customer doing $XXM annual run rate of inference spend on their platform
I still don't get the appeal of OpenRouter. Why not just generate API keys from the providers you want to use, which should not exceed 3-4, I assume, and integrate them into your apps to call them? Are people so lazy, or am I missing something?
If you're making a commercial SaaS, that's probably the way to go. For individual users like me, with a coding harness and some extra BYOK tools, OpenRouter is convenient and the few extra percent don't hurt much. I appreciate being able to try out any new model with just an ID swap, same with inference providers when they prices change. I know I wouldn't enjoy managing 5+ accounts, each with their own balance, in 3+ tools, so this is one thing I'm happy to outsource.
I have multiple commercial products using multiple LLM APIs. My concern is adding an extra 3rd party dependency and markup on top of the API usage. It just freaks me out to build entire apps on 3rd party single-point dependency.
I never tested this but always suspected it. When I learned that providers differentiate themselves on how they optimize and host the model (otherwise, why would you choose one over another?), I figured some were less "give me the best possible experience" than others.
I also believe some providers fallback to another model entirely. I was recently using Kimi K3 and saw that some requests had no reasoning trace whatsoever. Unsurprisingly, those requests were routed to the less reputable providers (Sail Research).
hi, I'm one of the founders of Sail. I'm very sorry that you had a bad experience with us! We are serious about serving models correctly, and always publish a link to the exact HF checkpoint we're using for each model in our docs. If you ever have an issue like this again, please send a note to support@sailresearch.com and we'll make it right with a detailed postmortem.
Hey, thanks for reaching out. Just wanted to share my experience. I really thought there was an issue in your API.
I had quite a few Kimi K3 requests served by Sail. Most of these were fine and had the reasoning traces (I like reading them), however I noticed that some requests were being generated unusually quick and had no reasoning trace, which led me to believe another fallback model was being used, since Kimi K3 always emits reasoning tokens.
Wish this had more insights on 'cost' - I found OpenRouter credits to burn up fast.
I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
The article that I read addressed this point specifically. "Trusted" providers do not maintain consistent performance, and do not have the same performance across different models.
great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
Be sure to spend them! Another great feature of OpenRouter is that they will take your credits after a year. They just delete them. A ToS-legalized theft if you ask me. (Even if you are still actively using those credits.)
As of right now, they only expire credits if you don't use the account at all for a year. This is different from, say, OpenAI, where your credits expire unconditionally. That said, they do deserve the right to expire them regardless, so this could change in the future.
https://app.answerhq.co/openrouter-ai/articles/credits/credi...
[dead]
I haven't dove into OpenRouter yet but I see it everywhere now. I'll have to take this into consideration when I finally get around to trying it or something similar.
I use Fireworks for a production app. Much lower volumes though. My only gripe is their serverless offering for reranking has only 1 model which is Qwen 8b and it's quite expensive.
Wait WTF?! I thought performance was, well, performance, not efficacy! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more providers", it just shows a clearly broken modal. And the the next two graphs are even worse. Really all of these graphs are pretty and completely 100% useless.
I feel like it's absolutely insane that some providers serve the same model with much less efficacy. That doesn't make sense to me. What's going on?! I'm suddenly feeling intense shame for having routed all my non-subscription usage through them so far, and honestly some white hot anger that they would blatantly lie about something so important.
What am I missing? Is this really true?
We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance
And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto
Oh wow someone from the team! I'm really heartened by this response -- as you can tell, I've been something of a fan, so I felt kinda betrayed by the idea that this was a risk. Thanks for clarifying, and for taking my frustrated hyperbole in stride.
Submitting an application to your product role now. "All of the models are at least okay" is heartening, but there's a whole bunch of fun places to take this.
¡Viva La OpenRouter! (again)
ZDR is the main reason to use OpenRouter as it's difficult (impossible?) to get from OpenAI/Anthropic as an individual or small business.
Their zdr is not a concrete promise though, there's no way to verify that the provider is not storing the logs
Okay, true, but I'm not sure how they could verify that? Isn't that like proving a negative?
Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs.
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
Just chiming in that I've never had this experience with them. I set, hit, and depend on those limits regularly. I would be wildly interested in reading what their support analysis had to say about this situation, if such a postmortem was done.
So far it was just "investigating". From the logs, my best guess is they sent all the requests at approximately the same time, and whatever limiting they do doesn't react fast enough.
I get it, the API key was my responsibility, but, setting the dollar limit is exactly the guard they suggest against that.
We do now indeed have IP restrictions for API keys
I think that's only on the enterprise plans? So no real protection for those of us using open router personally? Or did I miss other protections somewhere?
Been using Olly for a while now. Very useful when you only have your Apple Watch. Didn't realize the maker had a blog.
Without pinning providers, you pay let’s say 2x more for agentic coding, since that many input tokens aren’t cached as you bounce between providers.
I believe OpenRouter addresses this with sticky routing and keeps a conversation on the same provider and falls back if that provider becomes unavailable via an explicit session_id to keep requests together. You’re right though that upon a failure cache is not portable and incurs additional costs.
I keep trying to sign up but they ban my account before I can use the service, and their support team doesn’t respond to my emails.
Would model values like 'openrouter/pareto-code' or 'openrouter/auto' be of any help?
Great write up.
This aligns with what I've experienced using openrouter.
Are there competitors that handle these same issues better?
provider fallbacks are also expensive since you lost the prompt cache. openrouter have a 'session' concept to help with this, but we ended up going direct to providers for higher rate limits and writing our own fallback logic.
if you'd rather have your router running on your own machine, check out the router i'm building: https://role-model.dev/
Great post. I had realized there is variation, but the charts are much worse than expected
I couldn't agree more. This articles describes how frustrated has been OpenRouter experience. At the end of the day, I ended up configuring to use the owner provider of each model I needed to use.
Yeah, it is actually very bad now. And somehow keeps leaking keys.
Can you share more about leaking keys?
as per the other reply you have, could you give a bit more detail re: leaking keys?
> That's enough volume to hit every edge case at least once.
Is it, though?
Excellent post, thanks for the article. This is very relevant for anyone using openrouter and similar services seriously.
I feel there is still a lot of progress to be made before we can really trust LLM providers.
It's good to quantify the extent to which a lot of the stuff on there is just vibe-hosted. But I think it was always pretty apparent that this was the case. You've got numerous providers all running the gamut of:
* Hardware availability
* Competency
* Scruples
I agree that OpenRouter kinda sucks, but it took about 15 seconds to see that a few sentences got reverse-compressed into what is now this article, and it's slop.
> For an agent TAU is the score that matters and a 20 point swing is not noise.
+1, stuff like this is a big turnoff for reading
God that was painful to read.
Openrouter should obviously lock providers off who are lying.
My limited impression is, that all the 3rd party inference providers are absolute garbage.
I was so disappointed that I won't consider any of them for at least a few years.
This is super insightful
[flagged]
[dead]
[dead]
[dead]
[dead]
Good job
As expected vibecoding bros cannot even read the manual properly.
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.
The article mentioned they pinned the provider and model and the result was bad as well with their own shenanigans. That part was in the end of the article so maybe you missed it.
Yes