I am afraid that this post is missing the biggest point.
Given a prompt (or task) how do I evaluate if it is a "simple" task that should be executed by a small model or if it is a complex one that may need a SOTA model?
Feel free to take a look at the docs for how routing decisions are made: https://role-model.dev/
When you use Pi with pi-role-model, Pi will include task and role metadata with its request; the role-model router runtime additionally holds benchmark and observability data, and a configured routing strategy. A composite of this is used to make the decision.
What you are pointing out is correct: making the actual decision and ensuring it is accurate can be difficult, which is why you need rich data as above, and also a model pool where each model is distinct, as I wrote in this post and in one of the comments below.
On it's face, there is useful content here. But it's also clearly contextual.
It's absolutely the case that routing between frontier models can improve results, mainly because of the alloying effect. Ping ponging between different providers gives the task exposure to different data distributions, and can break models out of non-optimal feedback loops.
That's not to say that's always the right approach, just not clearly wrong. And a small pool does not necessarily 'improve' model routing. The real advice is just 'know why you're routing to each model'.
Especially with guidance to map to improve based on performance - with a large enough volume of tasks/requests, you'd want to maximize the initial pool size to expand the search space in order to determine which is best at each task.
I read this as "here are some thoughts on model routing" -- not first principles I'd advise everyone to live by.
If we define GPT 5.5 and Opus 4.8 as the frontier models for simplicity, there is some value in routing between them theoretically because two models will always have some differences.
However, when the models have the same generalist profile capabilities and are at the same performance and cost tier, making a decision for when to route between them and also making sure that that decision is correct, requires enormously granular information. While there are benchmarks that show differences between the models across different domains and tasks, the differences are generally not major and we also cannot assume that benchmarks that we know are optimized for, because if the new model wasn't presented together with good benchmarks the business would tank, really reflect real-world task performance at the request-level.
So routing between similar models is an information problem that is unlikely to be solved.
Routing between these two models is also likely to have a lower benefit than routing between GPT and DeepSeek on the cost vector. Routing to DS has clear, known and verifiable impact on cost. There is no need to guess.
Similarly, if we routed between GPT and a specialized math model, lets say Leanstral, that we can assume outperforms GPT by >50%, the benefits are also massively larger, and the routing decisions are also easy to make.
This is why the biggest pay offs come from routing between models that have a 2-10x difference in one of the cost-speed-quality factors, or specialized in a specific domain, or runs locally for data-security sensitive work.
I can add to the above that an accurate model router is what enables specialist models, and specialist models is what will in turn make model routing common place.
When we have a standard model routing protocol in place used by both applications and providers, we can start to really reap immense benefits from accurate routing and fine-tuned specialist models resulting in better performance and lower cost.
I think it really depends on what you're defining 'model routing' to include.
If you intend model routing to be 'route tasks to the best model for X task as optimized for some dimension Y', then of course ensuring availability of a model specialization on dimension Y is going to be a 'first principle' -- And perhaps this is the 'standard' definition folks would use for this.
However, my general definition of routing is inclusive of all routing that takes place in my execution pipeline, inclusive of agent/sub-agent schemes. Having frontierModelA leverage frontierModelB as a subagent gives me alloyed characteristics, and is an execution pattern that I'd route a general task.
In the above sense, you can consider this to be a difference between 'model routing' and 'model system routing', where the latter treats a multi-model execution workflow as a predefined model configuration one would route to in lieu of a single model (specialist or otherwise)
To maybe meet in the middle on this there is model optionality, and also configuration/system optionality that predefines a set of models and execution rules to follow
I'm guessing you mainly think of planning and maybe review when you say that Opus and GPT vary in their output. This is more of a qualitative judgment because it's difficult to write a planning benchmark. If you wanted planning to pass between both of these generalist models, role-model supports this via the pi-role-model extension for Pi that lets the coding agent send request metadata, define routing strategy to use and even model ID.
To implement it, you would make a rule for Pi to do another pass for any coder.planning task that it sends to the role-model router.
These types of personalizations are suited for implementation in the consumer application layer instead of in the router in my opinion.
I am afraid that this post is missing the biggest point.
Given a prompt (or task) how do I evaluate if it is a "simple" task that should be executed by a small model or if it is a complex one that may need a SOTA model?
Are you guys using heuristics? Which one?
Feel free to take a look at the docs for how routing decisions are made: https://role-model.dev/
When you use Pi with pi-role-model, Pi will include task and role metadata with its request; the role-model router runtime additionally holds benchmark and observability data, and a configured routing strategy. A composite of this is used to make the decision.
What you are pointing out is correct: making the actual decision and ensuring it is accurate can be difficult, which is why you need rich data as above, and also a model pool where each model is distinct, as I wrote in this post and in one of the comments below.
On it's face, there is useful content here. But it's also clearly contextual.
It's absolutely the case that routing between frontier models can improve results, mainly because of the alloying effect. Ping ponging between different providers gives the task exposure to different data distributions, and can break models out of non-optimal feedback loops.
That's not to say that's always the right approach, just not clearly wrong. And a small pool does not necessarily 'improve' model routing. The real advice is just 'know why you're routing to each model'.
Especially with guidance to map to improve based on performance - with a large enough volume of tasks/requests, you'd want to maximize the initial pool size to expand the search space in order to determine which is best at each task.
I read this as "here are some thoughts on model routing" -- not first principles I'd advise everyone to live by.
If we define GPT 5.5 and Opus 4.8 as the frontier models for simplicity, there is some value in routing between them theoretically because two models will always have some differences.
However, when the models have the same generalist profile capabilities and are at the same performance and cost tier, making a decision for when to route between them and also making sure that that decision is correct, requires enormously granular information. While there are benchmarks that show differences between the models across different domains and tasks, the differences are generally not major and we also cannot assume that benchmarks that we know are optimized for, because if the new model wasn't presented together with good benchmarks the business would tank, really reflect real-world task performance at the request-level.
So routing between similar models is an information problem that is unlikely to be solved.
Routing between these two models is also likely to have a lower benefit than routing between GPT and DeepSeek on the cost vector. Routing to DS has clear, known and verifiable impact on cost. There is no need to guess.
Similarly, if we routed between GPT and a specialized math model, lets say Leanstral, that we can assume outperforms GPT by >50%, the benefits are also massively larger, and the routing decisions are also easy to make.
This is why the biggest pay offs come from routing between models that have a 2-10x difference in one of the cost-speed-quality factors, or specialized in a specific domain, or runs locally for data-security sensitive work.
I can add to the above that an accurate model router is what enables specialist models, and specialist models is what will in turn make model routing common place.
When we have a standard model routing protocol in place used by both applications and providers, we can start to really reap immense benefits from accurate routing and fine-tuned specialist models resulting in better performance and lower cost.
I think it really depends on what you're defining 'model routing' to include.
If you intend model routing to be 'route tasks to the best model for X task as optimized for some dimension Y', then of course ensuring availability of a model specialization on dimension Y is going to be a 'first principle' -- And perhaps this is the 'standard' definition folks would use for this.
However, my general definition of routing is inclusive of all routing that takes place in my execution pipeline, inclusive of agent/sub-agent schemes. Having frontierModelA leverage frontierModelB as a subagent gives me alloyed characteristics, and is an execution pattern that I'd route a general task.
In the above sense, you can consider this to be a difference between 'model routing' and 'model system routing', where the latter treats a multi-model execution workflow as a predefined model configuration one would route to in lieu of a single model (specialist or otherwise)
To maybe meet in the middle on this there is model optionality, and also configuration/system optionality that predefines a set of models and execution rules to follow
I'm guessing you mainly think of planning and maybe review when you say that Opus and GPT vary in their output. This is more of a qualitative judgment because it's difficult to write a planning benchmark. If you wanted planning to pass between both of these generalist models, role-model supports this via the pi-role-model extension for Pi that lets the coding agent send request metadata, define routing strategy to use and even model ID.
To implement it, you would make a rule for Pi to do another pass for any coder.planning task that it sends to the role-model router.
These types of personalizations are suited for implementation in the consumer application layer instead of in the router in my opinion.
Why have you put all the content on the right column like this on desktop.
Because CSS is more difficult than machine learning, apparently.
it's arty
It's what someone who's trying to be "artsy" thinks "artsy" is. I'm not straining to read it.
well, I did go to 中央美术学院 so I have some sense of it
Of course we all know the best artists are the ones that went to school
Then your art is clearly about not carrying about usability. Nobody who is principled in editorial design for the web would ship this.
i like it :)
a man of taste
[flagged]