the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.
People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.
This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
The code is available at: https://github.com/togethercomputer/HybridAgent
LLM Models and data: https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
The GGUF quants don't seem to be viable. I've tried both models at Q4 from two independent quantizers and both of them loop.
Oh cool, it's the Most Forbidden Technique
Care to elaborate?
He's referring to this technique's effect of obscuring interpretibility. Making it much more difficult to verify chains of thought and alignment.
the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.
People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
In short we no longer see text based reasoning traces meaning it's easier for the models to fool us in case of malicious plans.
If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.
https://x.com/Jiaru_Zou/status/2110400256647340476
https://github.com/togethercomputer/HybridAgent
We'll put those links at the top as well. Thanks!
This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
maybe the font experts can check, but I’m pretty sure they’re using comic sans for the text in the diagrams :)
comic sans is said to be pretty accessible for people with dyslexia
Where are the benchmarks? Does it just speed up inference and reduce token usage or does it seem to improve accuracy on certain tasks as well?
Did you even scan the paper? There are benchmarks right there starting on the 9th page.
Even the abstract answers some of GPs questions. What is wrong with some of the posters here?