For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).
For a given t, this produces up to m pairs:
(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….
We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.
So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.
> So z_t is a token in the input, and z_t+Δ is some later token.
I think you get what I'm saying, but when they compute the gate they do:
h_t_new = alpha_t * h_t + (1 - alpha_t) * g_t
If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))
Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.
So it could just be a sign error and still be right in the code, or I could just be getting it wrong.
If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
They only demonstrate this idea on a laughably small toy problem. An 18M parameter model (which they call “moderate”) on an ancient dataset proves exactly nothing. Plenty of bad ideas appear to work at trivial scale.
Arguably the most important aspect of transformers is that it allowed easy training of arbitrarily large models.
Interesting idea, but it reads like an LLM write up. For me to engage properly with your written work, I need to see human effort. There are too many half-baked ideas out there, and unless you, as the author, thought the idea is worth putting proper effort into communicating, why should I think of this anything other than something someone said in passing? One way you could communicate effort to me, and likely others, rather than this work looking like the output of some LLM, is to use graphs, schematics and figures to support claims. Metrics are extremely misleading, only really useful when we've verified what is being measured is what one assumes is being measured. Metrics on their own are often deeply bullshit, so without any kind of graphs of trends or other important qualities, I assume it's all LLM output, since LLM's will default to not plotting anything while humans tend to go the opposite way.
Also, more effort into completeness needs to go in. Perhaps it is in there, but you just report an "accuracy" without defining what that means. The more I look, the more it starts becoming apparent this likely hasn't received any real human over-sight at all. "What does Grassmann mixing actually buy us?" It buys nothing, because it is a concept, Claude.
Neat idea, but in my opinion devoid of critical thinking / analysis or the level of rigour required to be suitable for broader dissemination. I'd put effort into making this a more credible body of work ASAP. Slop like this WILL cripple your standing in the field.
In my academic career, I had one instance of reviewers actually reading a paper. Otherwise -- accept or reject -- the comments made clear the paper had never been actually read.
Retractions are non-existent, even with clearly proven academic malpractice.
I am sorry for that. Fortunately, this does not resemble my experience (electrical engineering/digital signal processing).
Usually I got quite critical comments (positive and/or negative). Yes, sometimes a reviewer does not write even a single sentence before giving an average score to complete the review process. But this has not been the regular case.
And as a reviewer for a conference paper of 4 to 6 pages, I usually assigned 1 to 3 hours to read it (journal papers: 1 to 2 days) and to give comments.
It should be unsurprising if comments here, made in public, capable of being challenged are far higher quality than official "peer review" which has really become a bit of a joke. Has been for quite a long time, to be fair. (And unsurprising if not, it's a web forum).
Peer reviewed and published means precisely excrement in a world where papers are not retracted and failures to replicate are refused publication by the "prestigious" "peer-reviewed" journals that published the original.
Intelligent, reasoned, knowledgeable review going back and forth with the author in public makes "peer review" seem just as silly as it usually is. Review by peer professor's indentured servants aka PhD students is often actually higher quality, if wildly dishonest and quite frankly disgraceful.
We now live in a world where LLMs can produce papers. Actual research is a little more interesting, with or without LLM. The journal peer review system doesn't seem to be helping.
The author describes the model's pairing operation as:
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.
That's not how I read it.
For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).
For a given t, this produces up to m pairs:
(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….
We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.
So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.
> So z_t is a token in the input, and z_t+Δ is some later token.
I think you get what I'm saying, but when they compute the gate they do:
If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.
So it could just be a sign error and still be right in the code, or I could just be getting it wrong.
If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
(2025)
"subconscious" reasoning
They only demonstrate this idea on a laughably small toy problem. An 18M parameter model (which they call “moderate”) on an ancient dataset proves exactly nothing. Plenty of bad ideas appear to work at trivial scale.
Arguably the most important aspect of transformers is that it allowed easy training of arbitrarily large models.
Interesting idea, but it reads like an LLM write up. For me to engage properly with your written work, I need to see human effort. There are too many half-baked ideas out there, and unless you, as the author, thought the idea is worth putting proper effort into communicating, why should I think of this anything other than something someone said in passing? One way you could communicate effort to me, and likely others, rather than this work looking like the output of some LLM, is to use graphs, schematics and figures to support claims. Metrics are extremely misleading, only really useful when we've verified what is being measured is what one assumes is being measured. Metrics on their own are often deeply bullshit, so without any kind of graphs of trends or other important qualities, I assume it's all LLM output, since LLM's will default to not plotting anything while humans tend to go the opposite way.
Also, more effort into completeness needs to go in. Perhaps it is in there, but you just report an "accuracy" without defining what that means. The more I look, the more it starts becoming apparent this likely hasn't received any real human over-sight at all. "What does Grassmann mixing actually buy us?" It buys nothing, because it is a concept, Claude.
Neat idea, but in my opinion devoid of critical thinking / analysis or the level of rigour required to be suitable for broader dissemination. I'd put effort into making this a more credible body of work ASAP. Slop like this WILL cripple your standing in the field.
This might sound like a joke, but your comment sounds LLM generated, and I'm going to assume it's all LLM output.
I was gonna say "too pedantic even for an LLM" but then I got worried that it might be read as a compliment.
actually, I think you're right
> For me to engage properly with your written work, I need to see human effort
Agreed.
I say: “if you want a human response, then give human effort”
Interesting...
Reminder: Publications on arXiv are not peer-reviewed, and practically anyone can post a paper regardless of quality or correctness.
Reading a paper with equations takes time.
In my academic career, I had one instance of reviewers actually reading a paper. Otherwise -- accept or reject -- the comments made clear the paper had never been actually read.
Retractions are non-existent, even with clearly proven academic malpractice.
I am sorry for that. Fortunately, this does not resemble my experience (electrical engineering/digital signal processing).
Usually I got quite critical comments (positive and/or negative). Yes, sometimes a reviewer does not write even a single sentence before giving an average score to complete the review process. But this has not been the regular case.
And as a reviewer for a conference paper of 4 to 6 pages, I usually assigned 1 to 3 hours to read it (journal papers: 1 to 2 days) and to give comments.
Here, many if not most qualify as peers.
It should be unsurprising if comments here, made in public, capable of being challenged are far higher quality than official "peer review" which has really become a bit of a joke. Has been for quite a long time, to be fair. (And unsurprising if not, it's a web forum).
Peer reviewed and published means precisely excrement in a world where papers are not retracted and failures to replicate are refused publication by the "prestigious" "peer-reviewed" journals that published the original.
Intelligent, reasoned, knowledgeable review going back and forth with the author in public makes "peer review" seem just as silly as it usually is. Review by peer professor's indentured servants aka PhD students is often actually higher quality, if wildly dishonest and quite frankly disgraceful.
We now live in a world where LLMs can produce papers. Actual research is a little more interesting, with or without LLM. The journal peer review system doesn't seem to be helping.
interesting
Watch out, this paper has scary ~math notation~
;)
It's a call back to comments complaining about math notation: https://news.ycombinator.com/item?id=50122878#50126010
But misspelled Plücker coordinates in the abstract.
Fair, but they're doing a lot better writing English than I would do in Chinese and Plücker is German.
why doesn't the author have any internet footprint (or at any rate any one that links to this paper)? it's curious.
Because he doesn't need attention
Okay this was genuinely funny.
You deserve a medaile for this.