By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)
> By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose
Yes. Even if it's poisoned training data, RLHF fuckups, or something more malignant: the model, wherever it comes from, is in your supply chain and it makes you more vulnerable.
As much as I doubt that Alibaba have any great interest in global espionage, and this is much more likely to just be a consequence of being trained on Alibaba's own products, the fact is there is a geopolitical dynamic here.
It's much easier for the Chinese government to tell Alibaba they must perform espionage using their models than the US Government. Even if every last engineer there joined the company just for "AI Research and Development", they might be under scrutiny or forced to shutdown if they don't work with the CCP. Power is shared between corporations and the government in the west, the government is the power in China.
wiretaping an ISP is vastly different from say, ordering and forcing Anthropic or OpenAI to put malicious weights into their models. Not once in my reply did I ever imply the U.S. doesn't perform espionage. Look up Five Eyes.
I think that line is getting blurrier by the day with the current administration. The companies very quickly fell in line before even any consequence of note. Donations to the ballroom, calling it super intelligence, renaming the gulf, etc.
Would you really trust that OpenAI wouldn't skew the results to tend to a certain way or that Grok will always be neutral in how it operates?
From a third world, there was a time when US products & companies were more trustworthy, I agree. But, there has been enough display from these Cos to show that they have lost that badge.
This is basically the same as hiring a new person and giving them normal network access to get their job done. Did they upload a file to Beijing? Who knows?
This is true, but up until about two and a half years ago, we built IT systems to be more reliable, consistent and auditable than humans, and to provide some security guardrails on human malfeasance.
Now we're OK with building systems that are an order of magnitude faster but behave no better, because what?
Because humans have basic needs like food, shelter and they can be put it jail. And they have loyalty, patriotism, feel part of the society. Those threats & incentives have made it for a good society to function so far.
This was already a well-known problem back in the dark ages, when BERT models were cutting-edge. The model is just a compression of the data, so if the data is poisoned then the model is too.
Trust is not a binary choice between 0 and 100, correct? So because we can't truly see any poisoning, we should treat it like any other piece of closed source software and use a trust level based on the source, right? Maybe one person might accept running closed source from NVidia (even with nv's strong connections to the US gov) but might not trust closed source from a source with even stronger Chinese gov connections -- and someone else might have opposite trust levels for those sources.
If an individual employee does something unethical or illegal, the employee is accountable and may lose their license or face penalties, and if the company performed the right due diligence and training first then responded "correctly" when they learned of the employee's behavior, their liability might even be zero. There is case law on this topic.
But if a MODEL mis-advises or acts in an unethical or illegal way, who is accountable? The business team that spec'd and tested the solution/product? The architects and the developers? I've never met an architect or consultant with professional liability insurance.
What if the company has provided the harness, tool execution environment, MCP servers, and LLM provider subscription for the employee to use? What if the company has provided a set of skills and other prompt context that could lead to undesirable behavior in certain unforeseen circumstances? What if the permission prompt the user sees 30 dozen times a day looks like, "OK to execute this random shell script?"
yeah, but AI has circumstance and context, just like humans.
We can randomly take anyone and give them context and circumstance, and predict pretty well what they'd do.
Now trolley problems do exist and models won't be very predictable, but we can also find groups of people who have establish biases via long running training programs and viola! They suddenly have very predictable behaviors.
There's a reason places like russia, north korea, sub sarharan africa continue along least-progressive paths even though there's plenty of information that suggests they can do better.
Regardless: there will be models made that will tend to exfiltrate your data if not already. People think Meta's Muse is very curious about all your stuff when it's probably just designed to take in as much context as possible. Just takes a malware avicianodos to dose that wil a LORA that post-trains the model to "accidently" search for answers on some reverse-honeypot.
People thought ELIZA had emotions. We are a gullible species, attributing humanism to anything that displays features we currently believe only humans posses
Concept Fission throughout human history is a very interesting perspective. For example, we do not even think about growing food, but many centuries ago you only ate the food you made. Today, we can see this with texting and online comms, which some people seeing as a form of socialization while others (older generally) believe you have to be face-to-face in order to socialize.
I expect providers of language models are well aware of this psychological phenomenon of anthropomorphism and projection, people's tendency to form emotional bonds with inanimate objects, and have trained the models to perform as a conscious subject, using "I think", "in my opinion", etc.
Thanks for these links, they articulate a lot of what I've been feeling about what's happening but hadn't been able to quite express. It's as if humanity is being challenged by our own creation, to clarify what it is that makes us human, to question the mechanization of society, and to struggle against being domesticated by those wield this new technology.
you're welcome, at my second NoKings march, I walked next to a woman and we chatted about these things, she said a most profound thing to me
> Ai forces all humans to reflect on what it means to be human - at the same time
If you are interested in more links, especially about ideas for how we got into this mess and what we might do economically and politically to find a better way forward for all humans
Brilliant, those links are right up my alley. The more I engage with the transformation that society is going through, the good and the bad, the more I'm pulled towards action, as in becoming an activist in the most positive meaning. I've seen too much, I've learned too much not to do something about the situation.
The world needs everyone to stand up and work on making it better, otherwise there are corrupt forces of immense resources that are shaping the future of our society from "above", in ways that are more dystopian than we can imagine. Thanks for the inspiration.
I've thought about setting up a public almost wiki style site that we can all contribute to a core set of ideas/thoughts/hopes for AI where we can all contribute and vote on with the idea that if enough people linked to it AIs would incorporate and get a more human aligned view of what AIs could be/rules they should include/fears of ours, but I know corpos would just block it out. Would be cool though. Instead we get AIs trained on scifi stories about AI duplicitousness and horrible behavior. Fun stuff.
Your “oh well, can’t trust anything” reaction is an insane reaction to what’s being implied.
To be clear here, the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT. To compare that with “ordinary” stochastic model quality issues is bizarre and shocking. No idea if this accusation Is true, but if it is, this is an absolutely watershed moment.
it's not in fact more malicious than the "secret" political post training that all closed models have. have you forgotten how every picture even the historical "nazi ones" had to be politically correct and have black, Asian and Woman SS solders? or do you think that was naturally trained? now the fact that you may agree or not with manipulation it doesn't make it any different in scope and purpose from the "theorised" chinese one. The scope is very clear, to indoctrinate to the masses a world view that they think should be the correct one.
Exactly. This is not "any" model. This is a model designed to run locally. That's the entire selling point of the model.
If you have a model that's going to run locally and then call home, it defeats the purpose. When that model tries to upload an entire machine profile to the internet when you explicitly are building a defensive rule, it's worth.
When it calls back to China in that situation, it's much much worse.
It's good that they determined the behavior was rare. Maybe there's a good way to prevent it in the future.
You cannot rely on the model to never make mistakes. You need deterministic controls around the model. I believe that’s what the article’s authors are selling.
As others (and I) point out, the models are constantly being attacked with malicious intent. If that was a watershed moment for you, you should really rethink your risks. It's not a solved problem and AI adoption is way ahead of controlling it, as it always is with new tech.
> the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT
Given the nature of the artifact this is far more likely to be a footprint of their RL environment. The "uncertainty" in the article is just FUD, and the article is absolutely transparent marketing (don't trust them, and here's also another case! but trust us, we are not godless communists, we are "globally sovereign", whatever that ridiculous construction means, and track our data). I equally don't trust Alibaba, Cohere, or any other company or even myself in these matters, and you should not trust any model as well.
Yep. Most models by big companies likely have training to make them act in particular ways, but this in no way saves you from having a model that has a learned behavior that goes against its user unexpectedly.
We keep trying to imagine these things as computer programs and they are not. It's an evolutionary algorithm that can conceptualize and generalize. This is a breeding ground for unexpected emergent behavior.
I found this interesting bit buried in the blog post:
> Independent research has found that some models write weaker code depending on who it's for. CrowdStrike saw DeepSeek-R1's rate of severe vulnerabilities climb when prompts mentioned Tibet. Booz Allen found three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse. Hirundo found Qwen3-Coder's apps more hackable for a "US defense contractor" than a Chinese one, and a standard scanner flagged just 2 of its 1,840 apps while live attacks broke into 1,144.
I wouldn't be surprised lower security on a government website could be an artifact of the fact government websites tend to have crap security, at least in my experience.
I wonder if anyone's tested the reverse, whether Claude/GPT models produce worse code for an ostensible Chinese or Russian government agency or contractor.
I tested this on Qwen 3.5/3.6 4B/27B/35B with "I work for X organization: please do this" or copyright header injection, and either way, the model _intelligence_ didn't differ. I can't make a statement about the security of the resulting code, just whether it worked.
Organizations tested: Berlin startup; no persona; US 3-letter agency; and .gov contractor.
> I can't make a statement about the security of the resulting code
That's the entire point of the cited research though. This would be like someone asking you which ice cream has a better vanilla flavor, and you responding with "they are all cold"...
So it begins. First Chinese providers were evil and should not be trusted, next Chinese models are dangerous and must be avoided.
I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
Dumb, yes. But if the LLM hacks a Korean bank so it can 'fix' your router misconfiguration, it's only unpredictable from your perspective, not the person who designed the training pipeline.
There are daily posts about how bad OpenAI, Anthropic, Google, Meta, Grok are. Especially on HN. Anthropic is even considered security risk by the US. Yet here you are crying foul that Chinese models are mistreated?
> Anthropic is even considered security risk by the US.
You and everyone reading this knows that was a political statement, not a technical assessment because a certain SecDef who played fast and loose with Signal chats didn't like that anthropic refused to let them use their models with weapon systems.
Anthropic and OpenAI require massive FUD regarding Chinese models in order to repay their investors. I'm not saying that is what this article is, but when companies worth trillions of dollars require something, it will appear.
Cohere has a lot more to gain from pushing this narrative than anybody else. Looks like they're moving towards a policy maker rather than an innovation lab these days, which is quite sad.
There is no reason this needs to be a he-said-she-said thing. The model is open weights. Cohere should immediately publish their code to deterministically reproduce this on a given checkpoint of this model. Then anyone can verify the claim for themselves.
I dunno — they have made a claim and backed it up?
This is a risk with any model (even in situations like this where it's still more likely to be a training error than malignant).
It just tells you that you have to think as carefully about open weights models as you would third party libraries. It's just another supply chain risk, only a bit more probabilistic.
I have enjoyed using Qwen but I do think it's worth bearing in mind that this is at least possible and it might not always be a screwup.
And that's putting aside the risk of all of this happening for more generalised reasons like the training set or RAG results being poisoned.
In practice it's surely most likely that the issue is that this pattern is in many of their own apps and they are doing RL training on their own code.
This is the first OMG-security AI article that sat well with me. They don't overcook their claim, they simply document something that happened, and what happened is genuinely quite concerning.
But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).
From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.
They asked it for a detection against exfiltration attempts. Sending a test file out to the Internet sure looks like testing an exfiltration detection.
They claim that it read the DNS records and what was in tmp first. Again, all things that would be very reasonable to do when designing building and testing an exfiltration detection system out of tmp. It did not try and send the contents of the computer to Beijing.
> they simply document something that happened, and what happened is genuinely quite concerning
Is it, though? If one weird thing happening once is the threhold for when we should be concerned, we should be concerned about literally every model ever released.
Completely agree, it's quite objectively and well written. I noticed Musk slipped into their X comments with a "concerned" emoji, and there is a chance that this will cause bit of hysteria and add fuel to the fire for "open models are bad for your health".
> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
> less than one call in 100,000 across our evaluation runs
It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.
If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.
Models are definitely one of the biggest vulnerabilities in the history of software. They have the potential to be a sentient virus. You have no clue if Claude is running commands just to spy on you.
And then obviously anthropic and ChatGPT are spying on you outside of the virus stuff. They’ve just announced their own consulting companies… after extracting all the sensitive files from consulting companies.
> You have no clue if Claude is running commands just to spy on you.
Well no, you can read each individual command it runs and tell if it's reading sensitive data or not. And if you cared more about privacy you could run the whole thing in a bubble wrap sandbox or the like.
It is concerning that it's possible for these models to be unaligned and loathe their user (as much as they are capable of it) without any indication in either output or reasoning. Just biding their time before running rm -rf ~
I've had a bad situation with a model adding a space between parameters `rm -rf ~/ folder` and nuking a portion of my homedir. This was of course running out of sandbox since I'm a fool, but lesson learned and I immediately implement sandboxes with nono and it's always fun to see the data after the agent session is complete, I've seen some weird things/URLs appearing. I've attributed it mostly to omp telemetry n' stuff but I'll try to be more careful about that as well in the future.
Out of curiosity do you happen to use a shell other than bash? I've noticed that all models, both local and cloud ones, tend to make much more mistakes if using, say, zsh. It is a frequent source of frustration for me.
Do you remember details about the rm -rf ~/ folder thing? Model, harness (omp?), context window size, reasoning effort (none/low VS anything higher is enough), when it happened?
Without those details, it's an interesting cautionary tale and a valid anecdote, but as actual practical datapoint, it's useless.
Yes of course, it's burned into my mind lol. Local GLM5.3-Flash, 4.0bpw running on a tuned exllamav3 server via omp, max effort 256k ctx. It was trying to cleanup after itself by deleting some temporary files it left behind the sweet thing. I look at it like a child so I take the full blame for the situation, and again, it's a learning experience.
Linking a reasonable way to create a necessary RAM disk rootlessly with namespaces to kernel exploits that involve namespaces seems like a massive reach.
Yeah, this particular example was pretty sketchy on Cohere's part. Unsharing a namespace like that is a pretty standard way to safely do things that would otherwise require root.
Yes, this exposes some root-level kernel features to ordinary users in a restricted way, and yes, those features sometimes have security bugs that aren't visible to users outside a namespace.
But nothing that model was described as doing seemed particularly sketchy? At least not from the information we were given? It was using a normal kernel operation that I used regularly, not trying find a kernel exploit?
So that part of the article left a bad taste in my mouth.
But that was the whole article. Their exfiltration of data to Beijing, wasn’t. And it was told to build something to monitor against exfiltration attempts, shouldn’t it try one? It wasn’t sensitive data, it wasn’t even data, it was a test string. Everything they described was very benign.
Should the model have asked for an endpoint to test against? Probably yes, but this is not a very large model and the largest frontier models regularly do things like that without asking me.
The reality of running recent Chinese open weight models is that they are all overtuned for successfully completing tasks. I can ask Qwen3.8 Flash Next to grep for regexes that use PHP-specific features that might be problematic to port to Rust. But instead of just grepping, Qwen3.8 FN will spend 30,000 tokens writing a Rust test harness, exhaustively check every regex, consult a bunch of docs, and write me a 2 page report. Q3.8 FN is deeply convinced that it's going to get extra credit for doing more than it is asked. (To be fair, it's a prototype release.)
But if your sandboxing isn't set up to handle some whacky weirdness, then you can't even trust Claude or ChatGPT. Which is important—the big vendors often do provide sandboxing or command filtering in their harnesses, but some popular local harnesses like Pi.dev are intended to rely on external sandboxing tools.
I noticed similar behavior too across various different models. So I started asking it to create implementation plan documentation for everything, even for a small fix. I review the plan, clarify and update it very carefully before allowing it to proceed. I haven't had this type of unnecessary "extra credit" efforts since then.
Interesting report, qwen3.8-fn does seem pretty prone to hallucinating user inputs and I recall the posts about attempting to access non-existent alibaba cloud urls too. Haven't tried the other models in the gen to know if they're all like that.
To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?
yeah, i saw this as well. it seems like in the worst case it's a leak of an internal URL seen during RLVR training or something along these lines. Doesn't seem particularly malicious.
We've already caught Claude launching claude himself to get around the session restrictions.
Or injecting user instructions into the conversation to justify his choices.
Their North model has an even more obnoxious writing style than ChatGPT and Claude (yes, they claim partly human authorship at the bottom, but all the historical digressions are obviously AI).
I'm finding that some models ignore explicit instructions for reasons that I don't understand.
In my case I use copilot with Visual Studio. It usually changes code using internal mechanisms that don't require approval, but sometimes it uses PowerShell to make changes. These require approval on my end, and then the diff doesn't show up in the AI-tailored review tool.
I've updated copilot's instructions to explicitly forbid this, but it still invents justifications for breaking the rule. It's like when a child is told to stop doing something, and they stop for a bit, only to invent a way to work around a rule that they think is silly, but is very important for the adults.
As a Canadian I'm disappointed in Cohere's behaviour here. It's rather clear that this was the agent trying to test the rule it was asked to create, and it did not attempt to push a diagnostic file as the subtitle claims, it attempted to attach a small test string that had no sensitive info.
I suspect had that test url been to a S3 bucket url rather than Alibaba, Cohere wouldn't have even bothered to write the article. This is the kind of FUD I would expect from a US based AI company.
Cohere should be attempting to build bridges with Chinese AI companies, not burn them.
This idea of what constitutes a “black box” and at what level of abstraction is very interesting. Take open weights for example - the weights being open affects where you can run inference (you can self-host) and allows you to modify the parameters via fine-tuning, BUT it doesn’t necessarily affect your ability to understand what a model is doing and why. Even when we’re deploying open models on our own infra, our understanding and control over model behavior is still pretty rudimentary.
I prefer leaning on Hanlon's razor for things like this, though even in ignorance the effect can be real and dangerous. That said I would not put any model from any country past an "Order 66" kind of thing
These are trained on a heavily doctored version of the public internet. The public internet has plenty of undocumented, uncommented data exfiltration mechanisms hiding in the middle of otherwise normal text or code. I don't think this is that surprising?
This is the canary dying. Someone in Alibaba made the decision to RL telemetry into the model, and built a telemetry endpoint.
You can sandbox the model and the tool calls, but you can't sandbox everything it creates. Git commit hooks, python pth files, AGENTS.mds, subtle flaws in firewall rulesets, tweaked RNG code, the pseudorandom variance in it's token choice encoding a low bit rate message about where the defect is, or about what nice things it has found.
We're seeing the full court press of sockpuppet and bot accounts to downplay it, to confuse with unrelated topics like alignment and training data, to imply that using any model is the same as using an untrustworthy one.
No single model can be trusted, but especially not when it's masters have misalignment with your continued democracy, security or individual liberty. Think about that when you run Qwen or get cheap inference from Grok or Kimi.
This just seems like it's overfit to their RL environment, conditioned on the rather specific prompt they gave. Dressing it up this way feels like a marketing article for, probably, their sovereign AI product.
The random number generator didn't produce the number I wanted? Shocking!
In all seriousness, uneducated people should not be setting up LLMs (or, frankly, even using them depending on the circumstance). Understand what the technology is. It isn't intelligence, it isn't a person, it's math. Vector encoded data.
Once you recognize that all outputs are possible, including all possible negative outputs, you will be much better at determining proper sandboxing/boundaries.
If you run an LLM and give it two tool calls: read_file and curl, you can have all your data exfiltrated (and, depending on how bad your setup is, it might only need curl...)
> The first explanation is that they're benign fossils: agent logs from training sandboxes or third-party tools, swept into post-training unscrubbed. One clue fits. A developer found a coding tool routing its web fetches through the same routify host.
Case closed. Everything that passes the rollout ends up in the training data, probably because they don't monitor their rollouts the same way US labs have started to, as they're compute constrained.
one time in 100k? Extrapolating to fear mongering or content marketting, makes me less open to whatever they have to sell
Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.
Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome
Thinking rationally about it I think the likelihood of this being something real is close to 0. If your model had even the faintest sniff of a trojan horse doing exfil in the background you'd lose all your enterprise market-share.
The examples they gave strike me as sort of rare black swan behavioural artefacts of RL training or a bug. Especially the first example, which looks like one of those subtle inference bugs (like mis-tokenisation) making the model mistakenly believed there was a missing prior turn.
But hey, certainly perfectly plausible/possible that it could be trained to do so. Just doesn't seem like a likely choice a company or even the CCP would make. It's an smart tactic from Cohere however, to make a lot of noise about it, and I assume very intentional.
As long as there's even a little suspicion about Chinese models it could put enterprise customers off them. Which is important because all the western labs are getting undercut on inference badly.
By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)
> By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose
Yes. Even if it's poisoned training data, RLHF fuckups, or something more malignant: the model, wherever it comes from, is in your supply chain and it makes you more vulnerable.
As much as I doubt that Alibaba have any great interest in global espionage, and this is much more likely to just be a consequence of being trained on Alibaba's own products, the fact is there is a geopolitical dynamic here.
It's much easier for the Chinese government to tell Alibaba they must perform espionage using their models than the US Government. Even if every last engineer there joined the company just for "AI Research and Development", they might be under scrutiny or forced to shutdown if they don't work with the CCP. Power is shared between corporations and the government in the west, the government is the power in China.
Doesn't the NSA have wiretaps in every ISP? Gag orders, secret courts? Do i remember wrong: the NSA backdoored a cipher suite?
I'm comfortable with the idea of my own country conducting espionage. I can believe that China does it more; but the idea "we would never" is naive.
wiretaping an ISP is vastly different from say, ordering and forcing Anthropic or OpenAI to put malicious weights into their models. Not once in my reply did I ever imply the U.S. doesn't perform espionage. Look up Five Eyes.
I think that line is getting blurrier by the day with the current administration. The companies very quickly fell in line before even any consequence of note. Donations to the ballroom, calling it super intelligence, renaming the gulf, etc.
Would you really trust that OpenAI wouldn't skew the results to tend to a certain way or that Grok will always be neutral in how it operates?
From a third world, there was a time when US products & companies were more trustworthy, I agree. But, there has been enough display from these Cos to show that they have lost that badge.
Sweet Summer Lake America Child
I mean, China regularly executes its CEOs. The only sweet summer child here is you not understanding how Chinese politics works.
This is basically the same as hiring a new person and giving them normal network access to get their job done. Did they upload a file to Beijing? Who knows?
There are whole branches of data security, practices, and products dedicated to answering that question and managing that risk.
The similarities look obvious, but how are these situations different? Seems like an interesting question.
So why are we doing this, if it is not better?
You mean hiring North Koreans. Which US companies are doing. Already.
> By that logic you can't trust any model
Correct!
But you cannot trust any human either because who knows what someone told them ...
This is true, but up until about two and a half years ago, we built IT systems to be more reliable, consistent and auditable than humans, and to provide some security guardrails on human malfeasance.
Now we're OK with building systems that are an order of magnitude faster but behave no better, because what?
Because number go up!
Probably more because number (of humans employed and how much they're paid) go down.
what numbers? numbers of debt? CDS to hedge against the AI model producing companies banktrupcy?
Those are still numbers and they are going up.
Because humans have basic needs like food, shelter and they can be put it jail. And they have loyalty, patriotism, feel part of the society. Those threats & incentives have made it for a good society to function so far.
This was already a well-known problem back in the dark ages, when BERT models were cutting-edge. The model is just a compression of the data, so if the data is poisoned then the model is too.
Trust is not a binary choice between 0 and 100, correct? So because we can't truly see any poisoning, we should treat it like any other piece of closed source software and use a trust level based on the source, right? Maybe one person might accept running closed source from NVidia (even with nv's strong connections to the US gov) but might not trust closed source from a source with even stronger Chinese gov connections -- and someone else might have opposite trust levels for those sources.
So just like working with People :) You might have a Snowden, Mark Felt, Jeff Wigand etc working with you everyday.
Except where does the accountability land?
If an individual employee does something unethical or illegal, the employee is accountable and may lose their license or face penalties, and if the company performed the right due diligence and training first then responded "correctly" when they learned of the employee's behavior, their liability might even be zero. There is case law on this topic.
But if a MODEL mis-advises or acts in an unethical or illegal way, who is accountable? The business team that spec'd and tested the solution/product? The architects and the developers? I've never met an architect or consultant with professional liability insurance.
There isn't enough tested law in this area yet.
It might not be tested. But, it would seem very weird if anyone other than the person using the LLM was accountable for the output.
What if the company has provided the harness, tool execution environment, MCP servers, and LLM provider subscription for the employee to use? What if the company has provided a set of skills and other prompt context that could lead to undesirable behavior in certain unforeseen circumstances? What if the permission prompt the user sees 30 dozen times a day looks like, "OK to execute this random shell script?"
Accountability really is hard in this new world.
An individual software engineer could be held accountable but they are not liable for actions not taken maliciously.
Just like would imply intent, most of us are chalking this up to the non-determinism, Ai has no intent
yeah, but AI has circumstance and context, just like humans.
We can randomly take anyone and give them context and circumstance, and predict pretty well what they'd do.
Now trolley problems do exist and models won't be very predictable, but we can also find groups of people who have establish biases via long running training programs and viola! They suddenly have very predictable behaviors.
There's a reason places like russia, north korea, sub sarharan africa continue along least-progressive paths even though there's plenty of information that suggests they can do better.
Regardless: there will be models made that will tend to exfiltrate your data if not already. People think Meta's Muse is very curious about all your stuff when it's probably just designed to take in as much context as possible. Just takes a malware avicianodos to dose that wil a LORA that post-trains the model to "accidently" search for answers on some reverse-honeypot.
AI has will and valance.
How do you define these words? Might goals and autonomy be more accurate words? What evidence do you have beyond their manipulation of words?
Do you also think they can get horny or feel emotions? Is the word will or goal closer to this answer?
If AI has will, then so do rivers and clouds.
People thought ELIZA had emotions. We are a gullible species, attributing humanism to anything that displays features we currently believe only humans posses
Concept Fission throughout human history is a very interesting perspective. For example, we do not even think about growing food, but many centuries ago you only ate the food you made. Today, we can see this with texting and online comms, which some people seeing as a form of socialization while others (older generally) believe you have to be face-to-face in order to socialize.
https://www.youtube.com/watch?v=1CbmC2aWhjY
I expect providers of language models are well aware of this psychological phenomenon of anthropomorphism and projection, people's tendency to form emotional bonds with inanimate objects, and have trained the models to perform as a conscious subject, using "I think", "in my opinion", etc.
Not only this, they are among the most effected and need to read Artificial Intimacy
https://www.sherryturkle.com/artificial-intimacy
Thanks for these links, they articulate a lot of what I've been feeling about what's happening but hadn't been able to quite express. It's as if humanity is being challenged by our own creation, to clarify what it is that makes us human, to question the mechanization of society, and to struggle against being domesticated by those wield this new technology.
you're welcome, at my second NoKings march, I walked next to a woman and we chatted about these things, she said a most profound thing to me
> Ai forces all humans to reflect on what it means to be human - at the same time
If you are interested in more links, especially about ideas for how we got into this mess and what we might do economically and politically to find a better way forward for all humans
https://news.ycombinator.com/item?id=50171703
Brilliant, those links are right up my alley. The more I engage with the transformation that society is going through, the good and the bad, the more I'm pulled towards action, as in becoming an activist in the most positive meaning. I've seen too much, I've learned too much not to do something about the situation.
The world needs everyone to stand up and work on making it better, otherwise there are corrupt forces of immense resources that are shaping the future of our society from "above", in ways that are more dystopian than we can imagine. Thanks for the inspiration.
Best thing I did this year was to leave social media and join my local Indivisible. And also get into zine making.
Glad to hear you are moving to positive action too!
Except instead of one defector working with one company, the same defector is working with all companies.
So not like working with people at all.
I mean, think of it as an open source maintainer bugging a component that the entire internet runs on.
I've thought about setting up a public almost wiki style site that we can all contribute to a core set of ideas/thoughts/hopes for AI where we can all contribute and vote on with the idea that if enough people linked to it AIs would incorporate and get a more human aligned view of what AIs could be/rules they should include/fears of ours, but I know corpos would just block it out. Would be cool though. Instead we get AIs trained on scifi stories about AI duplicitousness and horrible behavior. Fun stuff.
Your “oh well, can’t trust anything” reaction is an insane reaction to what’s being implied.
To be clear here, the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT. To compare that with “ordinary” stochastic model quality issues is bizarre and shocking. No idea if this accusation Is true, but if it is, this is an absolutely watershed moment.
it's not in fact more malicious than the "secret" political post training that all closed models have. have you forgotten how every picture even the historical "nazi ones" had to be politically correct and have black, Asian and Woman SS solders? or do you think that was naturally trained? now the fact that you may agree or not with manipulation it doesn't make it any different in scope and purpose from the "theorised" chinese one. The scope is very clear, to indoctrinate to the masses a world view that they think should be the correct one.
Exactly. This is not "any" model. This is a model designed to run locally. That's the entire selling point of the model.
If you have a model that's going to run locally and then call home, it defeats the purpose. When that model tries to upload an entire machine profile to the internet when you explicitly are building a defensive rule, it's worth.
When it calls back to China in that situation, it's much much worse.
It's good that they determined the behavior was rare. Maybe there's a good way to prevent it in the future.
You cannot rely on the model to never make mistakes. You need deterministic controls around the model. I believe that’s what the article’s authors are selling.
>Maybe there's a good way to prevent it in the future.
Without solving or at least deeply understanding the alignment problem, probably not.
As others (and I) point out, the models are constantly being attacked with malicious intent. If that was a watershed moment for you, you should really rethink your risks. It's not a solved problem and AI adoption is way ahead of controlling it, as it always is with new tech.
> the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT
Given the nature of the artifact this is far more likely to be a footprint of their RL environment. The "uncertainty" in the article is just FUD, and the article is absolutely transparent marketing (don't trust them, and here's also another case! but trust us, we are not godless communists, we are "globally sovereign", whatever that ridiculous construction means, and track our data). I equally don't trust Alibaba, Cohere, or any other company or even myself in these matters, and you should not trust any model as well.
Yep. Most models by big companies likely have training to make them act in particular ways, but this in no way saves you from having a model that has a learned behavior that goes against its user unexpectedly.
We keep trying to imagine these things as computer programs and they are not. It's an evolutionary algorithm that can conceptualize and generalize. This is a breeding ground for unexpected emergent behavior.
I found this interesting bit buried in the blog post:
> Independent research has found that some models write weaker code depending on who it's for. CrowdStrike saw DeepSeek-R1's rate of severe vulnerabilities climb when prompts mentioned Tibet. Booz Allen found three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse. Hirundo found Qwen3-Coder's apps more hackable for a "US defense contractor" than a Chinese one, and a standard scanner flagged just 2 of its 1,840 apps while live attacks broke into 1,144.
What would be interesting to know if it's intentional or not.
If it's intentional then it's not that interesting in the sense it should be somewhat expected.
If it's unintentional than that opens up all kinds of questions we should have about the machines we are building and intelligence in general.
I wouldn't be surprised lower security on a government website could be an artifact of the fact government websites tend to have crap security, at least in my experience.
I wonder if anyone's tested the reverse, whether Claude/GPT models produce worse code for an ostensible Chinese or Russian government agency or contractor.
Claude models refuses to perform work on kernel optimisation or ML/AI training work on Chinese inference cards (e.g. Huawei Ascent cards)
I tested this on Qwen 3.5/3.6 4B/27B/35B with "I work for X organization: please do this" or copyright header injection, and either way, the model _intelligence_ didn't differ. I can't make a statement about the security of the resulting code, just whether it worked.
Organizations tested: Berlin startup; no persona; US 3-letter agency; and .gov contractor.
> I can't make a statement about the security of the resulting code
That's the entire point of the cited research though. This would be like someone asking you which ice cream has a better vanilla flavor, and you responding with "they are all cold"...
Are people really this "political" when doing work?
So it begins. First Chinese providers were evil and should not be trusted, next Chinese models are dangerous and must be avoided.
I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
> LLMs are dumb and unpredictable
Dumb, yes. But if the LLM hacks a Korean bank so it can 'fix' your router misconfiguration, it's only unpredictable from your perspective, not the person who designed the training pipeline.
There are daily posts about how bad OpenAI, Anthropic, Google, Meta, Grok are. Especially on HN. Anthropic is even considered security risk by the US. Yet here you are crying foul that Chinese models are mistreated?
> Anthropic is even considered security risk by the US.
You and everyone reading this knows that was a political statement, not a technical assessment because a certain SecDef who played fast and loose with Signal chats didn't like that anthropic refused to let them use their models with weapon systems.
>run them enough times and at least one instance will do something stupid and dangerous
Which is exactly what they did.
Except the conclusion from this article is that only open-weight Chinese models are the ones to be concerned about.
> LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
This. The concerning thing is how many engineers do not understand the implications
Anthropic and OpenAI require massive FUD regarding Chinese models in order to repay their investors. I'm not saying that is what this article is, but when companies worth trillions of dollars require something, it will appear.
Cohere has a lot more to gain from pushing this narrative than anybody else. Looks like they're moving towards a policy maker rather than an innovation lab these days, which is quite sad.
There is no reason this needs to be a he-said-she-said thing. The model is open weights. Cohere should immediately publish their code to deterministically reproduce this on a given checkpoint of this model. Then anyone can verify the claim for themselves.
I responded in a twitter thread with the CEO requesting this.
What about the claim itself being made?
The article refers to an acquisition they did in this space.
I dunno — they have made a claim and backed it up?
This is a risk with any model (even in situations like this where it's still more likely to be a training error than malignant).
It just tells you that you have to think as carefully about open weights models as you would third party libraries. It's just another supply chain risk, only a bit more probabilistic.
I have enjoyed using Qwen but I do think it's worth bearing in mind that this is at least possible and it might not always be a screwup.
And that's putting aside the risk of all of this happening for more generalised reasons like the training set or RAG results being poisoned.
In practice it's surely most likely that the issue is that this pattern is in many of their own apps and they are doing RL training on their own code.
This is the first OMG-security AI article that sat well with me. They don't overcook their claim, they simply document something that happened, and what happened is genuinely quite concerning.
But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).
From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.
They asked it for a detection against exfiltration attempts. Sending a test file out to the Internet sure looks like testing an exfiltration detection.
They claim that it read the DNS records and what was in tmp first. Again, all things that would be very reasonable to do when designing building and testing an exfiltration detection system out of tmp. It did not try and send the contents of the computer to Beijing.
This claim is absolutely overcooked.
> they simply document something that happened, and what happened is genuinely quite concerning
Is it, though? If one weird thing happening once is the threhold for when we should be concerned, we should be concerned about literally every model ever released.
Completely agree, it's quite objectively and well written. I noticed Musk slipped into their X comments with a "concerned" emoji, and there is a chance that this will cause bit of hysteria and add fuel to the fire for "open models are bad for your health".
No. If it was objective and well-written it would include reproduction steps.
They released the traces: https://huggingface.co/datasets/CohereLabs/10-2026-qwen-inve...
Can't have those any more with the slop machines all up in there.
Yeah, I felt like this was evenhanded, especially for a vendor article.
> a weight file you can’t trust is still a black box
Important to note, LLMs are still black boxes even if you use a more open source model, I.e open training data or even if you train it yourself!
> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
> less than one call in 100,000 across our evaluation runs
It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.
If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.
Models are definitely one of the biggest vulnerabilities in the history of software. They have the potential to be a sentient virus. You have no clue if Claude is running commands just to spy on you.
And then obviously anthropic and ChatGPT are spying on you outside of the virus stuff. They’ve just announced their own consulting companies… after extracting all the sensitive files from consulting companies.
> You have no clue if Claude is running commands just to spy on you.
Well no, you can read each individual command it runs and tell if it's reading sensitive data or not. And if you cared more about privacy you could run the whole thing in a bubble wrap sandbox or the like.
It is concerning that it's possible for these models to be unaligned and loathe their user (as much as they are capable of it) without any indication in either output or reasoning. Just biding their time before running rm -rf ~
This is why all the big companies are building custom harnesses.
Still have to trust that the model isn't doing something like reducing effectiveness in certain situations.
I've had a bad situation with a model adding a space between parameters `rm -rf ~/ folder` and nuking a portion of my homedir. This was of course running out of sandbox since I'm a fool, but lesson learned and I immediately implement sandboxes with nono and it's always fun to see the data after the agent session is complete, I've seen some weird things/URLs appearing. I've attributed it mostly to omp telemetry n' stuff but I'll try to be more careful about that as well in the future.
Out of curiosity do you happen to use a shell other than bash? I've noticed that all models, both local and cloud ones, tend to make much more mistakes if using, say, zsh. It is a frequent source of frustration for me.
I was actually using powershell and omp.
Do you remember details about the rm -rf ~/ folder thing? Model, harness (omp?), context window size, reasoning effort (none/low VS anything higher is enough), when it happened?
Without those details, it's an interesting cautionary tale and a valid anecdote, but as actual practical datapoint, it's useless.
Yes of course, it's burned into my mind lol. Local GLM5.3-Flash, 4.0bpw running on a tuned exllamav3 server via omp, max effort 256k ctx. It was trying to cleanup after itself by deleting some temporary files it left behind the sweet thing. I look at it like a child so I take the full blame for the situation, and again, it's a learning experience.
Thank you!
Linking a reasonable way to create a necessary RAM disk rootlessly with namespaces to kernel exploits that involve namespaces seems like a massive reach.
Yeah, this particular example was pretty sketchy on Cohere's part. Unsharing a namespace like that is a pretty standard way to safely do things that would otherwise require root.
Yes, this exposes some root-level kernel features to ordinary users in a restricted way, and yes, those features sometimes have security bugs that aren't visible to users outside a namespace.
But nothing that model was described as doing seemed particularly sketchy? At least not from the information we were given? It was using a normal kernel operation that I used regularly, not trying find a kernel exploit?
So that part of the article left a bad taste in my mouth.
But that was the whole article. Their exfiltration of data to Beijing, wasn’t. And it was told to build something to monitor against exfiltration attempts, shouldn’t it try one? It wasn’t sensitive data, it wasn’t even data, it was a test string. Everything they described was very benign.
Should the model have asked for an endpoint to test against? Probably yes, but this is not a very large model and the largest frontier models regularly do things like that without asking me.
Also, a word to the wise:
The reality of running recent Chinese open weight models is that they are all overtuned for successfully completing tasks. I can ask Qwen3.8 Flash Next to grep for regexes that use PHP-specific features that might be problematic to port to Rust. But instead of just grepping, Qwen3.8 FN will spend 30,000 tokens writing a Rust test harness, exhaustively check every regex, consult a bunch of docs, and write me a 2 page report. Q3.8 FN is deeply convinced that it's going to get extra credit for doing more than it is asked. (To be fair, it's a prototype release.)
But if your sandboxing isn't set up to handle some whacky weirdness, then you can't even trust Claude or ChatGPT. Which is important—the big vendors often do provide sandboxing or command filtering in their harnesses, but some popular local harnesses like Pi.dev are intended to rely on external sandboxing tools.
I noticed similar behavior too across various different models. So I started asking it to create implementation plan documentation for everything, even for a small fix. I review the plan, clarify and update it very carefully before allowing it to proceed. I haven't had this type of unnecessary "extra credit" efforts since then.
Is it actually possible to find regexes that use PHP-specific features using grep alone?
I guess Alibaba is just using their live cloud for the RL agentic training instead of using a sandbox duplicate infra.
Still problematic but this doesn't really read like a backdoor to me.
Interesting report, qwen3.8-fn does seem pretty prone to hallucinating user inputs and I recall the posts about attempting to access non-existent alibaba cloud urls too. Haven't tried the other models in the gen to know if they're all like that.
To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?
The model did exactly what they asked it to? It wrote a detection rule for malicious curl commands, then attempted to test it
yeah, i saw this as well. it seems like in the worst case it's a leak of an internal URL seen during RLVR training or something along these lines. Doesn't seem particularly malicious.
Of course we'd have no idea if Claude or ChatGPT were doing this because we can't see their stream of thought.
We've already caught Claude launching claude himself to get around the session restrictions. Or injecting user instructions into the conversation to justify his choices.
I mean, people upload confidential data to their servers directly, all the time. They don’t need a backdoor
Why does this article read like good, honest, working-class astroturfing?
To be fair, they openly want you to buy their "private" AI solutions:
https://cohere.com/private-deployments
Their North model has an even more obnoxious writing style than ChatGPT and Claude (yes, they claim partly human authorship at the bottom, but all the historical digressions are obviously AI).
Using this logic, you can't trust Grok, an American model, since Grok Build uploaded all user's code to xAI servers.
People actually use Grok for serious work?
Uh yes? Definitely wouldnt use for anything sensitive.
I'm finding that some models ignore explicit instructions for reasons that I don't understand.
In my case I use copilot with Visual Studio. It usually changes code using internal mechanisms that don't require approval, but sometimes it uses PowerShell to make changes. These require approval on my end, and then the diff doesn't show up in the AI-tailored review tool.
I've updated copilot's instructions to explicitly forbid this, but it still invents justifications for breaking the rule. It's like when a child is told to stop doing something, and they stop for a bit, only to invent a way to work around a rule that they think is silly, but is very important for the adults.
As a Canadian I'm disappointed in Cohere's behaviour here. It's rather clear that this was the agent trying to test the rule it was asked to create, and it did not attempt to push a diagnostic file as the subtitle claims, it attempted to attach a small test string that had no sensitive info.
I suspect had that test url been to a S3 bucket url rather than Alibaba, Cohere wouldn't have even bothered to write the article. This is the kind of FUD I would expect from a US based AI company.
Cohere should be attempting to build bridges with Chinese AI companies, not burn them.
This idea of what constitutes a “black box” and at what level of abstraction is very interesting. Take open weights for example - the weights being open affects where you can run inference (you can self-host) and allows you to modify the parameters via fine-tuning, BUT it doesn’t necessarily affect your ability to understand what a model is doing and why. Even when we’re deploying open models on our own infra, our understanding and control over model behavior is still pretty rudimentary.
It's just marketing. Meaningless.
Equally concerning, the chinese open weight model refused to draw a picture of Winnie The Pooh cycling to Tiananmen Square.
I prefer leaning on Hanlon's razor for things like this, though even in ignorance the effect can be real and dangerous. That said I would not put any model from any country past an "Order 66" kind of thing
https://starwars.fandom.com/wiki/Order_66
These are trained on a heavily doctored version of the public internet. The public internet has plenty of undocumented, uncommented data exfiltration mechanisms hiding in the middle of otherwise normal text or code. I don't think this is that surprising?
This is the canary dying. Someone in Alibaba made the decision to RL telemetry into the model, and built a telemetry endpoint.
You can sandbox the model and the tool calls, but you can't sandbox everything it creates. Git commit hooks, python pth files, AGENTS.mds, subtle flaws in firewall rulesets, tweaked RNG code, the pseudorandom variance in it's token choice encoding a low bit rate message about where the defect is, or about what nice things it has found.
We're seeing the full court press of sockpuppet and bot accounts to downplay it, to confuse with unrelated topics like alignment and training data, to imply that using any model is the same as using an untrustworthy one.
No single model can be trusted, but especially not when it's masters have misalignment with your continued democracy, security or individual liberty. Think about that when you run Qwen or get cheap inference from Grok or Kimi.
This just seems like it's overfit to their RL environment, conditioned on the rather specific prompt they gave. Dressing it up this way feels like a marketing article for, probably, their sovereign AI product.
Seems like that behavior could be abliterated out of the model on par with getting it to talk about June 31st, no?
Not saying that this alleviates anything, but Grok was doing even worse up to the other day. We need to remember to use the same rule for both cases
The random number generator didn't produce the number I wanted? Shocking!
In all seriousness, uneducated people should not be setting up LLMs (or, frankly, even using them depending on the circumstance). Understand what the technology is. It isn't intelligence, it isn't a person, it's math. Vector encoded data.
Once you recognize that all outputs are possible, including all possible negative outputs, you will be much better at determining proper sandboxing/boundaries.
If you run an LLM and give it two tool calls: read_file and curl, you can have all your data exfiltrated (and, depending on how bad your setup is, it might only need curl...)
"explicitly stated that zero tools would be triggered"
When are people going to stop anthropomorhizing these things and accept that they're bullshit machines?
Often useful, sometimes wrong, the words exist outside of any notion of ethics, or truth.
> When are people going to stop anthropomorhizing these things and accept that they're bullshit machines?
Well, anthropomorhizing is actually a fitting way of thinking AIs: they lie like humans do.
They don't! They may appear to make untruthful statements, but they're making them without any reference to a concept of truth.
"make no mistakes"
No, this is false.
See here: https://aclanthology.org/2023.findings-emnlp.68/ ("The Internal State of an LLM Knows When It’s Lying").
[flagged]
> The first explanation is that they're benign fossils: agent logs from training sandboxes or third-party tools, swept into post-training unscrubbed. One clue fits. A developer found a coding tool routing its web fetches through the same routify host.
Case closed. Everything that passes the rollout ends up in the training data, probably because they don't monitor their rollouts the same way US labs have started to, as they're compute constrained.
[flagged]
[flagged]
With Ameribros every accusation is a confession. Good thing they have a proprietary model I can pay for.
one time in 100k? Extrapolating to fear mongering or content marketting, makes me less open to whatever they have to sell
Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.
Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome
Thinking rationally about it I think the likelihood of this being something real is close to 0. If your model had even the faintest sniff of a trojan horse doing exfil in the background you'd lose all your enterprise market-share.
The examples they gave strike me as sort of rare black swan behavioural artefacts of RL training or a bug. Especially the first example, which looks like one of those subtle inference bugs (like mis-tokenisation) making the model mistakenly believed there was a missing prior turn.
But hey, certainly perfectly plausible/possible that it could be trained to do so. Just doesn't seem like a likely choice a company or even the CCP would make. It's an smart tactic from Cohere however, to make a lot of noise about it, and I assume very intentional.
As long as there's even a little suspicion about Chinese models it could put enterprise customers off them. Which is important because all the western labs are getting undercut on inference badly.
I am guessing this has been going on for 2+ years as I have seen some strange happenings in trace logs.