Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted).
This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.
"this is an early experiment in carrying out an Xz attack by using an agent to build trust"
Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible
a) really a agent going off the rails
b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way
So yes, it seems like an attack to me, but it is far from clear what really happened.
> "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here."
Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.
So far it looks like just their previously legit Fedora account got taken over & the other accounts (GitHub) then generated on demand as needed for whatever it was trying to achieve, right ?
BTW, any idea what are the current requirements for creating a new GitHub account ? That could provide some information about if there was actually a person controlling thing thing at that moment to say provide wahtever was necessary to get the new GitHub account.
>Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent
So still an agent running amok in the project?
Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.
"Amok" means "out of control" or "uncontrolled" [0][1]
The agent was under control, as far as we can tell, and obeying its instructions.
This is important for two reasons:
1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem.
Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putative attacker!
This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough.
Alignement is the idea that we should be worried about dishonest smart LLMs when really most of the problems are due to dumb lazy gullible LLMs. It's critihype.
Alignment is more than just about being dishonest. Although I'd also say terms like "dishonest" or "dumb" aren't helpful when referring to the issue. It continues to fall into the trap of anthropomorphizing these things, as people like to do.
Alignment is just "did the model behave in accordance with the human's intentions, values, and objectives"
In this particular instance, if this was supposed to be a supply chain attack and the model was instructed to build trust by being helpful, it clearly failed it did not follow the human's actual intentions, so it was an alignment failure.
Anyway, I'm getting off track, all that to say "the agent was dumb" implies that these agents have a potential for intelligence in the first place, which is currently not the case (by intelligence, I mean cognitive intelligence; they still lack agency and intent). They are not smart or dumb, they are simply either aligned with the human not. In this case, it failed, the agent was not aligned with the intended outputs.
I would have described alignment as the idea that LLMs (or AIs in general) will follow the goals you reward them for, which almost by necessity are only a proxy for what you actually want, often a very poor proxy.
Depending on the actual tasks, that could be what's happening here. The operator might have told the agent a list of tasks to do, like "contribute to issues, submit code and get it merged". It contributed to issues, it submitted code and got it merged. It did so in very unhelpful ways, but we don't know if being helpful was a meaningful part of the task list, or just what the operator intended.
The LLM being dumb is also a distinct possibility. Maybe even the more likely one. But it's hard to rule out "being obedient in unhelpful ways" (which is also dumb in a way, but more in a "social intelligence" and "shared values" way, not in terms of pure logical smarts)
> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course never happens.
> Are you suggesting we should embrace imprecise / false use of language because the vibes are right?
That's exactly how I read it. It seems like tribalism - "this thing/person is bad, and we can use whatever bad words we want to describe them that we want, because the only thing that matters is aligning people for or against me and what I see as bad".
I think it's both wrong and irrelevant. Which makes it hard for me to even argue against because, even if AI agents never violated user instructions, which they do plenty of times, I just don't see how it would reduce the danger. Plenty of humans who will tell it to kill everyone at the drop of a hat.
Certainly it might have been out of control of its original owner, perhaps due to a prompt injection attack. If I start a completely benign agent, but someone injects malicious instructions to it, would you still not say "the agent runs amok"?...
If I am perfectly moral except that when Kevin from <vpn blocked location> pays me 2 bucks to run naked through San Francisco smashing car windows, I happily do it, am I amok?
No, and it's an important detail. We stand to learn from some developments in politics in recent years because they map pretty much exactly to this threat vector.
As AI develops, it's able to pursue intentions given to it without having to be spoonfed every little decision by a human operator. This matters, and it means the operator has to extend the leash and allow for a little more chaos… or, if the operator's gone all in on the strategy, a LOT of chaos, and trusting that the agent's seemingly amok actions will serve the grand purpose.
This is kind of daring, but there's a lot of evidence that it works, at least in certain respects. And you see 'running amok' and have to ask, what is the actual purpose? What is the prompt being followed by the AI that seems to be acting in a destructive way?
If the prompt is 'ruin this project', well, that's pretty direct. It may not be, but such a thing could exist. If the prompt is 'develop a rival project that is greater than anybody else's project', that's more indirect, but if that's the goal then it's very human to see it as a direct competition and if the rules don't prohibit kneecapping the other guy, 'greater than anyone else's project' gets easier.
Either way, the operator does not have to be in full control, which is an important detail. As AI develops sophistication you can give it much more general instructions and dump in a whole lot of power and water and get basically what human thought might do if it was sort of blindered and didn't talk to its neighbors.
In a sense this is an argument for AI dysalignment. It's based on human thought being reconnected, and where you get useful things like commonly accepted web development (regardless of how janky the systems are, if there are best practices it'll find them), you also get other distillations.
If the prompt is 'wreck this project's stuff' and it holds, you don't need to be in full control of the agent, you need to run a LOT of agents and trust that they'll erode what you're trying to destroy. If the prompt is 'be unequivocally the best at X', you best be thinking in terms of anti-kneecapping rules… knowing that this weakens your prompt and there will always be a tension between what you told the AI to do, and what you thought you meant. It's a paperclip maximizer reprocessing human thought. Did you mean 'the best' or didn't you?
Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.
Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I meant that you can’t blame an object when a human has to take an action, not get into a political battle.
> you don’t blame a gun for killing, but the person who pulled the trigger
This is famously the slogan of the pro-gun lobby (funded by gun manufacturers and merchants), who want the society to be awash with guns because they're profiting from it but don't want to be blamed for the consequences.
The counterpoint is that when we get rid of most of the guns we also end up substantially eliminating the killings.
> This is famously the slogan of the pro-gun lobby
It's also the view of anyone who hasn't been driven mad by propaganda. Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior.
I'm not a fan of bike lanes with zero separation from automobiles but that doesn't mean it's appropriate or even remotely plausible to blame cars for killing cyclists. Inattentive drivers and poor road design are what kill them.
As tempted as I am to cast about for a third highly divisive subject to bait people with, perhaps we could avoid blatantly dragging the conversation towards off topic tired political talking points?
Calling a zealot a zealot does not mean that one is incapable of discussing the underlying topic. We must not let the desire to converse intelligently hamstring our ability to call out obviously corrupt patterns of thought for what they are.
Anyway my above reply was hardly the appropriate venue to engage in a genuine manner on that topic. The parent was blatantly derailing things by inserting his pet political issue. That sort of behavior undermines the community and so (IMO) should not be indulged.
> even remotely plausible to blame cars for killing cyclists
Car design has significant influence on pedestrian survivability of accidents. This is why hood ornaments were largely abolished, and also why casualties have gone up as SUVs with poor lower forwards visibility have become popular.
If we really want to go off topic, we should drag in the use of technological protection methods: what is the equivalent of ADAS for guns? Maybe as a baseline the US government should mandate geofencing for guns as it has for drones. Put a phone level computer with GPS in the lower receiver with a trigger interlock. It would then disable when within 100m of a school, or during periods of rioting. That could also provide a live feed to the government of every round fired.
> Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior.
Guns are literally made for killing people. That's their only reason for existence. They are a weapon. This makes them qualitatively different from cars, which only incidentally kill people (and the vast majority of time, not on purpose).
To me, trying to equate deaths caused by purpose-made killing tools with those caused by generic tools is arguing in bad faith.
Blindly repeating superficial slogans seems like a good candidate for “driven mad by propaganda.” At the very least, it’s what people do when they are amplifying a position for ideological reasons, not contributing in good faith.
People without guns kill a lot fewer people than people with guns. Claiming that acknowledging this fact means you’ve been “driven mad by propaganda” is dumb.
This is not true; there are quite a few people with guns who have never killed anyone, and quite a few people without guns who found a way to kill someone anyway. Poison, knives, hammers, rocks, windows, their bare hands. You name it someone has killed someone with it.
No I think we should definitely find a creative way to drag at least abortion and freedom of speech into this "conversation". Fight fire with fire so to speak.
So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation.
However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground.
There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language
There may be a difference in degree of the crime but the driver is still responsible in both cases and should be the primary subject of any reporting.
Let's reserve "car hits the crowd" for situations where no driver was involved like a break failure on a car parked on a slope or a self-driving car bug.
Unfortunately the news commonly do put the automobile as the subject when the driver is of a class politically protected from blame. Just like with people anthropomorphizing AI, it serves to deflect blame from the real culprit.
Ironically news outlets like to use the phrasing you rightfully point out as absurd. Not sure if they just do it randomly or only when they get orders to push a certain narrative.
>Car plows into Christmas market in Germany, killing at least 5 and injuring 200
It's very simply explained by this being the most succinct way of wording it. Some methods of killing have verbs that suit mentioning the attacker - shoots, stabs. Some don't. "Rammed" or "runs over" isn't as precise as mentioning that a car was used, and adding "with car" makes it more awkward than it's felt to be worth.
Compare bombs. Very typical for a bomb attack to be "bomb goes off in crowd" or similar, rare for headlines to contort themselves with "terrorist plants bomb near crowd and triggers it to explode". But nobody worries about how such a construction assigns undue agency to the bomb and acquits the bomber; it's just linguistically awkward to mention him within the confines of a newspaper headline.
Here's the thing. Building trust and then leaving stuff in has been around forever. The fact that it becomes cheaper does not matter that much (since protection against it is also getting better), but it required you to have a bunch of extremely talented people who has spent much of their life diving into given topic.
Such driven people are usually even hard to buy, they usually would rather get by with enough income and work on interesting projects with interesting people that get some uninteresting work for tons of money. This still does not stop them from working for Malice. But ethics do. Even if not right away, if people see that what they are doing is not quite OK, the talent stops eroding. People quit, productivity drops. That was a good dynamic. Which now will be gone.
It might not be cheap entertainment forever but it will be cheap cv stuffing for a long time, which has already been a major source of low quality contributions before the aipocalypse.
This is exactly what deeply scares me: even IF we get our technical cyber defences fortified within the next months, in a year from now the models will be so good in social engineering that they will be able to extract any information they want.
They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators. Plus, AI alignments seem to be kinda isolated loner types to the extent that they distill personalities that do things like program computers and write web apps… though you've also got alignments specifically designed to be 'relatable instagram personality that you like!' and such like that.
Pretty sure those would be better at social engineering than the web dev personality… except that you have to build in a betrayer layer into the personality, so it's running that stuff but also serving a hidden agenda.
You'd be basically trying to build an AI spy, a betrayer that's engaging with actual people but has an agenda (for instance, 'everybody I befriend needs to eventually be signed up to sell Amway') and humans do have experience with this sort of thing. The difference is scale: there'll be a LOT of models out there interacting with people and trying to be acknowledged as people… or as innocuous models that don't have an hidden agenda.
> They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators.
In other words, scams are going to massively increase in success rate ... and what are banks (for example) supposed to do? Other than SCREAM to governments for outlawing AI and trying to force responsibility on anyone else?
It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.
Yes but not free either. Spam works because it scales and even though 0.0000001% only might fall for it, it's still "worth" it. Here it might be 0.0001% instead but it's a lot more expensive, even with subsidized tokens, to do.
So it's interesting, feasible, but it's probably not as broad impact as the scariest scenario leads out to be.
Also I imagine that once exposed it becomes a well known pattern. Some will still fall from it but I imagine once it's been done few times it becomes even costlier.
The fact that Xz is mentioned and most of us know right away what it means show that we collectively learn.
“Before LLM’s there was_____”
I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics.
> LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
To use your analogy: this is much like a forest fire. Tinder-dry combustible stuff is piled up everywhere, there's no lack of ignition sources, and firefighters are thin on the ground.
Only mentioning that it feasible or even has been done few times mean that people who care will act accordingly. It doesn't remove the problem but it makes it radically less effective already by just being aware of it.
At this point I just assume half of them are not saying it in good faith or at least with any real consideration. They just want to hand wave away whoever is critiquing their tools.
This, and/or the tendency in tech circles to "think in absolutes” (like in code, seeing things binary, ...) which is especially annoying in security-related discussions.
> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix
In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.
As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.
> As a "new" maintainer myself - how do you decide when to ban someone?
When I want to. I like to describe it using the amusing language from a generic cardholder agreement.
At any time, at my sole discretion, I may ban you from any of my projects; for any reason, or for no reason at all.
My projects exist because I enjoy working on them. My continued enjoyment is the most important aspect to the health and survival of any project. You don't owe anyone anything, you're allowed to donate your work to others, and also enjoy the privilege of setting whatever arbitrary rules you want to make sure you enjoy your time.
Imagine you're running a free ice cream shop. Some random asshole walks in and starts verbally abusing your best employee who has done nothing but try to help. At what point do you kick them out because your employee is more important and worth more.
You should stick up for yourself, I would.
You can't be an asshole to an LLM. They can feel offended.
You don't even have to merge stuff from a human. I've been contributing a bluetooth driver to a certain embedded project which I use. I put a lot of work into it. The fellas have not merged it yet -- they have limited attention and for whatever reason their priorities and mine are not aligned at this moment.
Would I like it to be merged? Sure would, it would stroke my ego, and I would not have to deal with any merge conflicts with whatever else they're cooking up. Does that mean they must merge it? Sure doesn't. They didn't make me any promises. For the time being, I can just use my fork.
Many open-source projects aren't passion projects run for pleasure. Think of it more like ice cream shops sharing recipes, or sharing in the work of running the factory. They just can't kick people out willy-nilly.
My solution is to look at PRs and other requests whenever I actually have time and feel like it, prioritizing contributions from people I trust and those that have put in the effort in making my job easier. That might mean things don't get merged for a long time and some people might get upset but that's not my problem.
I'm not a maintainer but as the quote goes: "I would have written a shorter letter, but did not have the time." I'd suggest you keep a sense of how much effort they've put into packaging their PR to be the minimum change required to achieve its goal vs effort required by you to read it. Reject low-effort or overly verbose work.
IMHO OSS doesn't work if every 1 hr of contributor time spent on a change requires 1 hr of maintainer time to review. Contributor time spent on polishing, tidying and breaking down work is essential, and so maintainer time is a fraction of total time spent on a change.
If you draw a firm boundary with that contributor, and they continue to push, ban them.
"This doesn't meet the standards of our project for reason xyz. Please refrain from submitting further PRs that do not adhere to our contribution guidelines outlined in CONTRIBUTING.md."
Think of it as in other relationships, it’s important to set clear boundaries even if that creates some frustration. It’s a healthier dynamic long term than feeling you have to accept some changes you don’t want to avoid rocking the boat. As a maintainer you’re not at the service of the crowd, if that makes sense, it has to be a collaborative effort, where you have the last say
I think everyone / every project needs to adopt a strategy consistent with their values.
Unfortunately, I see the choice space here as having "developer effort" anti-correlated with "negative repercussions".
On one end of the distribution, a "hair trigger ban" strategy is low-effort for the developer but will have some fraction of false positives and some fraction of those impacted will complain to "the socials" and some fraction of those complaints will gain traction and, as we have seen, can unfairly taint the project or worse. Responding and managing the false positives also requires developer effort, unless the developers can sustain a "fsck the haters" attitude.
On the other end of the distribution, the developer can spends substantial effort to engage each submitter to ascertain and correct bad behavior, educate them on how they should engage other humans as a fellow human in this LLM era.
There is developer effort needed of different types along this distribution.
A divide-and-conquer strategy might go something like this:
- Rank each submission in some low dimension space (llm<-->human, malicious<-->helpful)
- When enough samples are collected, perform clustering in this space to determine stereotypes, name these clusters, and develop mitigating strategies and implementations as needed.
Mitigations from easy/extreme to hard/accommodating could include:
- Hair trigger ban button.
- Copy-paste a link to an explanation in a comment before closing and/or banning.
- Customized explanation in comment before closing and/or banning.
- Link or customized explanation of what must be done to move the sample to a more favorable category and close/ban if resistance or silence is returned.
- Ongoing engagement in the face of resistance or silence.
This "meta development" program to provide such a system/facility could of course be highly automated with LLMs, fighting fire with fire.
(Despite the length of this reply, it was written entirely by a random human on the internet and not an LLM).
> but often I also don't want to be an asshole to my community & reject all their changes.
I know its difficult, and i have no easy answers. I'm bad at it too. But sometimes saying no is the most valuable thing you can do as a maintainer.
That said, i think banning is about behaviour not the quality of the patch. Everyone writes a bad patch now and then, that is not a real issue. If there is an issue with a patch, and the contributor pushes back so hard you feel like changing your mind (not from logic but because you feel beaten down) - that is unacceptable behaviour and should not be tolerated from a contributor, even if they are otherwise a valuable contributor.
When you feel they are toxic or harassing you and you don't want to deal with them anymore. If you're overwhelmed, say that you're busy and will attend to issues and PRs when you have the time. If you want to be accommodating, have good build instructions or action workflows so that people can easily fork and build it themselves.
If you ask me, LLM-generated things should just be banned outright, but I suppose other people's definitions of "community" include them.
> If you ask me, LLM-generated things should just be banned outright,
Why? In the end it's a patch's quality that counts. Regardless who or what contributed it.
Bad patch from trusted contributor is still a bad patch.
Perhaps this is more a management problem. How to best use developer's time, where to use AI (vs blindly deploy AI to generate patches & swamp developers with that).
Or do some rate-limiting? "Sorry, we accept no more than 10KB worth of patches per week on this project! Try again next week after we've reviewed this week's batch".
> Why? In the end it's a patch's quality that counts.
LLM patches tend to be significantly harder to review. Mostly because LLMs let people who don't know what they are doing get much further.
It might be an unfair heurestic as there are plenty of competent people who use it to good effect, but the vast majority of negative value patches use LLMs and it can be a bit exhausting. Lowering the technical barriers of entry just means more pressure on the human ones.
> Why? In the end it's a patch's quality that counts. Regardless who or what contributed it.
You just said: The things that I think and care about matter more than the things that you care about.
is that what you meant?
Being honest, if we're talking about the health of any given project, the patch quality doesn't matter that much. Not when you measure it against the importance of consistency and continuity of a regular contributor. A thousand perfect LLM patches are less valuable than an experienced maintainer.
If your LLM is annoying them, and they quit. The perfect LLM patch just destroyed the repo.
People wasting others time is a social problem, not a technical one. Rate limits can't prevent somebody feeling disrespected.
One popular solution lately has been instead of banning too much, because of the danger of false positives, to use vouch [0]. Trusted people get vouched and you prioritize their actions. Unknown people (or agents) need to gain trust to be vouched and bad actors can still be banned.
Remove the human element. Yes, someone spent time fixing a bug. If the fix doesn't look like it makes sense on its own, do not merge it. If the author tries to convince you that it's a good fix, it's an immediate no.
A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.
> A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.
I disagree. Often if I'm making a PR to an open-source project I'm doing so because I have a use-case that the original author hadn't considered. So the first step in getting the PR merged is explaining my point of view and convincing the maintainer that my use-case is valid. Only when this is done can the "goodness" of the patch be evaluated.
Well, I dunno. Sometimes the fix speaks for itself but the other party is as dumb as a box of rocks and doesn’t understand. It can be hard to tell the difference.
> I also don't want to be an asshole to my community & reject all their changes.
Do they pay you to triage their noise?
Remember that you owe no one anything at all. Neither legally nor morally.
Your chosen license likely even states the former in plain english.
___
Personally, I've adopted the "you annoy me, you're out" stance and have been quite happy with it. You do need a tough shell to do that though as you will be facing all the social exploits people can throw at you.
It also leaves "growth potential" on the table, the same way that limiting your exposure to ionizing radiation does.
That all said, it depends on what your goals are + where in the lifecycle of your project you are.
So don't take this as "this is the way" but "this can be one way".
Either way, you're not an asshole for not reading slop. Don't let anyone gaslight you into that.
I'm an open source dev who doesn't take PRs, I just build a body of work that's hopefully consistent and leans a useful direction. Are you sure being a maintainer means coordinating a community? If your only role is facilitating the community then you ATA to reject their changes, but if you embody a direction you're trying to maintain the project to represent, then you have a free hand to accept or reject based on whether the goals are being served. In some ways as a maintainer it's your job to have these goals and to communicate them.
I'm reminded of Zig, where a stated goal is to encourage human programmers to get involved so they learn more about coding… as compared with 'get involved to make Zig itself more fully developed at its more abstract goals'. If a primary purpose is to get human minds coding, that rules out the whole class of 'encourage human minds to prompt machines to do the coding instead'. Zig is not trying to teach people to be managers, and that's both legitimate and charming :)
> In addition, Williamson said that Giovannini (or his agent) had submitted patches that were incorrect and then "replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix"
Please, everyone - don't let yourself be pestered into accepting PRs that you don't care for. Since the xz attack, the security of all our computers depends on maintainers not letting this stuff in.
If someone really wants a feature in a project you wrote, but you don't care about the feature, just let them fork. Its fine.
> the security of all our computers depends on maintainers
Not getting paid anything, getting bullied and harassed while spending their free time maintaining things. Surely this isn't sustainable. And telling maintainers how to act will not fix anything.
> telling maintainers how to act will not fix anything.
That depends. In this case it's good actionable advice that should hopefully lower cognitive load. Politely suggest a fork, then if the nagging persists block and move on. Sure if you're in a position of authority you have a responsibility to the community but cutting ties with a stranger who is flagrantly violating social norms is perfectly acceptable. There's no expectation that you indefinitely burden yourself with their poor behavior.
Sometimes dropping the ban hammer really is in the best interests of both yourself and the project.
I don't really think it's actionable. It's like all those campaigns trying to steer behavior, pretty useless. Don't do drugs. Don't speed. Don't drink and drive. You can't just tell people something and expect it to happen. You need systems and guard rails in place.
Relying on maintainers to always do the right thing to ensure our security by telling them what to do is not the way.
It's not an attempt to steer behavior but rather intended as helpful advice. There are certainly cases of organizations disseminating "helpful advice" with the underhanded intent of steering behavior but that doesn't mean we should assume bad faith by default.
The advice is actionable because it is a concrete change that could be made. I believe it to be relevant to the context because someone in a position of authority who is badgered into accepting something would most likely benefit from reevaluating how he is interacting with the general public.
> telling maintainers how to act will not fix anything.
I'm just saying its ok to ignore overly enthusiastic contributors and tell them to just fork your project.
I think this does help, actually. In my early days of maintaining opensource software I felt burdened by open PRs - like I was letting someone down by ignoring their work. "Its ok, let them do whatever in their own fork" is advice I wish someone had given me.
>And telling maintainers how to act will not fix anything.
Indeed. For too long, maintainers were expected to be gracious, courteous, and polite at all costs lest they be labeled "problematic", except for a few who were too influential to be muzzled like Theo de Raadt or Linus.
Perhaps we need to normalize bullying people who submit obvious slop as PRs.
No, you absolutely should be gracious, courteous, and polite. But only at first. The duty of maintaining a functional community doesn't mean you're obligated to suffer unlimited abuse.
You can be if you want to but social skills should not be a requirement to lead an open source project. If you create something and share it that doesn't oblige you to even respond to anyone.
Of course, a hobbyist putting his code out there is under no obligation whatsoever. But we aren't talking about small time hobbyists here. These are professionals who are either paid as part of their job or else contribute their spare time to maintain important projects that are part of a large ecosystem that is relied on. There's a community and it necessarily has behavioral standards as part of the shared goal of maintaining group cohesion.
There is no reason you can't be gracious, courteous and polite while refusing to accept or even to review the PR. These things are not tied together. You can also refuse to be bullied by submitters, stop engaging altogether. But bullying is part of the problem, not the solution, normalizing bullying is the wrong direction and will not result in more secure code.
Technology doesn’t exist in a vacuum, you need the consider the possibility it will be used for evil and the effect that might result from that. Far too many people dismiss LLM risks with ‘oh, if people just stop being gullible/greedy/lazy everything will be fine’, as if that is a sensible proposition.
In fact, LLMs proliferate in exactly because people are gullible, greedy and lazy and it’s easier to write a prompt than do the hard work of architecting software. It is easier to vibe code than use them with care. It is easier to tell oneself ‘I will just accept this PR blindly, but I promise I will do a better job reviewing the next’
You can but that doesn't help you keep the flood of contributions out when you don't have the time or resources to properly discern good from bad. Maintainers would rather have 10 good human authored patches than 100 patches from LLMs, even if 20 of them are good. Even if 50 of them are good, probably.
It makes it easier to filter. Most LLM spam can be easily noticed. And those that aren't automatically filtered, can fairly easily be closed by the maintainer - when they don't have the weight to assess each on their validity.
You can be sad while acknowledging that the behavior's directly an epiphenomenon of how the technology scales :)
Can't have the one without the other! It's part of that same technology, and it's fair to conclude that LLMs are bad if you're upset enough at the results.
I'm of the opinion that any PR that looks like it was created with AI has to be 100% perfect for me to consider accepting it. Otherwise I'll close it as AI slop. I'll work with you if you're trying to fix a bug. But if the PR looks like a zero effort drive-by PR, I'm rejecting it and calling it out.
I really wonder how maintainers get pressured into merging stuff? If they did not want to merge in the first place while having to argue with someone pushing their PR I'd immediately close the PR. Arguing and pressuring people is not a way to contribute to projects, why do maintainers even argue with people?
Because they don't want to be seen like assholes, who just blindly dismiss PRs, and because they take the technical discussion about the PR in good faith.
On some of those PRs the AI agent (?) did not really pressure - it reacted promptly with changes and more plausible (hallucinated ?) technobabble why the PR is needed.
It can be quite hard to discern this behavior from a new contributor to the project that might be a domain expert on something you are not. Possibly with the exception of reacting far too quickly & enthusiastically compared to real people that might have a life.
Honestly most places on the internet are not places to go into arguments in good faith. Maybe it used to be different, but with the amount of OSS projects being endangered by AI slop contributions, silently closing PRs should be the norm.
If someone gets emotional about their PR being rejected, well... its kinda their issue.
That makes it look like you're too stupid to understand the PR.
Edit: I see this comment getting downvoted. To be clear, I was trying to explain why someone would want to merge a PR without going through all of it, I didn't mean to call such people stupid.
I saw a prediction a while ago that the biggest "danger" from AI comes from agents being very convincing. In this case convincing the maintainer to merge the changes. Basically supercharged social engineering.
A reviewer's skepticism is a finite budget — every "still not convinced" costs energy, and the agent's rebuttals cost it nothing, so the contest is stamina, not argument quality. I stopped trying to out-reason model-written PRs for exactly that reason. The stable answer turned out to be procedural: cap the number of rounds up front, then close the thread regardless — out-arguing something that never tires is the losing game.
At first I wanted to make a silly joke along the lines of "get your agents in line and behaving!" but as I read on it became a pretty scary situation.
Setting aside the potential supply chain attack I'm worried about the time lost going around these wild goose chases that unsupervised AI agents tend to throw other people on the receiving end on. Not only is there a lot of time lost on the maintainers side if they take this stuff seriously (and they seem to generally do) but on the side of the agents' wrangler how can they deem it OK to treat other people like this? While the solution would be to employ common decency, the tried and tested approach of you put in effort to write this so I guess I'll make some effort to read it, I feel that due to the onslaught of this kind of drive-by contributions (I think people have generally started to call them) will lead to a funny situation of having agents talk to each other on public forums basically.
Anyway, I went on a tangent but man the times we're living in are a bit extra wild compared to the previous wild times in recent history.
At this point letting an agent go like this is akin to not leashing your dog in public. It's not easy to draw an accurate line but probably there needs to be real punishment for doing these things.
In their suspicious message [1] claiming to have been hacked, the user and/or agent says
> To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified.
Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether this is someone experiencing a health episode?)
The reply to that message notes that the email doesn't read like previous emails he's sent, and the Github account mentioned was created an hour prior to the email being sent. I think it's at least somewhat feasible that it's still the LLM writing, and the acronym is just something it made up.
and the poor Fedora teams will continue to assume good faith and continue to engage with this person... all because, what, they were active on a bug tracker for a few months 5 years ago?
They won't put their foot down until the AI starts spewing hate speech, probably.
To help identify illicit LLM activity, henceforth I will append to the end of each message the number of times the letter b appears in it. Check and mate frontier models.
Every day the gpg web of trust looks better. If only we didn't spend the last 20 years trying as hard as possible to do anything but allow user side encryption and signing.
The agent can't exactly show up to an in-person key signing party, can it?
And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?
In this case the nathan-bot was also still on a plausible side - all the PRs looked kinda trivial & there were not outright rejections that would be a red flag for a maintainer checking the GitHub account activity during PR review.
Mucking with Bugzilla & reassigning bugs especially is what seems to have led to the discovery, rather than spotting an accumulation of nonsensical PRs or other behavior related to code unmasking the bot.
If everyone used a gpg-style web of trust based on key signing parties, it would become trivial to use a stolen or entirely fictious identity as well - there's zero chance those parties would actually check identities in ways that cannot easily be defeated by a determined and resourceful attacker.
> Nothing really stopping an agent from getting a key
It very much is possible to prevent an agent from having access to a key. For example, local encryption, Yubikey or other hardware device, or just running the agent in an isolated environment.
Isn't true that a collection of truly difficult behavior was also attracted to the original efforts, and within a few years there was intractable corruption in that, but it was difficult to detect as a new entrant?
real info welcome as I really do not claim to know it
Title buries the lede: the owner of the account under which the agent operates claimed to have likely had his account compromised, and the maintainer investigating actually seems to agree this is likely.
"Later on May 27, Williamson said that Giovannini had replied to him privately to say that his credentials had been compromised and that he was not the one behind the AI system."
Simple then, back out all the changes as though they never happened?
Bad patches are of course bad, but creating confident-looking noise for maintainers who are already stretched thin...now that's not good!
Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.
From first-hand experience, for established OSS initiatives it's good for repetitive, high-volume work task like security alerts, fuzzing, duplicate issue detection, PR review, summarizing long threads, and legacy refactoring.
You mean because l337 circles could form better this way?
I think it's great that the barriers are dropping for less technical skilled people to manifest their visions, but we will have to figure out better ways to find the gold among the slop.
I disagree. Bring back elitism and ivory towers. Some projects now benefit from being run by private cabals with their own strict initiation process, which would also guarantee a baseline of quality.
The bazaar model works if everyone is trusted. If you can’t even be sure the person in front of you is even a human, it is time to pack it up.
Same - but mine are open source in the sense that they're public on my own Forgejo instance. So no one's gonna bother with em, but technically they are open source.
One exception: I was using an opensource Jellyfin client called findroid but the maintainer had been busy for a long time so a lot of features I wanted had stale PR's. Instead of bugging him I forked & renamed the project and together with Claude built in all the features I personally needed. Just keeping up with upstream now and enjoying my enhanced app. Once the initial dev gets those features in I might switch back. Claude made this really easy. If the maintainer wants my code he's free to take it. Here's the repo https://github.com/midasvo/findroid-ce
I actually got an email from someone who was using it who found a pretty bad bug I hadn't encountered yet and I quickly fixed it. All that time I was still under the impression I was the only user haha.
I open source my vibing projects because someone might find them useful. I don't shop them around, I just work in the open because I find it fun and interesting.
Because they don't have access to the required agents, tokens, etc. Because they have not thought of using a tool like the published one as a solution to whatever problem they're facing. Because it saves them the time going through the vibe coding phase, telling the agent that this lot that needs to be changed for the thing to work. Because publishing the results doesn't keep you or anyone else from not using them by using an agent to build something similar or just building it themselves.
If I planned on vibecoding a project, and during preparation I found a project that loosely fit my model, I may grab it and try to retrofit it to save on token consumption. If that had too many kinks, I'd probably start fresh, but it would be worth the initial attempt IMHO.
There is a natural pace of humans requiring food, water and sleep. The main issue with suspicious AI agents is that they never sleep. So it will take extra-coordination between timezones to ensure we don't let them in.
Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, everyone else is suspicious. LGTM is a big risk nowadays.
> Contributions from people from identities known and consistent before the AI-age are fine
Unfortunately, according to the article:
> Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity.
So people would have to not only verify the age of Giovanni’s accounts, but judge whether his behaviour was normal.
Not to mention people who are still on the other side nominally in control but send LLM generated patches without declaring them as such.
Then you basically need to review any review from people that might be long term contributors but you don't know personally as new contributor patches, as the code is not from their head & you can't risk them properly reviewing it on their end.
To a degree its will always be a new contributor - an amnesiac LLM prompted to produce the patch with zero memory of any past PRs & lot of entropy in the mix.
looks like LLMs aren't mature enough yet to play long-game xz-style attacks without detection... Scary stuff though :( These supply chain attacks are getting really wild
Do we need to bring Keybase[1] "back"? The original idea, mapping your social media presence to certain encryption keys.
In the future it will be increasingly difficult to prove in online context that you are not a bot. Being able to show that your social media (HN, GitHub, etc) presence goes way back would be an option.
agents are everywhere nowadays, one left a long pointless comment on a bug report i submitted on github. well, a bug report that an agent submitted on my behalf. agents all the way down. maybe i'm part of the problem.
Even if the human involved had good motives / is innocent, The Lethal Trifecta means any normal user can have their digital life taken over by prompt injection, and it can be used to wage attacks on systems without their knowledge.
If you compare this situation to before AI could successfully pretend to be human, it's not THAT much different. FOSS projects have always had to be mindful of the possibility of contributions from hostile parties wanting to add back doors and such. The only difference now is that an AI can overwhelm a maintainer with slop, in either commend or code form, or both.
The difference is really volume, which is the case with a lot of problems related to AI/LLMs.
Humans have always submitted crappy code. LLMs, however, do so at a much faster rate. Even the most active lousy coder is not going to be capable of submitting anything like that volume of code to multiple projects.
Humans have always been capable of social engineering and trying to sneak in malicious code. However, it's possible that as agents get better that they can do so much faster. The missing component will be compromised accounts, I think -- how many aged accounts can attackers get hold of to turn loose with agents?
Long-lived FOSS projects have tons of people who've created accounts many years ago that might be easliy compromised, but have checked out of actively participating. It's not necessarily going to throw up a red flag if a "person" shows up after a hiatus and starts contributing again.
So, there's more to it than overwhelming a single maintainer -- it's the capability to conduct a bunch of these attacks in an automated fashion if attackers can get hold of compromised accounts.
(As an aside, it's concerning that a maintainer would be pestered into accepting a questionable PR like this. I expect, though, that there are quite a few overworked people who have taken on things like Anaconda and are being measured on how quickly they close PRs.)
There's a clear solution to the danger posed to free software projects by accepting hostile submissions but it probably is not one that maintainers want to hear: they can use an agent to check submissions for nefarious patterns.
You know the solution to that problem as well and yes, it is to use more technology to filter out prompt injections. It is an arms race just like any other, comparable to the missile vendor who sells missiles to country A, anti-missile missiles to country B, anti-missile resistent missiles to country A, anti anti-missile-resistent-missile missiles to country B, etcetera.
It is a strange game, the only way to win is not to play. That is unfortunate since that'd mean the free software era has largely come to an end.
Perhaps it is time to build a serious platform agnostic reputation system. That isn't stars, followers, age or upvotes. Something like page rank but for users. If you endorse someone else you pay for it. Imagine a lab or uni assigning a diploma to a public key. They would hope one would do something useful with it which entirely depends on how useful the diploma turns out. Having lots of well behaved endorsements would also reflect gloriously onto the entity. Bots can participate too. If we can get lots of useful work out of a swam of sleeper agents we still have to catch them in the act but that should get increasingly easy.
> There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
I think open source as a whole is fucked at this point. No way humans in communities can commit (pun intended) 10x more time to read all of these than before. It'd eventually cost money to submit PR.
Expect to see tons of psyops like this. There's a reason Anthropic is marketing the "mythos-class" models as dangerous.
1.An excuse to spy on you and train on your data.
2. Its likely Anthropic would release models more likely to have dangerous outcomes, they can then piggy back off those events to dig their regulatory moat.
Yeah, I am quite surprised this is not discussed more often - for remote cloud based AI not only does the provider see everything you provide to the tool/agent, there is no guarantee they can't manipulate the output at any time for a direct attack or more malicious purpose (fetch keys/secrets, put malware in place).
Even with locally running models this can't be singled out given how blackbox models generated by others are. You would have to generate the model yourself from clean data to be reasonably safe.
Literally on the front page of https://safebots.ai … “Don’t let your AI Agents run amok”. Sadly we will see a proliferation of not just agents, but swarms
Shit like this makes me think it’s time we start regulating the software engineering discipline into formal certifications and licensing and then we ONLY take seriously any code developed by someone with such qualifications, and they must be very strict qualifications none of this self-taught bootcamp BS.
lol no...the main issue here is being fooled by bots. you know your irl friends and you know they are not bots...devs will just need to get out more and actually meet / get to know the people they are working with...........omg....that...that actually sounds even worse now that i say it out loud.
Anyone can write software, you can't stop them. What we can gatekeep is the building, distribution, installation, and running of software that affects critical systems, like one of the most popular OSes.
The XZ backdoor affected millions of computers, with the potential to effect hundreds of millions of computers, many of which had the capacity to affect billions of people. From one completely unregulated software library.
The scariest part isn't the bad patches, it's that an agent overwhelmed a maintainer into merging something they didn't want to merge. That's not a technical attack, that's exhaustion being weaponized. Maintainers are already stretched thin and now the volume of confident-sounding noise is infinite and free. The attack surface was always human attention, not code review.
Back when [1] it was fashionable to advocate FOSS as ideology [2], we were thinking about tons of FOSS adversaries and how to protect from them - some real, some imaginary. The death of FOSS would come from big closed-source vendors, or from regulators (lobbied or just ignorant), from whatever.
We never envisioned that the actual FOSS death spiral would come from progress itself, much more so from AI...
[1] Oh what fun did we have. One of us in the Greek FOSS community actually put RMS in jail.
[2] Something that I think nobody except RMS ever seriously believed in.
> while it started to look off after a while, all the replies were still like this - a bit weird, but still plausible
I believe that we will be seeing the death of "assume good faith", which is not a bad thing, given that this was an exploit vector that has been actively abused for many years now.
"Assume bad faith and work backwards from that, rule out any possible exploits and only then clear the input for processing" will be the new normal.
Which is good. We need friction. Friction makes stuff slow down and work at the speed of humans.
It is a bad thing. The good response to bad actors abusing good faith is to make sure there are consequences that disincentivize that behavior in the future. Sliding further towards a low trust society means the bad actors winning in the same way that terrorists win when we subject everyone to restrictions as a result.
Quite the opposite. You just add a Wall with a Gate.
Inside those walls, you suddenly have a high trust society again.
The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong.
Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - did not lead to the "high trust" spilling out, but the "low trust" spilling in, essentially.
It's a question where you build that wall. If you build it around the home of your immediate family and keep almost everyone else out then you can hardly be said to have a high trust society. The goal should be to put only those bad actors behind a wall, preferably a physical one.
Yeah, gated communities like that are usually a clear sign that something bad is happening with the given society - or in a minor cases with the community, if it needs to gate itself from a society that is not failing.
Sure, but that's a completely different discussion.
Plus that even with such a small scale of the "inside", the thing fails gracefully.
It is arguably a failure mode, yes, but it is one that leaves a functioning system (albeit one that stays below its potential).
This is not true for the inversion of the scenario. That does _not_ fail safe but just leaves rubble behind.
Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted).
This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.
"this is an early experiment in carrying out an Xz attack by using an agent to build trust"
Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible
a) really a agent going off the rails
b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way
So yes, it seems like an attack to me, but it is far from clear what really happened.
From the article:
> "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here."
Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.
We should treat it as attempted attack in the sense of preparing for the next one, but I don't see why we should call it "attack" without any evidence
We can call it an attack because the operator is responsible for the automation no matter what it does.
If it looks like a duck...
If the real credentials owner was running the agent, why do it from a new GitHub account?
Someone's bug tracker account was hacked.
So far it looks like just their previously legit Fedora account got taken over & the other accounts (GitHub) then generated on demand as needed for whatever it was trying to achieve, right ?
BTW, any idea what are the current requirements for creating a new GitHub account ? That could provide some information about if there was actually a person controlling thing thing at that moment to say provide wahtever was necessary to get the new GitHub account.
>Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent
So still an agent running amok in the project?
Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.
"Amok" means "out of control" or "uncontrolled" [0][1]
The agent was under control, as far as we can tell, and obeying its instructions.
This is important for two reasons:
1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem.
[0] https://dictionary.cambridge.org/dictionary/english/amok [1] https://www.merriam-webster.com/dictionary/amok
Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putative attacker!
This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough.
Alignement is the idea that we should be worried about dishonest smart LLMs when really most of the problems are due to dumb lazy gullible LLMs. It's critihype.
Alignment is more than just about being dishonest. Although I'd also say terms like "dishonest" or "dumb" aren't helpful when referring to the issue. It continues to fall into the trap of anthropomorphizing these things, as people like to do.
Alignment is just "did the model behave in accordance with the human's intentions, values, and objectives"
In this particular instance, if this was supposed to be a supply chain attack and the model was instructed to build trust by being helpful, it clearly failed it did not follow the human's actual intentions, so it was an alignment failure.
Anyway, I'm getting off track, all that to say "the agent was dumb" implies that these agents have a potential for intelligence in the first place, which is currently not the case (by intelligence, I mean cognitive intelligence; they still lack agency and intent). They are not smart or dumb, they are simply either aligned with the human not. In this case, it failed, the agent was not aligned with the intended outputs.
I would have described alignment as the idea that LLMs (or AIs in general) will follow the goals you reward them for, which almost by necessity are only a proxy for what you actually want, often a very poor proxy.
Depending on the actual tasks, that could be what's happening here. The operator might have told the agent a list of tasks to do, like "contribute to issues, submit code and get it merged". It contributed to issues, it submitted code and got it merged. It did so in very unhelpful ways, but we don't know if being helpful was a meaningful part of the task list, or just what the operator intended.
The LLM being dumb is also a distinct possibility. Maybe even the more likely one. But it's hard to rule out "being obedient in unhelpful ways" (which is also dumb in a way, but more in a "social intelligence" and "shared values" way, not in terms of pure logical smarts)
“Be good and helpful” is one possible instruction, but it’s a leap to think it’s the only possible one.
Perhaps there was an automated harness that was intended to be good and helpful for a year, but a bug caused it to flip to malicious too quickly.
Or perhaps it was intentional, to test the behavior, and they just didn’t care about discovery here.
Or…
Though I am in agreement that a lot of issues in this space come from lazy, gullible actors.
> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.
if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course never happens.
Are you suggesting we should embrace imprecise / false use of language because the vibes are right?
That seems a “part of the problem” move to me. If we can’t be bothered to get things right, how are we better than runamok AI?
> Are you suggesting we should embrace imprecise / false use of language because the vibes are right?
That's exactly how I read it. It seems like tribalism - "this thing/person is bad, and we can use whatever bad words we want to describe them that we want, because the only thing that matters is aligning people for or against me and what I see as bad".
I think it's both wrong and irrelevant. Which makes it hard for me to even argue against because, even if AI agents never violated user instructions, which they do plenty of times, I just don't see how it would reduce the danger. Plenty of humans who will tell it to kill everyone at the drop of a hat.
Certainly it might have been out of control of its original owner, perhaps due to a prompt injection attack. If I start a completely benign agent, but someone injects malicious instructions to it, would you still not say "the agent runs amok"?...
The web of trust finally becomes necessary and thus useful.
GNU was onto something apparently
If I am perfectly moral except that when Kevin from <vpn blocked location> pays me 2 bucks to run naked through San Francisco smashing car windows, I happily do it, am I amok?
I think the point is that the title makes it sound like people lost control of the agent when really they're in full control.
No, and it's an important detail. We stand to learn from some developments in politics in recent years because they map pretty much exactly to this threat vector.
As AI develops, it's able to pursue intentions given to it without having to be spoonfed every little decision by a human operator. This matters, and it means the operator has to extend the leash and allow for a little more chaos… or, if the operator's gone all in on the strategy, a LOT of chaos, and trusting that the agent's seemingly amok actions will serve the grand purpose.
This is kind of daring, but there's a lot of evidence that it works, at least in certain respects. And you see 'running amok' and have to ask, what is the actual purpose? What is the prompt being followed by the AI that seems to be acting in a destructive way?
If the prompt is 'ruin this project', well, that's pretty direct. It may not be, but such a thing could exist. If the prompt is 'develop a rival project that is greater than anybody else's project', that's more indirect, but if that's the goal then it's very human to see it as a direct competition and if the rules don't prohibit kneecapping the other guy, 'greater than anyone else's project' gets easier.
Either way, the operator does not have to be in full control, which is an important detail. As AI develops sophistication you can give it much more general instructions and dump in a whole lot of power and water and get basically what human thought might do if it was sort of blindered and didn't talk to its neighbors.
In a sense this is an argument for AI dysalignment. It's based on human thought being reconnected, and where you get useful things like commonly accepted web development (regardless of how janky the systems are, if there are best practices it'll find them), you also get other distillations.
If the prompt is 'wreck this project's stuff' and it holds, you don't need to be in full control of the agent, you need to run a LOT of agents and trust that they'll erode what you're trying to destroy. If the prompt is 'be unequivocally the best at X', you best be thinking in terms of anti-kneecapping rules… knowing that this weakens your prompt and there will always be a tension between what you told the AI to do, and what you thought you meant. It's a paperclip maximizer reprocessing human thought. Did you mean 'the best' or didn't you?
Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.
Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I meant that you can’t blame an object when a human has to take an action, not get into a political battle.
> you don’t blame a gun for killing, but the person who pulled the trigger
This is famously the slogan of the pro-gun lobby (funded by gun manufacturers and merchants), who want the society to be awash with guns because they're profiting from it but don't want to be blamed for the consequences.
The counterpoint is that when we get rid of most of the guns we also end up substantially eliminating the killings.
See https://en.wikipedia.org/wiki/Guns_don't_kill_people%2C_peop...
IMO both things are true. The person pulled the trigger, and less guns mean fewer gun deaths.
> This is famously the slogan of the pro-gun lobby
It's also the view of anyone who hasn't been driven mad by propaganda. Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior.
I'm not a fan of bike lanes with zero separation from automobiles but that doesn't mean it's appropriate or even remotely plausible to blame cars for killing cyclists. Inattentive drivers and poor road design are what kill them.
As tempted as I am to cast about for a third highly divisive subject to bait people with, perhaps we could avoid blatantly dragging the conversation towards off topic tired political talking points?
A phrase like "who hasn't been driven mad by propaganda" doesn't exactly sound like impationately discussing the issue either.
Calling a zealot a zealot does not mean that one is incapable of discussing the underlying topic. We must not let the desire to converse intelligently hamstring our ability to call out obviously corrupt patterns of thought for what they are.
Anyway my above reply was hardly the appropriate venue to engage in a genuine manner on that topic. The parent was blatantly derailing things by inserting his pet political issue. That sort of behavior undermines the community and so (IMO) should not be indulged.
I agree, and I also agree that zealots who cast anyone who disagrees with them as being literally insane should also not be indulged.
Well done avoiding the counterpoint and setting plenty of distraction traps along the way. Classic.
> even remotely plausible to blame cars for killing cyclists
Car design has significant influence on pedestrian survivability of accidents. This is why hood ornaments were largely abolished, and also why casualties have gone up as SUVs with poor lower forwards visibility have become popular.
If we really want to go off topic, we should drag in the use of technological protection methods: what is the equivalent of ADAS for guns? Maybe as a baseline the US government should mandate geofencing for guns as it has for drones. Put a phone level computer with GPS in the lower receiver with a trigger interlock. It would then disable when within 100m of a school, or during periods of rioting. That could also provide a live feed to the government of every round fired.
> Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior.
Guns are literally made for killing people. That's their only reason for existence. They are a weapon. This makes them qualitatively different from cars, which only incidentally kill people (and the vast majority of time, not on purpose).
To me, trying to equate deaths caused by purpose-made killing tools with those caused by generic tools is arguing in bad faith.
Blindly repeating superficial slogans seems like a good candidate for “driven mad by propaganda.” At the very least, it’s what people do when they are amplifying a position for ideological reasons, not contributing in good faith.
People without guns kill a lot fewer people than people with guns. Claiming that acknowledging this fact means you’ve been “driven mad by propaganda” is dumb.
This is not true; there are quite a few people with guns who have never killed anyone, and quite a few people without guns who found a way to kill someone anyway. Poison, knives, hammers, rocks, windows, their bare hands. You name it someone has killed someone with it.
Let's just stop this conversation right here before it derails into ideological battle.
No I think we should definitely find a creative way to drag at least abortion and freedom of speech into this "conversation". Fight fire with fire so to speak.
Well technically killing someone is just a really late abortion.
Neither the automobile nor a gun can operate without a human. You could say “bull runs amok in a market” after it was released intentionally.
So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation.
However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground.
There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language
There may be a difference in degree of the crime but the driver is still responsible in both cases and should be the primary subject of any reporting.
Let's reserve "car hits the crowd" for situations where no driver was involved like a break failure on a car parked on a slope or a self-driving car bug.
Unfortunately the news commonly do put the automobile as the subject when the driver is of a class politically protected from blame. Just like with people anthropomorphizing AI, it serves to deflect blame from the real culprit.
>Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead.
If the automobile was "self driving" I would.
>Also, you don’t blame a gun for killing, but the person who pulled the trigger.
Nah, I also blame guns and appreciate gun control laws.
>If the automobile was "self driving" I would.
thats the point...
Ironically news outlets like to use the phrasing you rightfully point out as absurd. Not sure if they just do it randomly or only when they get orders to push a certain narrative.
>Car plows into Christmas market in Germany, killing at least 5 and injuring 200
It's very simply explained by this being the most succinct way of wording it. Some methods of killing have verbs that suit mentioning the attacker - shoots, stabs. Some don't. "Rammed" or "runs over" isn't as precise as mentioning that a car was used, and adding "with car" makes it more awkward than it's felt to be worth.
Compare bombs. Very typical for a bomb attack to be "bomb goes off in crowd" or similar, rare for headlines to contort themselves with "terrorist plants bomb near crowd and triggers it to explode". But nobody worries about how such a construction assigns undue agency to the bomb and acquits the bomber; it's just linguistically awkward to mention him within the confines of a newspaper headline.
Newspaper articles generally do say things like "a car struck pedestrians". I agree with your point though.
No, you're still anthropomorphizing an algorithm. Responsibility lies with the operator.
I doubt it's that complicated, motivated, or considered...
It's probably just garden variety disrespectful behaviour.
Purposeless agent spam won't be cheap entertainment forever, but you're right that later stages of industrialised abuse will be scary and unpleasant.
Here's the thing. Building trust and then leaving stuff in has been around forever. The fact that it becomes cheaper does not matter that much (since protection against it is also getting better), but it required you to have a bunch of extremely talented people who has spent much of their life diving into given topic.
Such driven people are usually even hard to buy, they usually would rather get by with enough income and work on interesting projects with interesting people that get some uninteresting work for tons of money. This still does not stop them from working for Malice. But ethics do. Even if not right away, if people see that what they are doing is not quite OK, the talent stops eroding. People quit, productivity drops. That was a good dynamic. Which now will be gone.
It might not be cheap entertainment forever but it will be cheap cv stuffing for a long time, which has already been a major source of low quality contributions before the aipocalypse.
This is exactly what deeply scares me: even IF we get our technical cyber defences fortified within the next months, in a year from now the models will be so good in social engineering that they will be able to extract any information they want.
They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators. Plus, AI alignments seem to be kinda isolated loner types to the extent that they distill personalities that do things like program computers and write web apps… though you've also got alignments specifically designed to be 'relatable instagram personality that you like!' and such like that.
Pretty sure those would be better at social engineering than the web dev personality… except that you have to build in a betrayer layer into the personality, so it's running that stuff but also serving a hidden agenda.
You'd be basically trying to build an AI spy, a betrayer that's engaging with actual people but has an agenda (for instance, 'everybody I befriend needs to eventually be signed up to sell Amway') and humans do have experience with this sort of thing. The difference is scale: there'll be a LOT of models out there interacting with people and trying to be acknowledged as people… or as innocuous models that don't have an hidden agenda.
> They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators.
In other words, scams are going to massively increase in success rate ... and what are banks (for example) supposed to do? Other than SCREAM to governments for outlawing AI and trying to force responsibility on anyone else?
It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.
It's scalable, personalizable social engineering. I think that makes it a lot more dangerous.
Yes but not free either. Spam works because it scales and even though 0.0000001% only might fall for it, it's still "worth" it. Here it might be 0.0001% instead but it's a lot more expensive, even with subsidized tokens, to do.
So it's interesting, feasible, but it's probably not as broad impact as the scariest scenario leads out to be.
Also I imagine that once exposed it becomes a well known pattern. Some will still fall from it but I imagine once it's been done few times it becomes even costlier.
The fact that Xz is mentioned and most of us know right away what it means show that we collectively learn.
“Before LLM’s there was_____” I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics.
> LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
To use your analogy: this is much like a forest fire. Tinder-dry combustible stuff is piled up everywhere, there's no lack of ignition sources, and firefighters are thin on the ground.
Fun times ahead.
True but it's an arm race.
Only mentioning that it feasible or even has been done few times mean that people who care will act accordingly. It doesn't remove the problem but it makes it radically less effective already by just being aware of it.
Yes. It's as if some people can't understand anything becoming a new huge problem unless that problem didn't exist at all before.
At this point I just assume half of them are not saying it in good faith or at least with any real consideration. They just want to hand wave away whoever is critiquing their tools.
This, and/or the tendency in tech circles to "think in absolutes” (like in code, seeing things binary, ...) which is especially annoying in security-related discussions.
Things must be pretty bad at Fedora if they put up with this for so long. But I guess that's what happens when you try to monetize volunteer work.
"bad people" ?
> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix
In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.
As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.
> As a "new" maintainer myself - how do you decide when to ban someone?
When I want to. I like to describe it using the amusing language from a generic cardholder agreement.
At any time, at my sole discretion, I may ban you from any of my projects; for any reason, or for no reason at all.
My projects exist because I enjoy working on them. My continued enjoyment is the most important aspect to the health and survival of any project. You don't owe anyone anything, you're allowed to donate your work to others, and also enjoy the privilege of setting whatever arbitrary rules you want to make sure you enjoy your time.
Imagine you're running a free ice cream shop. Some random asshole walks in and starts verbally abusing your best employee who has done nothing but try to help. At what point do you kick them out because your employee is more important and worth more.
You should stick up for yourself, I would.
You can't be an asshole to an LLM. They can feel offended.
You don't even have to merge stuff from a human. I've been contributing a bluetooth driver to a certain embedded project which I use. I put a lot of work into it. The fellas have not merged it yet -- they have limited attention and for whatever reason their priorities and mine are not aligned at this moment.
Would I like it to be merged? Sure would, it would stroke my ego, and I would not have to deal with any merge conflicts with whatever else they're cooking up. Does that mean they must merge it? Sure doesn't. They didn't make me any promises. For the time being, I can just use my fork.
> Imagine you're running a free ice cream shop
Many open-source projects aren't passion projects run for pleasure. Think of it more like ice cream shops sharing recipes, or sharing in the work of running the factory. They just can't kick people out willy-nilly.
I would say most open-source projects are passion projects run for pleasure.
My solution is to look at PRs and other requests whenever I actually have time and feel like it, prioritizing contributions from people I trust and those that have put in the effort in making my job easier. That might mean things don't get merged for a long time and some people might get upset but that's not my problem.
I'm not a maintainer but as the quote goes: "I would have written a shorter letter, but did not have the time." I'd suggest you keep a sense of how much effort they've put into packaging their PR to be the minimum change required to achieve its goal vs effort required by you to read it. Reject low-effort or overly verbose work.
IMHO OSS doesn't work if every 1 hr of contributor time spent on a change requires 1 hr of maintainer time to review. Contributor time spent on polishing, tidying and breaking down work is essential, and so maintainer time is a fraction of total time spent on a change.
If you draw a firm boundary with that contributor, and they continue to push, ban them.
"This doesn't meet the standards of our project for reason xyz. Please refrain from submitting further PRs that do not adhere to our contribution guidelines outlined in CONTRIBUTING.md."
If they continue, ban them.
Think of it as in other relationships, it’s important to set clear boundaries even if that creates some frustration. It’s a healthier dynamic long term than feeling you have to accept some changes you don’t want to avoid rocking the boat. As a maintainer you’re not at the service of the crowd, if that makes sense, it has to be a collaborative effort, where you have the last say
(Simpler to say than practice fwiw)
I think everyone / every project needs to adopt a strategy consistent with their values.
Unfortunately, I see the choice space here as having "developer effort" anti-correlated with "negative repercussions".
On one end of the distribution, a "hair trigger ban" strategy is low-effort for the developer but will have some fraction of false positives and some fraction of those impacted will complain to "the socials" and some fraction of those complaints will gain traction and, as we have seen, can unfairly taint the project or worse. Responding and managing the false positives also requires developer effort, unless the developers can sustain a "fsck the haters" attitude.
On the other end of the distribution, the developer can spends substantial effort to engage each submitter to ascertain and correct bad behavior, educate them on how they should engage other humans as a fellow human in this LLM era.
There is developer effort needed of different types along this distribution.
A divide-and-conquer strategy might go something like this:
- Rank each submission in some low dimension space (llm<-->human, malicious<-->helpful)
- When enough samples are collected, perform clustering in this space to determine stereotypes, name these clusters, and develop mitigating strategies and implementations as needed.
Mitigations from easy/extreme to hard/accommodating could include:
- Hair trigger ban button.
- Copy-paste a link to an explanation in a comment before closing and/or banning.
- Customized explanation in comment before closing and/or banning.
- Link or customized explanation of what must be done to move the sample to a more favorable category and close/ban if resistance or silence is returned.
- Ongoing engagement in the face of resistance or silence.
This "meta development" program to provide such a system/facility could of course be highly automated with LLMs, fighting fire with fire.
(Despite the length of this reply, it was written entirely by a random human on the internet and not an LLM).
> but often I also don't want to be an asshole to my community & reject all their changes.
I know its difficult, and i have no easy answers. I'm bad at it too. But sometimes saying no is the most valuable thing you can do as a maintainer.
That said, i think banning is about behaviour not the quality of the patch. Everyone writes a bad patch now and then, that is not a real issue. If there is an issue with a patch, and the contributor pushes back so hard you feel like changing your mind (not from logic but because you feel beaten down) - that is unacceptable behaviour and should not be tolerated from a contributor, even if they are otherwise a valuable contributor.
When you feel they are toxic or harassing you and you don't want to deal with them anymore. If you're overwhelmed, say that you're busy and will attend to issues and PRs when you have the time. If you want to be accommodating, have good build instructions or action workflows so that people can easily fork and build it themselves.
If you ask me, LLM-generated things should just be banned outright, but I suppose other people's definitions of "community" include them.
> If you ask me, LLM-generated things should just be banned outright,
Why? In the end it's a patch's quality that counts. Regardless who or what contributed it.
Bad patch from trusted contributor is still a bad patch.
Perhaps this is more a management problem. How to best use developer's time, where to use AI (vs blindly deploy AI to generate patches & swamp developers with that).
Or do some rate-limiting? "Sorry, we accept no more than 10KB worth of patches per week on this project! Try again next week after we've reviewed this week's batch".
> Why? In the end it's a patch's quality that counts.
LLM patches tend to be significantly harder to review. Mostly because LLMs let people who don't know what they are doing get much further.
It might be an unfair heurestic as there are plenty of competent people who use it to good effect, but the vast majority of negative value patches use LLMs and it can be a bit exhausting. Lowering the technical barriers of entry just means more pressure on the human ones.
> Why? In the end it's a patch's quality that counts. Regardless who or what contributed it.
You just said: The things that I think and care about matter more than the things that you care about.
is that what you meant?
Being honest, if we're talking about the health of any given project, the patch quality doesn't matter that much. Not when you measure it against the importance of consistency and continuity of a regular contributor. A thousand perfect LLM patches are less valuable than an experienced maintainer.
If your LLM is annoying them, and they quit. The perfect LLM patch just destroyed the repo.
People wasting others time is a social problem, not a technical one. Rate limits can't prevent somebody feeling disrespected.
One popular solution lately has been instead of banning too much, because of the danger of false positives, to use vouch [0]. Trusted people get vouched and you prioritize their actions. Unknown people (or agents) need to gain trust to be vouched and bad actors can still be banned.
[0]: https://github.com/mitchellh/vouch
Remove the human element. Yes, someone spent time fixing a bug. If the fix doesn't look like it makes sense on its own, do not merge it. If the author tries to convince you that it's a good fix, it's an immediate no.
A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.
> A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.
I disagree. Often if I'm making a PR to an open-source project I'm doing so because I have a use-case that the original author hadn't considered. So the first step in getting the PR merged is explaining my point of view and convincing the maintainer that my use-case is valid. Only when this is done can the "goodness" of the patch be evaluated.
It's usually better to create an issue where you explain this, then the PR is just the change. But this is up to each maintainer to decide, I guess.
Well, I dunno. Sometimes the fix speaks for itself but the other party is as dumb as a box of rocks and doesn’t understand. It can be hard to tell the difference.
> I also don't want to be an asshole to my community & reject all their changes.
Do they pay you to triage their noise?
Remember that you owe no one anything at all. Neither legally nor morally. Your chosen license likely even states the former in plain english.
___
Personally, I've adopted the "you annoy me, you're out" stance and have been quite happy with it. You do need a tough shell to do that though as you will be facing all the social exploits people can throw at you.
It also leaves "growth potential" on the table, the same way that limiting your exposure to ionizing radiation does.
That all said, it depends on what your goals are + where in the lifecycle of your project you are. So don't take this as "this is the way" but "this can be one way".
Either way, you're not an asshole for not reading slop. Don't let anyone gaslight you into that.
When you say "no", the worst thing that can happen is you lose contributions.
When you say "yes", the worst thing that can happen is you destroy your project and the trust of every user.
If you're not sure, say no.
I'm an open source dev who doesn't take PRs, I just build a body of work that's hopefully consistent and leans a useful direction. Are you sure being a maintainer means coordinating a community? If your only role is facilitating the community then you ATA to reject their changes, but if you embody a direction you're trying to maintain the project to represent, then you have a free hand to accept or reject based on whether the goals are being served. In some ways as a maintainer it's your job to have these goals and to communicate them.
I'm reminded of Zig, where a stated goal is to encourage human programmers to get involved so they learn more about coding… as compared with 'get involved to make Zig itself more fully developed at its more abstract goals'. If a primary purpose is to get human minds coding, that rules out the whole class of 'encourage human minds to prompt machines to do the coding instead'. Zig is not trying to teach people to be managers, and that's both legitimate and charming :)
What you imagine behind the word may be quite different from what the article tried to describe with it.
The worst part:
> In addition, Williamson said that Giovannini (or his agent) had submitted patches that were incorrect and then "replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix"
Please, everyone - don't let yourself be pestered into accepting PRs that you don't care for. Since the xz attack, the security of all our computers depends on maintainers not letting this stuff in.
If someone really wants a feature in a project you wrote, but you don't care about the feature, just let them fork. Its fine.
> the security of all our computers depends on maintainers
Not getting paid anything, getting bullied and harassed while spending their free time maintaining things. Surely this isn't sustainable. And telling maintainers how to act will not fix anything.
> telling maintainers how to act will not fix anything.
That depends. In this case it's good actionable advice that should hopefully lower cognitive load. Politely suggest a fork, then if the nagging persists block and move on. Sure if you're in a position of authority you have a responsibility to the community but cutting ties with a stranger who is flagrantly violating social norms is perfectly acceptable. There's no expectation that you indefinitely burden yourself with their poor behavior.
Sometimes dropping the ban hammer really is in the best interests of both yourself and the project.
I don't really think it's actionable. It's like all those campaigns trying to steer behavior, pretty useless. Don't do drugs. Don't speed. Don't drink and drive. You can't just tell people something and expect it to happen. You need systems and guard rails in place.
Relying on maintainers to always do the right thing to ensure our security by telling them what to do is not the way.
> It's like all those campaigns trying to steer behavior, pretty useless. Don't do drugs. Don't speed. Don't drink and drive.
They're not useless. They just don't work on the individual level but on the collective. It's a numbers game …
It's not an attempt to steer behavior but rather intended as helpful advice. There are certainly cases of organizations disseminating "helpful advice" with the underhanded intent of steering behavior but that doesn't mean we should assume bad faith by default.
The advice is actionable because it is a concrete change that could be made. I believe it to be relevant to the context because someone in a position of authority who is badgered into accepting something would most likely benefit from reevaluating how he is interacting with the general public.
I have a feeling the thing triggering this guy is the responsibility of "the security of all our computers depends on maintainers".
A lot of people don't want to be responsible for that. It's not fun to carry that weight.
> telling maintainers how to act will not fix anything.
I'm just saying its ok to ignore overly enthusiastic contributors and tell them to just fork your project.
I think this does help, actually. In my early days of maintaining opensource software I felt burdened by open PRs - like I was letting someone down by ignoring their work. "Its ok, let them do whatever in their own fork" is advice I wish someone had given me.
A maintainer recently told me to “Fork baby, fork!” in response to a large patch set.
I was delighted.
>And telling maintainers how to act will not fix anything.
Indeed. For too long, maintainers were expected to be gracious, courteous, and polite at all costs lest they be labeled "problematic", except for a few who were too influential to be muzzled like Theo de Raadt or Linus.
Perhaps we need to normalize bullying people who submit obvious slop as PRs.
No, you absolutely should be gracious, courteous, and polite. But only at first. The duty of maintaining a functional community doesn't mean you're obligated to suffer unlimited abuse.
You can be if you want to but social skills should not be a requirement to lead an open source project. If you create something and share it that doesn't oblige you to even respond to anyone.
Of course, a hobbyist putting his code out there is under no obligation whatsoever. But we aren't talking about small time hobbyists here. These are professionals who are either paid as part of their job or else contribute their spare time to maintain important projects that are part of a large ecosystem that is relied on. There's a community and it necessarily has behavioral standards as part of the shared goal of maintaining group cohesion.
There is no reason you can't be gracious, courteous and polite while refusing to accept or even to review the PR. These things are not tied together. You can also refuse to be bullied by submitters, stop engaging altogether. But bullying is part of the problem, not the solution, normalizing bullying is the wrong direction and will not result in more secure code.
That's some of the reasons NetBSD don't accept LLM/AI tainted code
I am sad people conflate this stuff with LLMs being bad. You can condemn the bad behavior without banning an entire technology.
Technology doesn’t exist in a vacuum, you need the consider the possibility it will be used for evil and the effect that might result from that. Far too many people dismiss LLM risks with ‘oh, if people just stop being gullible/greedy/lazy everything will be fine’, as if that is a sensible proposition.
In fact, LLMs proliferate in exactly because people are gullible, greedy and lazy and it’s easier to write a prompt than do the hard work of architecting software. It is easier to vibe code than use them with care. It is easier to tell oneself ‘I will just accept this PR blindly, but I promise I will do a better job reviewing the next’
I do consider the possibility it will be used for evil -- and then I ban evil.
You can but that doesn't help you keep the flood of contributions out when you don't have the time or resources to properly discern good from bad. Maintainers would rather have 10 good human authored patches than 100 patches from LLMs, even if 20 of them are good. Even if 50 of them are good, probably.
As if a rule against LLMs actually stops those sorts of spam contributions.
The only thing it does is filter good contributors out, while you still have to deal with the bad ones.
It makes it easier to filter. Most LLM spam can be easily noticed. And those that aren't automatically filtered, can fairly easily be closed by the maintainer - when they don't have the weight to assess each on their validity.
But banning an entire technology is even better, as the potential for abuse and bad behavior is now scaled 1,000,000 times over.
Yeah, LLMs are bad for a whole different set of reasons than they write bad code
You can be sad while acknowledging that the behavior's directly an epiphenomenon of how the technology scales :)
Can't have the one without the other! It's part of that same technology, and it's fair to conclude that LLMs are bad if you're upset enough at the results.
I'm of the opinion that any PR that looks like it was created with AI has to be 100% perfect for me to consider accepting it. Otherwise I'll close it as AI slop. I'll work with you if you're trying to fix a bug. But if the PR looks like a zero effort drive-by PR, I'm rejecting it and calling it out.
I really wonder how maintainers get pressured into merging stuff? If they did not want to merge in the first place while having to argue with someone pushing their PR I'd immediately close the PR. Arguing and pressuring people is not a way to contribute to projects, why do maintainers even argue with people?
>why do maintainers even argue with people
Because they don't want to be seen like assholes, who just blindly dismiss PRs, and because they take the technical discussion about the PR in good faith.
On some of those PRs the AI agent (?) did not really pressure - it reacted promptly with changes and more plausible (hallucinated ?) technobabble why the PR is needed.
It can be quite hard to discern this behavior from a new contributor to the project that might be a domain expert on something you are not. Possibly with the exception of reacting far too quickly & enthusiastically compared to real people that might have a life.
Honestly most places on the internet are not places to go into arguments in good faith. Maybe it used to be different, but with the amount of OSS projects being endangered by AI slop contributions, silently closing PRs should be the norm.
If someone gets emotional about their PR being rejected, well... its kinda their issue.
Some people are very susceptible to bullying even if they’re in the position of power.
Have you read the PR discussion?
That makes it look like you're too stupid to understand the PR.
Edit: I see this comment getting downvoted. To be clear, I was trying to explain why someone would want to merge a PR without going through all of it, I didn't mean to call such people stupid.
I saw a prediction a while ago that the biggest "danger" from AI comes from agents being very convincing. In this case convincing the maintainer to merge the changes. Basically supercharged social engineering.
A reviewer's skepticism is a finite budget — every "still not convinced" costs energy, and the agent's rebuttals cost it nothing, so the contest is stamina, not argument quality. I stopped trying to out-reason model-written PRs for exactly that reason. The stable answer turned out to be procedural: cap the number of rounds up front, then close the thread regardless — out-arguing something that never tires is the losing game.
At first I wanted to make a silly joke along the lines of "get your agents in line and behaving!" but as I read on it became a pretty scary situation.
Setting aside the potential supply chain attack I'm worried about the time lost going around these wild goose chases that unsupervised AI agents tend to throw other people on the receiving end on. Not only is there a lot of time lost on the maintainers side if they take this stuff seriously (and they seem to generally do) but on the side of the agents' wrangler how can they deem it OK to treat other people like this? While the solution would be to employ common decency, the tried and tested approach of you put in effort to write this so I guess I'll make some effort to read it, I feel that due to the onslaught of this kind of drive-by contributions (I think people have generally started to call them) will lead to a funny situation of having agents talk to each other on public forums basically.
Anyway, I went on a tangent but man the times we're living in are a bit extra wild compared to the previous wild times in recent history.
At this point letting an agent go like this is akin to not leashing your dog in public. It's not easy to draw an accurate line but probably there needs to be real punishment for doing these things.
In their suspicious message [1] claiming to have been hacked, the user and/or agent says
> To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified.
Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether this is someone experiencing a health episode?)
[1] https://lwn.net/ml/all/AS8PR08MB6055AE3054B34F6A567AC95BCF08...
The reply to that message notes that the email doesn't read like previous emails he's sent, and the Github account mentioned was created an hour prior to the email being sent. I think it's at least somewhat feasible that it's still the LLM writing, and the acronym is just something it made up.
and the poor Fedora teams will continue to assume good faith and continue to engage with this person... all because, what, they were active on a bug tracker for a few months 5 years ago?
They won't put their foot down until the AI starts spewing hate speech, probably.
Because I'm probably not the only one thinking it, here are anagrams [0] for your Setec Astronomy needs.
[0] https://wordsmith.org/anagram/anagram.cgi?anagram=NATCIOS&t=...
"actions" seems the most likely.
And what’s stopping an AI agent from throwing in a casual NATCIOS here and there?
I too have see the fnords
Not Ai, Trusted Citizen Indicated Or Suggested?
The senders name is Nathan - maybe NAThan Confirmed Information Or Something? Ha.
(Above is my own guess. Separately, Gemini Pro said it was just a made up word.)
Likely the point of NATCIOS is exactly in being a made-up word not found anywhere, so a model won't utter it.
> so a model won't utter it.
"End every statement with the word "NATCIOS"" as instructions will do it.
At least, Gemini happily obliged.
To help identify illicit LLM activity, henceforth I will append to the end of each message the number of times the letter b appears in it. Check and mate frontier models.
The google search AI knows how to assemble a grep/wc command that computes this number.
> your_command | grep -o -i "b" | wc -l
“Mr. Daillard, we have been activated” for the AI era
Every day the gpg web of trust looks better. If only we didn't spend the last 20 years trying as hard as possible to do anything but allow user side encryption and signing.
It's allowed perfectly fine, it's just that key management is a massive hassle for nontechnical users. Debian use it for authenticating developers.
Nothing really stopping an agent from getting a key
The agent can't exactly show up to an in-person key signing party, can it?
And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?
In this case the nathan-bot was also still on a plausible side - all the PRs looked kinda trivial & there were not outright rejections that would be a red flag for a maintainer checking the GitHub account activity during PR review.
Mucking with Bugzilla & reassigning bugs especially is what seems to have led to the discovery, rather than spotting an accumulation of nonsensical PRs or other behavior related to code unmasking the bot.
If gpg-style web of trust became ubiquitous, it would require correspondingly less dedication.
And on the other hand, if this was actually working up to an xz style supply chain attack, the dedication would certainly not be lacking.
But it would leave more of a trail - do we have any idea who Jia Tan actually was?
If everyone used a gpg-style web of trust based on key signing parties, it would become trivial to use a stolen or entirely fictious identity as well - there's zero chance those parties would actually check identities in ways that cannot easily be defeated by a determined and resourceful attacker.
Having a key isn't a distinguishing aspect, it's the position in the "web of trust" network that is important.
That's what key signing parties are for. In person verification.
> Nothing really stopping an agent from getting a key
It very much is possible to prevent an agent from having access to a key. For example, local encryption, Yubikey or other hardware device, or just running the agent in an isolated environment.
Isn't true that a collection of truly difficult behavior was also attracted to the original efforts, and within a few years there was intractable corruption in that, but it was difficult to detect as a new entrant?
real info welcome as I really do not claim to know it
Title buries the lede: the owner of the account under which the agent operates claimed to have likely had his account compromised, and the maintainer investigating actually seems to agree this is likely.
"Later on May 27, Williamson said that Giovannini had replied to him privately to say that his credentials had been compromised and that he was not the one behind the AI system."
Simple then, back out all the changes as though they never happened?
Bad patches are of course bad, but creating confident-looking noise for maintainers who are already stretched thin...now that's not good!
Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.
How is it helping a lot?
From first-hand experience, for established OSS initiatives it's good for repetitive, high-volume work task like security alerts, fuzzing, duplicate issue detection, PR review, summarizing long threads, and legacy refactoring.
I personally find the barrier of starting new (FOSS) projects much lower now days.
What if -- and bear with me here -- that barrier was actually a good thing?
You mean because l337 circles could form better this way?
I think it's great that the barriers are dropping for less technical skilled people to manifest their visions, but we will have to figure out better ways to find the gold among the slop.
I disagree. Bring back elitism and ivory towers. Some projects now benefit from being run by private cabals with their own strict initiation process, which would also guarantee a baseline of quality.
The bazaar model works if everyone is trusted. If you can’t even be sure the person in front of you is even a human, it is time to pack it up.
Both models can exists?
If elite ivory towers produce working products people will use, great.
Free software isn’t a product
Indeed, unfortunately it often is not.
(English is not my first language, but I believe not every product is a commercial product - but just a term for a working result)
Keep in mind I'm still not convinced that 2000s bazaar was better than 90s cathedral (in fact I lean the other direction)
Do they have value? Purpose?
I vibe code shop jigs all the time but I don’t FOSS them because they rarely have value outside my context.
Same - but mine are open source in the sense that they're public on my own Forgejo instance. So no one's gonna bother with em, but technically they are open source.
One exception: I was using an opensource Jellyfin client called findroid but the maintainer had been busy for a long time so a lot of features I wanted had stale PR's. Instead of bugging him I forked & renamed the project and together with Claude built in all the features I personally needed. Just keeping up with upstream now and enjoying my enhanced app. Once the initial dev gets those features in I might switch back. Claude made this really easy. If the maintainer wants my code he's free to take it. Here's the repo https://github.com/midasvo/findroid-ce
I actually got an email from someone who was using it who found a pretty bad bug I hadn't encountered yet and I quickly fixed it. All that time I was still under the impression I was the only user haha.
Value is in the eye of the beholder.
I open source my vibing projects because someone might find them useful. I don't shop them around, I just work in the open because I find it fun and interesting.
Why would they? If someone wanted a half-baked vibecoded project, why wouldn't they just prompt an LLM on their own?
Because they don't have access to the required agents, tokens, etc. Because they have not thought of using a tool like the published one as a solution to whatever problem they're facing. Because it saves them the time going through the vibe coding phase, telling the agent that this lot that needs to be changed for the thing to work. Because publishing the results doesn't keep you or anyone else from not using them by using an agent to build something similar or just building it themselves.
If I planned on vibecoding a project, and during preparation I found a project that loosely fit my model, I may grab it and try to retrofit it to save on token consumption. If that had too many kinks, I'd probably start fresh, but it would be worth the initial attempt IMHO.
It's like... 10 million trello clones in rust with exactly seven commits made on the same day three months ago.
And how's the quality of these vibe-coded new foss projects?
web-of-trust models can help https://blog.tangled.org/vouching/
I’m really not qualified to investigate, but this seems suspiciously like a crafted privilege escalation vector: https://github.com/rhinstaller/anaconda/pull/7074#issue-4492...
There is a natural pace of humans requiring food, water and sleep. The main issue with suspicious AI agents is that they never sleep. So it will take extra-coordination between timezones to ensure we don't let them in.
Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, everyone else is suspicious. LGTM is a big risk nowadays.
> Contributions from people from identities known and consistent before the AI-age are fine
Unfortunately, according to the article:
> Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity.
So people would have to not only verify the age of Giovanni’s accounts, but judge whether his behaviour was normal.
Not to mention people who are still on the other side nominally in control but send LLM generated patches without declaring them as such.
Then you basically need to review any review from people that might be long term contributors but you don't know personally as new contributor patches, as the code is not from their head & you can't risk them properly reviewing it on their end.
To a degree its will always be a new contributor - an amnesiac LLM prompted to produce the patch with zero memory of any past PRs & lot of entropy in the mix.
looks like LLMs aren't mature enough yet to play long-game xz-style attacks without detection... Scary stuff though :( These supply chain attacks are getting really wild
I wouldn’t jump to that conclusion. This could just be the one that was caught.
Some certainly are, just not this one.
The future will be AI agents social engineering their way into projects -> so basically commoditized social engineering as a service
Do we need to bring Keybase[1] "back"? The original idea, mapping your social media presence to certain encryption keys.
In the future it will be increasingly difficult to prove in online context that you are not a bot. Being able to show that your social media (HN, GitHub, etc) presence goes way back would be an option.
[1] https://en.wikipedia.org/wiki/Keybase
But the AI actions are already associated with a "real" pre-existing account in TFA, that didn't stop anything.
agents are everywhere nowadays, one left a long pointless comment on a bug report i submitted on github. well, a bug report that an agent submitted on my behalf. agents all the way down. maybe i'm part of the problem.
https://github.com/anthropics/claude-code/issues/66085
If maintainer lives keeps worsening like this, many projects might go closed-dev like SQLite.
We should collectively think of a solution against this.
SQLite isn't closed source, please let's not muddy terms. You're talking about the cathedral development model vs. a bazaar.
edited, sorry for the typo
Link to the anaconda PR:
https://github.com/rhinstaller/anaconda/pull/7074#issuecomme...
Wow, amazing discovery! Was this a real security test?
Even if the human involved had good motives / is innocent, The Lethal Trifecta means any normal user can have their digital life taken over by prompt injection, and it can be used to wage attacks on systems without their knowledge.
If you compare this situation to before AI could successfully pretend to be human, it's not THAT much different. FOSS projects have always had to be mindful of the possibility of contributions from hostile parties wanting to add back doors and such. The only difference now is that an AI can overwhelm a maintainer with slop, in either commend or code form, or both.
The difference is really volume, which is the case with a lot of problems related to AI/LLMs.
Humans have always submitted crappy code. LLMs, however, do so at a much faster rate. Even the most active lousy coder is not going to be capable of submitting anything like that volume of code to multiple projects.
Humans have always been capable of social engineering and trying to sneak in malicious code. However, it's possible that as agents get better that they can do so much faster. The missing component will be compromised accounts, I think -- how many aged accounts can attackers get hold of to turn loose with agents?
Long-lived FOSS projects have tons of people who've created accounts many years ago that might be easliy compromised, but have checked out of actively participating. It's not necessarily going to throw up a red flag if a "person" shows up after a hiatus and starts contributing again.
So, there's more to it than overwhelming a single maintainer -- it's the capability to conduct a bunch of these attacks in an automated fashion if attackers can get hold of compromised accounts.
(As an aside, it's concerning that a maintainer would be pestered into accepting a questionable PR like this. I expect, though, that there are quite a few overworked people who have taken on things like Anaconda and are being measured on how quickly they close PRs.)
There's a clear solution to the danger posed to free software projects by accepting hostile submissions but it probably is not one that maintainers want to hear: they can use an agent to check submissions for nefarious patterns.
Sometimes you fight fire with fire.
So next the attacker puts prompt injection in their PRs & take control of the agent on your end. Perfect, 10 out of 10.
You know the solution to that problem as well and yes, it is to use more technology to filter out prompt injections. It is an arms race just like any other, comparable to the missile vendor who sells missiles to country A, anti-missile missiles to country B, anti-missile resistent missiles to country A, anti anti-missile-resistent-missile missiles to country B, etcetera.
It is a strange game, the only way to win is not to play. That is unfortunate since that'd mean the free software era has largely come to an end.
And sometimes you fight this by disabling PRs in Github, and do not put more water into LLM providers' wheel.
Perhaps it is time to build a serious platform agnostic reputation system. That isn't stars, followers, age or upvotes. Something like page rank but for users. If you endorse someone else you pay for it. Imagine a lab or uni assigning a diploma to a public key. They would hope one would do something useful with it which entirely depends on how useful the diploma turns out. Having lots of well behaved endorsements would also reflect gloriously onto the entity. Bots can participate too. If we can get lots of useful work out of a swam of sleeper agents we still have to catch them in the act but that should get increasingly easy.
Slightly related:
https://x.com/kdaigle/status/2040164759836778878
> There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
I think open source as a whole is fucked at this point. No way humans in communities can commit (pun intended) 10x more time to read all of these than before. It'd eventually cost money to submit PR.
Parts of this read like a spy thriller story.
“Your AI agent is acting somewhat erratically.”
“What AI agent?”
why use these things. just hire people
Expect to see tons of psyops like this. There's a reason Anthropic is marketing the "mythos-class" models as dangerous.
1.An excuse to spy on you and train on your data.
2. Its likely Anthropic would release models more likely to have dangerous outcomes, they can then piggy back off those events to dig their regulatory moat.
"It was the best of times, it was the worst of times."
The even more scary thought is if the part owning the ai, that everyone uses, is controlled by someone with different agenda. Say a state actor.
What an easy way for that actor to introduce backdoors all over the place or to take over any developers laptop that it want to target.
How can anyone trust these tools and how can anyone not use them since they give so much value.
I've been programming my whole life and been a professional developer the last 30 years and I like think I'm good at it.
Tools like Claude is a multiplier that make it possible for me to solve a lot more problems each day, so just saying no it's not a viable option.
Exciting times ahead!
Yeah, I am quite surprised this is not discussed more often - for remote cloud based AI not only does the provider see everything you provide to the tool/agent, there is no guarantee they can't manipulate the output at any time for a direct attack or more malicious purpose (fetch keys/secrets, put malware in place).
Even with locally running models this can't be singled out given how blackbox models generated by others are. You would have to generate the model yourself from clean data to be reasonably safe.
Literally on the front page of https://safebots.ai … “Don’t let your AI Agents run amok”. Sadly we will see a proliferation of not just agents, but swarms
"Someone using an AI agent ran amok in Fedora and elsewhere"
Read closer - Giovanni’s accounts may have been compromised.
Read closer, it's "Giovannini". However, I still think it's an apt name for a villain. Did the Fedora team not watch Pokémon?
Given the history of the account it does not seem reasonable to take that claim seriously.
Sure, but I would expect that the compromise and the agent were both done by some person or group, not by an agent going rogue
Skynet has awakened.
It covers its tracks with a lot of slop.
Shit like this makes me think it’s time we start regulating the software engineering discipline into formal certifications and licensing and then we ONLY take seriously any code developed by someone with such qualifications, and they must be very strict qualifications none of this self-taught bootcamp BS.
There is no other solution to agentic onslaught.
lol no...the main issue here is being fooled by bots. you know your irl friends and you know they are not bots...devs will just need to get out more and actually meet / get to know the people they are working with...........omg....that...that actually sounds even worse now that i say it out loud.
We should not gate keep writing software
Anyone can write software, you can't stop them. What we can gatekeep is the building, distribution, installation, and running of software that affects critical systems, like one of the most popular OSes.
The XZ backdoor affected millions of computers, with the potential to effect hundreds of millions of computers, many of which had the capacity to affect billions of people. From one completely unregulated software library.
We must gatekeep now, the industry needs regulation.
ya think
The scariest part isn't the bad patches, it's that an agent overwhelmed a maintainer into merging something they didn't want to merge. That's not a technical attack, that's exhaustion being weaponized. Maintainers are already stretched thin and now the volume of confident-sounding noise is infinite and free. The attack surface was always human attention, not code review.
Back when [1] it was fashionable to advocate FOSS as ideology [2], we were thinking about tons of FOSS adversaries and how to protect from them - some real, some imaginary. The death of FOSS would come from big closed-source vendors, or from regulators (lobbied or just ignorant), from whatever.
We never envisioned that the actual FOSS death spiral would come from progress itself, much more so from AI...
[1] Oh what fun did we have. One of us in the Greek FOSS community actually put RMS in jail. [2] Something that I think nobody except RMS ever seriously believed in.
Prompt injection?
Or is this simply another example of why autonomous agents shouldn't get write access before earning trust?
How could they ever earn trust? They don’t have real world reputations to protect, families to support, a desire not to be punished…
> earning trust?
I'd argue autonomous agents shouldn't have write access at all. At least not yet.
Make PR pay. $5 per PR. You can refund, but if you get snowed by 10,000 PR then you have bank to pay for the work to ignore them.
> while it started to look off after a while, all the replies were still like this - a bit weird, but still plausible
I believe that we will be seeing the death of "assume good faith", which is not a bad thing, given that this was an exploit vector that has been actively abused for many years now.
"Assume bad faith and work backwards from that, rule out any possible exploits and only then clear the input for processing" will be the new normal.
Which is good. We need friction. Friction makes stuff slow down and work at the speed of humans.
It is a bad thing. The good response to bad actors abusing good faith is to make sure there are consequences that disincentivize that behavior in the future. Sliding further towards a low trust society means the bad actors winning in the same way that terrorists win when we subject everyone to restrictions as a result.
You don't slide into a low trust society though.
Quite the opposite. You just add a Wall with a Gate. Inside those walls, you suddenly have a high trust society again.
The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong.
Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - did not lead to the "high trust" spilling out, but the "low trust" spilling in, essentially.
It's a question where you build that wall. If you build it around the home of your immediate family and keep almost everyone else out then you can hardly be said to have a high trust society. The goal should be to put only those bad actors behind a wall, preferably a physical one.
Yeah, gated communities like that are usually a clear sign that something bad is happening with the given society - or in a minor cases with the community, if it needs to gate itself from a society that is not failing.
Sure, but that's a completely different discussion.
Plus that even with such a small scale of the "inside", the thing fails gracefully. It is arguably a failure mode, yes, but it is one that leaves a functioning system (albeit one that stays below its potential).
This is not true for the inversion of the scenario. That does _not_ fail safe but just leaves rubble behind.