> It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
Now, ask yourself if you want AI taking decisions around your life... it will pull numbers... someone will have a "goal" of 1000 "hits" and there you go, 2-3 months in jail working to check what went wrong...
But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...
Important context is that the context wasn’t even in a connected system. Unclear if the human analysts would have even caught this, or if they would have known to check this unconnected system.
> Some US intelligence analysts took notice, the officials said. Bloomberg reported in June that one analyst detected changes to the site as early as 2019, and logged remarks about the modifications in a system that wasn’t connected to the primary military intelligence database, which informs targeting
"In the final days of February, the Trump administration demanded an overwhelming aerial assault. More than 1,000 Iranian targets were hit within the first 24 hours alone. That order compressed the time frame for finalizing targets, the officials said, instilling an urgency that — combined with gaps in underlying information about the site, cuts in civilian-protection personnel and an overreliance on artificial-intelligence technology — set the stage for the deadly error."
-------------------
Maven is an untested tool for rapidly selecting targets under time pressure during a war. It's reason for existing is speed. If you're in a shooting war, the ability to select good military targets faster than your enemy, but not necessarily with zero error, is an advantage.
Planning a first strike is a different situation. Time pressure should not be a significant factor. Humans should have been doing their due diligence on every single target. If an untested tool for rapid mid-battle target selection was relied upon completely because of a hasty executive order, then the responsibility falls entirely on the executive.
Except laundering our biases & discriminations, the unaccountability seems to be the purpose of a huge huge swarth of AI usage. To eliminate accountability. That sure seems like A Feature Not A Bug to the worlds worst people.
> “No more politically correct and overbearing rules of engagement,” - Hegseth
...
> Hegseth and the commander in charge of the Iran operation, Adm. Brad Cooper, were questioned about the strike before Congress. Hegseth said the U.S. does not target civilians and would investigate
So the administration does not intentionally target civilians, they have just stopped caring about who a civilian is.
Well Israel has defined "combatant" as "male between 14 and 55", and the Western world more or less shrugged and moved on without missing a beat. I'm not optimistic.
You might be confusing the approach for estimating combatant casualties after the fact. There were arguments made regarding the likelihood of military age casualties to actually have been combatants. That is not relevant. As far as who is considered a combatant during combat operations Israel follows international law.
EDIT: there is a lot of complexity here because Article 4 applies to soldiers (that e.g. fight in uniform) and not to organizations like Hamas and the PIJ that are considered "unlawful combatants" by Israel and fight in civilian clothing. But simply being a male of military age does not make one a combatant, lawful or legal, by Israel.
Instead of worrying about woke things like "is this person actually a combatant actively fighting" they want the much simpler rule of "everyone we kill was actually a terrorist that deserves to die".
Basically they stopped caring those woke UN rules about war crimes.
While I think that it's important to do root cause analysis, we need to be very careful not to absolve blame from the people that decided to shift a significant weight of decision-making authority to an AI. An AI can't be tried in a court. There has to be a responsible human for everyone one of its actions, especially when it's killing (-people-) children.
> The Pentagon chief last year slashed offices that didn’t contribute to his goal of “lethality,” including the group that assists in limiting risk to civilians, known as the Civilian Protection Center of Excellence. Around 200 employees who worked on the issue, including at that office, have been reduced by about 90 percent, according to two current and former officials and a person familiar with the effort. The team that handles civilian casualties at Central Command, which oversees the Middle East, has dropped from 10 to one.
Negligent homicide in pursuit of their DEI goals, basically.
> Officials said some users expected Maven to flag stale records or contradictions in the intelligence assembled for potential targets, though it is not clear why they thought the system would do that
This is what happens when people who don't understand AI and its limitations embrace it wholeheartedly, thinking it's going to behave like The ultimate analyst. Except it's not. Its got blind spots in more places than anyone knows. And not to mention the system itself has multiple areas of failure.
Chatbots in general can be quite opaque in how they operate, the biggest issue with agent products is communicating it's capabilities and limitations to users. If you have a bank app it's pretty clear whether or not it can send wire transfers, but a if that's a chatbot, users will have to think to ask it (and even then it could hallucinate)
No, this is what happens when software developers don't understand how users actually use their products.
This was a totally expected outcome, similar to how CV cameras arrest the wrong minorities in neighboring states 100s of miles away because it told the officer it was them; or how health insurance companies use ML algorithms to deny Americans care.
It's going to happen in other sectors of life and it will continue killing innocent civilians.
The pentagon says it was Palantir software’s fault, Palantir says it was the fault of bad data input to their software. Where’s the accountability by both organizations? How did we get to a world where military decisions leading to innocent deaths are responded to as if it was just another miscommunication implementing a B2B SaaS platform.
Small children often develop coping mechanisms for their lack of agency. It may be part of these people’s worldview since early childhood that anyone who opposes them should magically die.
you can also develop pro-social coping mechanisms like open-mindedly talking to someone who opposes you and trying to understand their worldview because you trust that it might expand your own
but, yeah, I doubt pro-social traits are common at the top of any hyperstrict hierarchical organization like the USAF
The new world is complex and AI technology has outpaced the regulation cycle, so it's inevitable that we come to a place where you don't have accountability and can never have it because 0) the courts can't handle the case, they don't have the frameworks for it, so it takes longer time 1) you can overload the courts with slop, 2) by the time there is a decision the world has moved on and technology has moved on
I disagree with this framing. Nothing in accountability has really changed other than our willingness to enforce it.
Replace "AI" with any other tool or software in your scenario, and the outcome is the same. The same people and organizations are accountable, it's just that we currently live in a world without accountability in the highest places where it's most needed, and that's unrelated to AI.
You disagree that there is no difference between the amount of material you can generate today compared to 10 years ago?
Or with that the EU AI Act has been redundant and postponed in 2023 because it was drafted on ResNets, and then redundant and postponed in 2025 because it was drafted on ChatGPT 3.5, and then this year probably again because it didn't have agents included?
Or with that other countries that getting their companies hacked by AI models are ignoring any kind of prosecution of said companies and delegating this to the US?
> The Minab site, which was cataloged as an Islamic Revolutionary Guard Corps facility due to outdated data, was fed into Maven with other candidates and came out as a recommended day-one target. Target-list work that once took hours was condensed into minutes.
I feel like we're optimizing the wrong metric here.
Even ignoring the risk of civilian casualties (which to be clear should be weighed heavily) it’s just bad ROI. Like each missile probably costs more than the extra targeting work they should have done.
Yeah, totally, I’m being cautious because I don’t know how large the team is that would spend a few more hours, but under most assumptions you’re right.
Israel likes to argue that Hamas embeds fighters with first responders, which even if so, they're still responding to injured. But Israel considers this not even collateral damage but justified.
Mind you Israel is currently funding a lobbyist organization in the UK to flag Médecins Sans Frontières (Doctors Without Borders) as a terrorist group (because many years ago a couple of people who had affiliations with other anti-Israel groups were found to be also affiliated with MSF).
Yeah, I've heard about it, but presumable Americans soldiers still care about IHL at least to some degree, even though they've historically not bothered with the Geneva convention and have tortured people.
Hamas doesn't issue uniforms to it's fighters besides select units for show.
Every Hamas fighter is in a superposition that collapses based on the perspective of the observer.
Also, in Afghanistan the Taliban routinely used ambulances for troop transport. I have no idea why people think terrorist organizations hold some kind of moral compass that keeps them playing fair. They are awful morally unhinged zealots. If you don't think terrorists will punch below the belt because "that would be evil", you are hopelessly lost in the sauce.
These are the same people who view women as property, homosexuality as a reason for execution, and 13 years olds as viable brides.
...but yeah, they are totally above using aid workers as cover. Definitely not crossing that line. No way.
Preemptively, I too would rather change the topic to Israel than try and defend that.
It's easy to blame the military and the higher-ups, after all they made this choice. But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
> A Palantir spokesperson told Bloomberg the company “is not responsible for the underlying data nor identifying intelligence deficiencies” and that there is no evidence its software was at fault.
If you work for Palantir, you are writing software that kills people. Software that, in this case, killed 123 children. What salary is worth that?
I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth [0]. I think about it nearly every day.
>It's easy to blame the military and the higher-ups
My brother in Cthulu, they're the ones making the life or death decision that killed people, who else should be to blame?
>But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
If you wanna go the historical tech-tree scapegoat route then none of this would have happened if DARPA wouldn't have invented the internet, or if Thomas Edison wouldn't have invented the lightbulb, or if the fish haven't ventured onto land and evolve into us land mammal apes who invented sharpened sticks as murder objects.
Look, if Palantir had any redeeming qualities then I might agree with you. Unlike electricity which runs our computers, homes, cars, etc. What does Palantir do? It spies on people and helps governments to create "people we don't like" lists. In the absolute best case it's a "minority report" engine, but it's still just there to try and stop pre-crime (not crime).
This isn't the case where we are talking about a company doing anything socially good. It's simply there to help authoritarian governments.
Palantir only exists because government wants and funds these types of tools. If it Palantir wouldn't exist then another dozen companies with different names would exist that would create the same Torment Nexus for the government due to inexhaustible demand for such tools.
>It's simply there to help authoritarian governments.
The fact that you're so close yet so far away to figuring out who's to blame still boggles my mind.
I know, right? I'm not even sure if this is the biggest installation on Earth. If it's possible for us, it's possible for any number of private military contractors. I bet there's a similar capability in the hands of every advanced county on Earth. Anyway, shaddup, the focusing crystal's not calibrating right, and tomorrow I have present the targeting parameters to Dr. Evil.
> I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth.
I know a guy who works for well-known hardware company. He isn't a software dev, but close enough, works with Verilog on hardware design and verification. One day he tells me how he is working on video processing module and he doesn't know if that module will be part of surveillance system, missile guidance or nanny cam. It's possible that even middle management doesn't know more than him, and very likely the module will find uses in multiple products. He is a small cogs in a large machine, maybe his work will save a baby's life, maybe it will kill a person, he cannot know.
Of course it could have happened. You don't even need targeting for this to happen. You can drop bombs at random and kill whoever is unlucky enough to be in the wrong place. The US has a long history of doing it (Japan, Vietnam, ...).
So yeah, the military and whoever decided that a war was a good idea are to blame.
People working at Palantir are as much responsible for the killing of these 123 children than seatbelt manufacturers are responsible for the people who died because their product failed. That is, because they didn't do their job correctly, people died, definitely worth feeling bad about. However, should people stop working in seatbelt manufacturing because of it? Absolutely not, we need seatbelts, because without them, things would be worse.
Palantir, or whoever does target identification, is like the seatbelt of the reckless drivers of the militaries, who, for the lack of precise information, would just carpet bomb the place. It is not a hypothetical, it is how the US has worked before they had precision ammunition.
This is essentially a trolley problem. The war is the trolley, people who work for Palantir are the ones pulling levers. And if there is no one to pull the lever, the trolley is gong on a track with a lot of people on it.
The govt is investing in the military whether you like it or not, there is a set amount of GDP going into the sector regardless of your moral choice.
Would you rather not have someone make good software that does not misidentify schools at sites? Or should we outsource this to worse engineers who don't care about morals and don't find this metric meaningful?
That's a reasonable argument, but I don't think I could make the decision to work on software that's designed to kill people. I consider myself a pretty good software developer, and yes, I could probably do a better job building software to help kill people than many others, but I don't think "if you don't do it, someone less capable will, and more innocent people will get killed" would sway me. I just don't want to be involved in that, and I'm lucky that I don't have to be.
Well you made a personal choice to stick your head in the sand to not have to deal with the hard emotions, and that's fine. But to then pass judgement on people who don't sounds a bit unfair given this.
It depends. Did you write a bug that caused the wrong drug to get ordered? Did you design or implement an algorithm that recommends drugs, and the algorithm was a bad one? Did you design or implement a confusing UI flow that led to the doctor ordering the wrong drug?
If so, I'd say yes, you share some of the responsibility. Legally, I'm sure you'd be fine, but if that were me, I'd feel responsible.
The doctor of course is responsible too; they're the one who didn't double-check that it was an appropriate drug. But multiple people can share responsibility for a bad thing.
Whether you may/should feel responsible depends on how the software works, I guess. Did it automatically recommend some drug, somewhat confidently, with gray text on gray background saying "AI sometimes makes mistakes"?
So I think we should behave responsibly when making software, even more now with AI
Let's not forget the shareholders who invested in this company knowing what they do. That includes some of my colleagues. I told them that whoever created and named this company is clearly a psycho.
Meanwhile, the Houthi's are trying to build missile technology with AI. We're only seeing the beginning of the consequences of an over-reliance on AI for military purposes.
If true, there was no verification of the targets selected by AI, meaning AI is the final arbiter of the deadly use of force. Another Rubicon passed.
Trump pretends like it didn’t happen: “We might never know what really happened there” is a frustrating lack of responsibility.
Still missing, as it always is from all of these kinds of articles, is: How common is this error rate compared to pure human evaluation and decision-making? I’m not saying I support AI-assisted killing, but why is nobody comparing it to human failure rates? If the article is true, then there were 1,000 targets hit in the first 24 hours. If the majority of the legwork and selection was done by AI, how many schools, or equivalents, would one expect to be wrongly hit if all the targeting work were done by humans only?
ultimately someone or a group of people need to be held criminally accountable for this crime. it is on the magnitude of the my lai massacre and there should be justice for the victims
There is no justice. And Iran will probably blow up some innocent civillians in response the opportunity arrives, instead of targeting the actual culprits.
I often see the argument "humans make mistakes too" when talking about AI, self driving cars, autonomous drones and such...
yes, they do, but most of the time they go to trial, there are consequences, there might even be justice.
when AI/self driving cars/drones make mistakes, which might very well be more rare in absolute [citation needed], there is (seems to me) a tendency to shrug it off, almost as if it were force majeure.
Never so happy to see such an obvious scapegoat get thrown under the bus. I don't buy it for a second, but no tears will be shed if the stock market does.
first, closer to 250 people were murdered by the $4 Million tomahawk
second, imagine if Iran even accidentally took out an elementary school in the US filled with 250 people
third, there was a human group that was supposed to review targets regardless of whomever picked them, the group was purposely disbanded, that's negligent homicide at best but more clearly a war crime on purpose
fourth, let's say it was 250 soldiers somehow among a civilian setting, IT CHANGED NOTHING, this war will change nothing, they cannot destroy pickaxe mountain, they will abandon the war in 2029 like the US abandoned Iraq, Afghanistan, Vietnam and Korea, WAR CHANGES NOTHING, you cannot "War to Peace"
nice, Karp really reeled in a nice big fish, and ofcourse he is going to blame OpenAI/Anthropic for giving the targets, this explains his CNBC rants about the frontiers, his tool was the trigger in a killchain of hallucination.
Completely disagree. When we’re talking about something like air, missile, or artillery strikes, the operator of the weapon system may literally never even see the target, on a screen or otherwise. They may simply be entering targeting information provided to them by an operations center. These decisions are social and group decisions, with distributed responsibility, which is why it’s so important to have institutions like the Civilian Protection Center of Excellence which historically has identified and helped implement the policies which prevent these kinds of disasters, but which unfortunately the current administration decided to shut down without a replacement.
Sometimes, the difference between a successful strike on a lawful combatant and disaster is one analyst pointing out that the building the enemy is hiding in is next to a square that’s used for a market on Tuesdays and wouldn’t you know it, it’s Tuesday. That’s not something the person “pulling the trigger,” can always be expected to know, which is why every ops center needs to be designed to elevate those voices, when the time comes
The one that pulls the trigger is often not the one doing any targeting, especially on a strategic strike weapon like a tomahawk. They are receiving a target data package with preprogrammed instructions. Their role is far more... mechanical. They're not seeing the targets at all.
I would argue `s/pulls the trigger/gives the go ahead/` is intuitively more appropriate. The sailor on the boat that fired off the tomahawk (iirc) was likely not in a position to make an in-the-moment determination
They are also openly and proudly using this system along with Starlink to assist Ukraine, who a couple months ago killed a bunch of teens in a college in East Ukraine under very similar circumstances that a simple Google or Yandex maps search would have red-flagged plus on the ground real human intel. Unless the point was actually to kill kids.
Okay, that one is fake because Russia bad. Iran one is real because Orange man bad. Same ISR and targeting systems. Mental gymnastics is a hell of thing.
Thank you for reminding me about these posts, time has proved my statements true. Especially after the Iran war exposed these outdated systems via direct hits on US bases, which can't be narrative'd away like in Ukraine.
The S-400 is fairing well? F-35s apparently walked all over Iran's air defense network without much challenge.
> Especially after the Iran war exposed these outdated systems via direct hits on US bases…
Interception rates have been extremely high. The problem, of course, is the ammo shortage and drone cost asymmetry. Which has been widely documented, and presents an issue for the S-400s too.
Is that really the case? Stories like this always have all sides muddying waters.
The terrorists are well known to use children and the rest of the public as hostage meatshields, and the US military is always looking for something/someone else to blame for the dirty work they have to do.
The broader public is so polarized by performative politics and botspam that they eat these stories up, but nothing I'm saying is secret. It's been this way since forever. Slow news today?
I'm not even defending AI at all and think these are crappy choices, but compared to humans gathering intelligence who can go rogue, what alternative does anyone really have?
What fucking terrorists? We're talking about a country bombing another country's school. A country with a population of 93 million. Are you trying to insinuate that all 93 million Iranians are terrorists?
Maybe you meant the American terrorists?
> Stories like this always have all sides muddying waters.
All sides agree that the US bombed a school full of children, actually.
DOGE strikes again. Edit actually looks like it was Hegseth himself, king Testosterone.
> Staffing on civilian harm mitigation teams across the U.S. Department of Defense had also fallen by roughly 90% over recent years, shrinking to fewer than 20 people overall. Centcom’s group shrank from 10 people to one.
> It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
Now, ask yourself if you want AI taking decisions around your life... it will pull numbers... someone will have a "goal" of 1000 "hits" and there you go, 2-3 months in jail working to check what went wrong...
But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...
Important context is that the context wasn’t even in a connected system. Unclear if the human analysts would have even caught this, or if they would have known to check this unconnected system.
> Some US intelligence analysts took notice, the officials said. Bloomberg reported in June that one analyst detected changes to the site as early as 2019, and logged remarks about the modifications in a system that wasn’t connected to the primary military intelligence database, which informs targeting
"In the final days of February, the Trump administration demanded an overwhelming aerial assault. More than 1,000 Iranian targets were hit within the first 24 hours alone. That order compressed the time frame for finalizing targets, the officials said, instilling an urgency that — combined with gaps in underlying information about the site, cuts in civilian-protection personnel and an overreliance on artificial-intelligence technology — set the stage for the deadly error."
-------------------
Maven is an untested tool for rapidly selecting targets under time pressure during a war. It's reason for existing is speed. If you're in a shooting war, the ability to select good military targets faster than your enemy, but not necessarily with zero error, is an advantage.
Planning a first strike is a different situation. Time pressure should not be a significant factor. Humans should have been doing their due diligence on every single target. If an untested tool for rapid mid-battle target selection was relied upon completely because of a hasty executive order, then the responsibility falls entirely on the executive.
Seems like a scenario where two completely avoidable wrongs....killed 120 children. AI being one of the wrongs.
"A computer can never be held accountable / Therefore a computer must must never make a management decision" https://simonwillison.net/2025/Feb/3/a-computer-can-never-be...
Except laundering our biases & discriminations, the unaccountability seems to be the purpose of a huge huge swarth of AI usage. To eliminate accountability. That sure seems like A Feature Not A Bug to the worlds worst people.
https://www.propublica.org/article/hegseth-civilian-harm-mit...
> “No more politically correct and overbearing rules of engagement,” - Hegseth
...
> Hegseth and the commander in charge of the Iran operation, Adm. Brad Cooper, were questioned about the strike before Congress. Hegseth said the U.S. does not target civilians and would investigate
So the administration does not intentionally target civilians, they have just stopped caring about who a civilian is.
Well Israel has defined "combatant" as "male between 14 and 55", and the Western world more or less shrugged and moved on without missing a beat. I'm not optimistic.
This is false. Can you provide a source? As to what Israel actually follows there are details here: https://ihl-databases.icrc.org/en/customary-ihl/v2/rule1?cou...
You might be confusing the approach for estimating combatant casualties after the fact. There were arguments made regarding the likelihood of military age casualties to actually have been combatants. That is not relevant. As far as who is considered a combatant during combat operations Israel follows international law.
EDIT: there is a lot of complexity here because Article 4 applies to soldiers (that e.g. fight in uniform) and not to organizations like Hamas and the PIJ that are considered "unlawful combatants" by Israel and fight in civilian clothing. But simply being a male of military age does not make one a combatant, lawful or legal, by Israel.
The accepted answer here has a decent source:
https://skeptics.stackexchange.com/questions/56556/does-the-...
That, unfortunately, originates more from the US than Israel.
https://en.wikipedia.org/wiki/Military-age_male
Not sure wtf “politically correct” in the context of rules of engagement even means.
What a fucking moron
Instead of worrying about woke things like "is this person actually a combatant actively fighting" they want the much simpler rule of "everyone we kill was actually a terrorist that deserves to die".
Basically they stopped caring those woke UN rules about war crimes.
While I think that it's important to do root cause analysis, we need to be very careful not to absolve blame from the people that decided to shift a significant weight of decision-making authority to an AI. An AI can't be tried in a court. There has to be a responsible human for everyone one of its actions, especially when it's killing (-people-) children.
I cannot believe this has come to pass.
https://www.politico.com/news/2026/03/10/pentagon-iran-schoo...
> The Pentagon chief last year slashed offices that didn’t contribute to his goal of “lethality,” including the group that assists in limiting risk to civilians, known as the Civilian Protection Center of Excellence. Around 200 employees who worked on the issue, including at that office, have been reduced by about 90 percent, according to two current and former officials and a person familiar with the effort. The team that handles civilian casualties at Central Command, which oversees the Middle East, has dropped from 10 to one.
Negligent homicide in pursuit of their DEI goals, basically.
Do you just chuck that on the end of any sentence, because this makes less sense than an LLM.
People were killed, i.e. homicide.
Negligence was involved.
DEI for drunk white men who look good on TV got us Hegseth, who's now rather hilariously requiring gender-affirming treatment for the troops. (https://www.pbs.org/newshour/world/pentagon-moves-forward-wi...)
The headline says "contributed to missile strike", not caused it.
can -> can't
ty, fixed
https://www.merriam-webster.com/grammar/cannot-vs-can-not-is...
He's referring to this sentence:
> An AI can be tried in a court.
An AI can't be tried in a court.
> Officials said some users expected Maven to flag stale records or contradictions in the intelligence assembled for potential targets, though it is not clear why they thought the system would do that
This is what happens when people who don't understand AI and its limitations embrace it wholeheartedly, thinking it's going to behave like The ultimate analyst. Except it's not. Its got blind spots in more places than anyone knows. And not to mention the system itself has multiple areas of failure.
Chatbots in general can be quite opaque in how they operate, the biggest issue with agent products is communicating it's capabilities and limitations to users. If you have a bank app it's pretty clear whether or not it can send wire transfers, but a if that's a chatbot, users will have to think to ask it (and even then it could hallucinate)
No, this is what happens when software developers don't understand how users actually use their products.
This was a totally expected outcome, similar to how CV cameras arrest the wrong minorities in neighboring states 100s of miles away because it told the officer it was them; or how health insurance companies use ML algorithms to deny Americans care.
It's going to happen in other sectors of life and it will continue killing innocent civilians.
The pentagon says it was Palantir software’s fault, Palantir says it was the fault of bad data input to their software. Where’s the accountability by both organizations? How did we get to a world where military decisions leading to innocent deaths are responded to as if it was just another miscommunication implementing a B2B SaaS platform.
Small children often develop coping mechanisms for their lack of agency. It may be part of these people’s worldview since early childhood that anyone who opposes them should magically die.
*anti-social coping mechanisms
you can also develop pro-social coping mechanisms like open-mindedly talking to someone who opposes you and trying to understand their worldview because you trust that it might expand your own
but, yeah, I doubt pro-social traits are common at the top of any hyperstrict hierarchical organization like the USAF
To me it's extremely obvious that this is the desired outcome - no accountability for anyone with this setup.
The new world is complex and AI technology has outpaced the regulation cycle, so it's inevitable that we come to a place where you don't have accountability and can never have it because 0) the courts can't handle the case, they don't have the frameworks for it, so it takes longer time 1) you can overload the courts with slop, 2) by the time there is a decision the world has moved on and technology has moved on
Really good time to do various crime.
I disagree with this framing. Nothing in accountability has really changed other than our willingness to enforce it.
Replace "AI" with any other tool or software in your scenario, and the outcome is the same. The same people and organizations are accountable, it's just that we currently live in a world without accountability in the highest places where it's most needed, and that's unrelated to AI.
You disagree that there is no difference between the amount of material you can generate today compared to 10 years ago?
Or with that the EU AI Act has been redundant and postponed in 2023 because it was drafted on ResNets, and then redundant and postponed in 2025 because it was drafted on ChatGPT 3.5, and then this year probably again because it didn't have agents included?
Or with that other countries that getting their companies hacked by AI models are ignoring any kind of prosecution of said companies and delegating this to the US?
Yeah, all this was going on decades before AI.
> The Minab site, which was cataloged as an Islamic Revolutionary Guard Corps facility due to outdated data, was fed into Maven with other candidates and came out as a recommended day-one target. Target-list work that once took hours was condensed into minutes.
I feel like we're optimizing the wrong metric here.
Even ignoring the risk of civilian casualties (which to be clear should be weighed heavily) it’s just bad ROI. Like each missile probably costs more than the extra targeting work they should have done.
probably -> definitely
These missiles cost a couple million each.
Yeah, totally, I’m being cautious because I don’t know how large the team is that would spend a few more hours, but under most assumptions you’re right.
Related: the U.S. nearly boarded a Chinese boat that AI incorrectly flagged as carrying nuclear weapons materiel.
https://gizmodo.com/almost-started-a-war-us-military-nearly-...
This is actually a much bigger close call than the hugging face incident. A hallucination almost triggered a possible precursor to WWIII.
You're absolutely right, those were innocent children. I apologize for the error. Would you like to know more about war crimes?
Wait... I thought they said they didn't kill children and that was Iranian propaganda.
> That didn't happen.
> And if it did, it wasn't that bad.
> And if it was, it was AI's fault, not mine.
I'll add a check that all individuals within are genuinely adults to avoid this.
C'mon, a Palantir AI isn't gonna talk like Claude.
It's gonna say something like "they were all military-aged males". https://en.wikipedia.org/wiki/Military-age_male
Personally I'm more interested in the justification for the second and third strikes.
There needs to be some protection for first responders and others helping the injured.
Israel likes to argue that Hamas embeds fighters with first responders, which even if so, they're still responding to injured. But Israel considers this not even collateral damage but justified.
Mind you Israel is currently funding a lobbyist organization in the UK to flag Médecins Sans Frontières (Doctors Without Borders) as a terrorist group (because many years ago a couple of people who had affiliations with other anti-Israel groups were found to be also affiliated with MSF).
Yeah, I've heard about it, but presumable Americans soldiers still care about IHL at least to some degree, even though they've historically not bothered with the Geneva convention and have tortured people.
Hamas doesn't issue uniforms to it's fighters besides select units for show. Every Hamas fighter is in a superposition that collapses based on the perspective of the observer.
Also, in Afghanistan the Taliban routinely used ambulances for troop transport. I have no idea why people think terrorist organizations hold some kind of moral compass that keeps them playing fair. They are awful morally unhinged zealots. If you don't think terrorists will punch below the belt because "that would be evil", you are hopelessly lost in the sauce.
These are the same people who view women as property, homosexuality as a reason for execution, and 13 years olds as viable brides.
...but yeah, they are totally above using aid workers as cover. Definitely not crossing that line. No way.
Preemptively, I too would rather change the topic to Israel than try and defend that.
I'm going to bet that absolutely nobody higher up will end up going to jail for this.
Hegseth should be in prison
https://www.propublica.org/article/hegseth-civilian-harm-mit...
Jail? I'm going to bet that absolutely nobody higher up will face any consequences for this. The outcome will be "we will review our processes".
It's easy to blame the military and the higher-ups, after all they made this choice. But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
> A Palantir spokesperson told Bloomberg the company “is not responsible for the underlying data nor identifying intelligence deficiencies” and that there is no evidence its software was at fault.
If you work for Palantir, you are writing software that kills people. Software that, in this case, killed 123 children. What salary is worth that?
I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth [0]. I think about it nearly every day.
[0] https://calebhearth.com/dont-get-distracted
>It's easy to blame the military and the higher-ups
My brother in Cthulu, they're the ones making the life or death decision that killed people, who else should be to blame?
>But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
If you wanna go the historical tech-tree scapegoat route then none of this would have happened if DARPA wouldn't have invented the internet, or if Thomas Edison wouldn't have invented the lightbulb, or if the fish haven't ventured onto land and evolve into us land mammal apes who invented sharpened sticks as murder objects.
Look, if Palantir had any redeeming qualities then I might agree with you. Unlike electricity which runs our computers, homes, cars, etc. What does Palantir do? It spies on people and helps governments to create "people we don't like" lists. In the absolute best case it's a "minority report" engine, but it's still just there to try and stop pre-crime (not crime).
This isn't the case where we are talking about a company doing anything socially good. It's simply there to help authoritarian governments.
Palantir only exists because government wants and funds these types of tools. If it Palantir wouldn't exist then another dozen companies with different names would exist that would create the same Torment Nexus for the government due to inexhaustible demand for such tools.
>It's simply there to help authoritarian governments.
The fact that you're so close yet so far away to figuring out who's to blame still boggles my mind.
I know, right? I'm not even sure if this is the biggest installation on Earth. If it's possible for us, it's possible for any number of private military contractors. I bet there's a similar capability in the hands of every advanced county on Earth. Anyway, shaddup, the focusing crystal's not calibrating right, and tomorrow I have present the targeting parameters to Dr. Evil.
Thank you for sharing the talk.
> I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth.
I know a guy who works for well-known hardware company. He isn't a software dev, but close enough, works with Verilog on hardware design and verification. One day he tells me how he is working on video processing module and he doesn't know if that module will be part of surveillance system, missile guidance or nanny cam. It's possible that even middle management doesn't know more than him, and very likely the module will find uses in multiple products. He is a small cogs in a large machine, maybe his work will save a baby's life, maybe it will kill a person, he cannot know.
Of course it could have happened. You don't even need targeting for this to happen. You can drop bombs at random and kill whoever is unlucky enough to be in the wrong place. The US has a long history of doing it (Japan, Vietnam, ...).
So yeah, the military and whoever decided that a war was a good idea are to blame.
People working at Palantir are as much responsible for the killing of these 123 children than seatbelt manufacturers are responsible for the people who died because their product failed. That is, because they didn't do their job correctly, people died, definitely worth feeling bad about. However, should people stop working in seatbelt manufacturing because of it? Absolutely not, we need seatbelts, because without them, things would be worse.
Palantir, or whoever does target identification, is like the seatbelt of the reckless drivers of the militaries, who, for the lack of precise information, would just carpet bomb the place. It is not a hypothetical, it is how the US has worked before they had precision ammunition.
This is essentially a trolley problem. The war is the trolley, people who work for Palantir are the ones pulling levers. And if there is no one to pull the lever, the trolley is gong on a track with a lot of people on it.
The govt is investing in the military whether you like it or not, there is a set amount of GDP going into the sector regardless of your moral choice.
Would you rather not have someone make good software that does not misidentify schools at sites? Or should we outsource this to worse engineers who don't care about morals and don't find this metric meaningful?
That's a reasonable argument, but I don't think I could make the decision to work on software that's designed to kill people. I consider myself a pretty good software developer, and yes, I could probably do a better job building software to help kill people than many others, but I don't think "if you don't do it, someone less capable will, and more innocent people will get killed" would sway me. I just don't want to be involved in that, and I'm lucky that I don't have to be.
Well you made a personal choice to stick your head in the sand to not have to deal with the hard emotions, and that's fine. But to then pass judgement on people who don't sounds a bit unfair given this.
If I write healthcare software, and a doctor orders the wrong drug that kills the patient, am I responsible?
It depends. Did you write a bug that caused the wrong drug to get ordered? Did you design or implement an algorithm that recommends drugs, and the algorithm was a bad one? Did you design or implement a confusing UI flow that led to the doctor ordering the wrong drug?
If so, I'd say yes, you share some of the responsibility. Legally, I'm sure you'd be fine, but if that were me, I'd feel responsible.
The doctor of course is responsible too; they're the one who didn't double-check that it was an appropriate drug. But multiple people can share responsibility for a bad thing.
Legally, probably not...?
Whether you may/should feel responsible depends on how the software works, I guess. Did it automatically recommend some drug, somewhat confidently, with gray text on gray background saying "AI sometimes makes mistakes"?
So I think we should behave responsibly when making software, even more now with AI
Let's not forget the shareholders who invested in this company knowing what they do. That includes some of my colleagues. I told them that whoever created and named this company is clearly a psycho.
[delayed]
Meanwhile, the Houthi's are trying to build missile technology with AI. We're only seeing the beginning of the consequences of an over-reliance on AI for military purposes.
That is absolutely irrelevant. USA will never take responsibility. Nobody will be prosecuted. No compensations will be paid.
Things that jumped out at me.
If true, there was no verification of the targets selected by AI, meaning AI is the final arbiter of the deadly use of force. Another Rubicon passed.
Trump pretends like it didn’t happen: “We might never know what really happened there” is a frustrating lack of responsibility.
Still missing, as it always is from all of these kinds of articles, is: How common is this error rate compared to pure human evaluation and decision-making? I’m not saying I support AI-assisted killing, but why is nobody comparing it to human failure rates? If the article is true, then there were 1,000 targets hit in the first 24 hours. If the majority of the legwork and selection was done by AI, how many schools, or equivalents, would one expect to be wrongly hit if all the targeting work were done by humans only?
Whole world is turning into an unlivable place and people who are doing it are not taking responsibility.
They never have. Their money insulates them from the consequences of their actions.
There’s a threshold where the money won’t insulate anymore.
Especially if the most well funded companies won't even pay their security a living wage: https://www.kqed.org/news/12100512/bay-area-security-workers...
They'll probably be dead by then. It's nihilistic. After us, the deluge...
Related
Iran school bombing: grounds to believe US was behind atrocity, UN finds
https://news.ycombinator.com/item?id=49739018
ultimately someone or a group of people need to be held criminally accountable for this crime. it is on the magnitude of the my lai massacre and there should be justice for the victims
There is no justice. And Iran will probably blow up some innocent civillians in response the opportunity arrives, instead of targeting the actual culprits.
Title: Inside the AI "Kill Chain" That Destroyed an Iranian School
(https://news.ycombinator.com/item?id=49763739 https://news.ycombinator.com/item?id=49761106)
https://archive.ph/0V37g
A poor craftsman blames his tools.
Responsibility can fall on multiple parties, including both the tool user and the tool builder.
Can't wait to read "Karp in Al-Quds: A Report on the Banality of Evil" in 2030.
so who is paying for this? "humans must always remain accountable for artificial intelligence (AI) decisions"
An Maven AI used Claude, which was denied here multiple times. Maybe call the next model after Fable Minab!
I don't want to see the Pentagon try to pass the blame here, but I am all for giving half the blame to Palantir.
a random musing:
I often see the argument "humans make mistakes too" when talking about AI, self driving cars, autonomous drones and such...
yes, they do, but most of the time they go to trial, there are consequences, there might even be justice.
when AI/self driving cars/drones make mistakes, which might very well be more rare in absolute [citation needed], there is (seems to me) a tendency to shrug it off, almost as if it were force majeure.
is it just me?
Tragically this won't be the last time AI kills a lot of innocent people.
This is not AI killing poeole, this is US government killing poeole.
AI didn’t kill innocent people here, the Department of Defense and its decision-makers did.
Never so happy to see such an obvious scapegoat get thrown under the bus. I don't buy it for a second, but no tears will be shed if the stock market does.
first, closer to 250 people were murdered by the $4 Million tomahawk
second, imagine if Iran even accidentally took out an elementary school in the US filled with 250 people
third, there was a human group that was supposed to review targets regardless of whomever picked them, the group was purposely disbanded, that's negligent homicide at best but more clearly a war crime on purpose
fourth, let's say it was 250 soldiers somehow among a civilian setting, IT CHANGED NOTHING, this war will change nothing, they cannot destroy pickaxe mountain, they will abandon the war in 2029 like the US abandoned Iraq, Afghanistan, Vietnam and Korea, WAR CHANGES NOTHING, you cannot "War to Peace"
time to make Palantir the next Flock
France gave them the boot, you can too
nice, Karp really reeled in a nice big fish, and ofcourse he is going to blame OpenAI/Anthropic for giving the targets, this explains his CNBC rants about the frontiers, his tool was the trigger in a killchain of hallucination.
The one that pulls the trigger bears the responsibility.
No. That's too easy on the higher ups, they never pull triggers.
Completely disagree. When we’re talking about something like air, missile, or artillery strikes, the operator of the weapon system may literally never even see the target, on a screen or otherwise. They may simply be entering targeting information provided to them by an operations center. These decisions are social and group decisions, with distributed responsibility, which is why it’s so important to have institutions like the Civilian Protection Center of Excellence which historically has identified and helped implement the policies which prevent these kinds of disasters, but which unfortunately the current administration decided to shut down without a replacement.
Sometimes, the difference between a successful strike on a lawful combatant and disaster is one analyst pointing out that the building the enemy is hiding in is next to a square that’s used for a market on Tuesdays and wouldn’t you know it, it’s Tuesday. That’s not something the person “pulling the trigger,” can always be expected to know, which is why every ops center needs to be designed to elevate those voices, when the time comes
the operator was following orders...
The one that pulls the trigger is often not the one doing any targeting, especially on a strategic strike weapon like a tomahawk. They are receiving a target data package with preprogrammed instructions. Their role is far more... mechanical. They're not seeing the targets at all.
I would argue `s/pulls the trigger/gives the go ahead/` is intuitively more appropriate. The sailor on the boat that fired off the tomahawk (iirc) was likely not in a position to make an in-the-moment determination
Given that everything emanating from this administration has a political agenda, does this mean that Thiel is on the outs?
> unreleased Pentagon probe
From TFA.
He's rich and connected, of course not.
They are also openly and proudly using this system along with Starlink to assist Ukraine, who a couple months ago killed a bunch of teens in a college in East Ukraine under very similar circumstances that a simple Google or Yandex maps search would have red-flagged plus on the ground real human intel. Unless the point was actually to kill kids.
https://en.wikipedia.org/wiki/2026_Starobilsk_strike
I think your source is undermining your claim. That Wikipedia article strongly suggests that the whole event was fabricated by Russia.
Okay, that one is fake because Russia bad. Iran one is real because Orange man bad. Same ISR and targeting systems. Mental gymnastics is a hell of thing.
Says the Russian Emergency Situations Ministry, which then immediately refused to allow the UN access to verify the claim.
The account in question has a history of being… deeply credulous of Russian claims.
https://news.ycombinator.com/item?id=47441817
https://news.ycombinator.com/item?id=36559017
etc.
Thank you for reminding me about these posts, time has proved my statements true. Especially after the Iran war exposed these outdated systems via direct hits on US bases, which can't be narrative'd away like in Ukraine.
> time has proved my statements true
The S-400 is fairing well? F-35s apparently walked all over Iran's air defense network without much challenge.
> Especially after the Iran war exposed these outdated systems via direct hits on US bases…
Interception rates have been extremely high. The problem, of course, is the ammo shortage and drone cost asymmetry. Which has been widely documented, and presents an issue for the S-400s too.
Is that really the case? Stories like this always have all sides muddying waters.
The terrorists are well known to use children and the rest of the public as hostage meatshields, and the US military is always looking for something/someone else to blame for the dirty work they have to do.
The broader public is so polarized by performative politics and botspam that they eat these stories up, but nothing I'm saying is secret. It's been this way since forever. Slow news today?
I'm not even defending AI at all and think these are crappy choices, but compared to humans gathering intelligence who can go rogue, what alternative does anyone really have?
It was a school. No one on either side is denying this.
> Stories like this always have all sides muddying waters.
Which makes the administration's near complete silence on the strike rather suspicious, no?
https://www.propublica.org/article/hegseth-civilian-harm-mit...
Yes it is really the case here.
> The terrorists
What fucking terrorists? We're talking about a country bombing another country's school. A country with a population of 93 million. Are you trying to insinuate that all 93 million Iranians are terrorists?
Maybe you meant the American terrorists?
> Stories like this always have all sides muddying waters.
All sides agree that the US bombed a school full of children, actually.
In a just world, Pete Hegseth and the entire board of Palantir would be in prison for this.
TIL about this attack.
https://en.wikipedia.org/wiki/2026_Minab_school_attack
This looks as much, if not more, like terrorism to me than other incidents the media endlessly portrays.
The US has done so much evil around the world. Enough is enough.
And yeah, Alex Karp is a complete idiot. I cannot imagine how vile one has to be to make a profit off this.
DOGE strikes again. Edit actually looks like it was Hegseth himself, king Testosterone.
> Staffing on civilian harm mitigation teams across the U.S. Department of Defense had also fallen by roughly 90% over recent years, shrinking to fewer than 20 people overall. Centcom’s group shrank from 10 people to one.
https://www.propublica.org/article/hegseth-civilian-harm-mit...
Edit: hilarious downvotes. Does DOGE get Elon Stan bots or something?
"stupid and energetic, the worst kind of general" - Indy Neidell
Paraphrasing the German General Kurt von Hammerstein-Equord
in reference to a WW1 Austrian General (for others who pass by and are curious, since you seem to be in the know)
https://en.wikipedia.org/wiki/Franz_Conrad_von_H%C3%B6tzendo...
"maximum lethality, not tepid legality."
It was intentional and it's part of the zionist culture.
Depressing, why should my tax dollars fund this incompetence and insanity. We need more accountability for such military decisions
Despite what AI told them, they knew it was a school full of kids. They still chose to kill them all with bomb. Only maniac would do such thing.
> they knew it was a school full of kids.
I would like to know how you know this
I read the early life section.