In the piece Dave Willner and I published last week about why “the apocalypse won’t be sexy,” we discussed how the real threat wasn’t “rogue” out of control AI tools, but rather humans doing stupid shit, sometimes because they stop thinking in the presence of these AI machines. A deeply reported Bloomberg article by Ben Bartenstein and Krishna Karra details how the now infamous bombing of an Iranian girls elementary school was due to Pentagon officials deep over-reliance on Palantir’s Maven system.
Inside Centcom, which conducted the US attack, some personnel relied too much on the artificial intelligence embedded in Maven Smart System, the officials said. Maven, which is made by Palantir Technologies Inc., allows users to view and coordinate complex military operations — from targeting to command and control — using AI-enabled software that fuses more than 150 different data inputs into a coherent set of information to inform commanders’ decisions.
In the past year, the Defense Department has rapidly made Maven a cornerstone of the US military apparatus. Several former senior US military officials now work for Palantir, including some with high-level security clearances in Centcom’s Tampa headquarters. Their role is to keep the software functioning inside secure military environments.
To understand how we got here, you need a timeline that stretches back the better part of a decade. Project Maven was, somewhat famously, a project to use AI in the Pentagon, that Google was originally tapped to build. However, in 2018, due in large part to employee protests, Google announced it would drop the contract. Days later, Google announced that it would not allow its AI to be used for “weaponized systems.”
Around that time, I was asked to be on a panel debating whether or not it was good for employees to protest contracts like this with one of Palantir’s founders. Little did I know that the reason that founder was so vehement in his contempt for the Google employees protesting was because, right at that moment, Palantir was swooping in to grab the contract that Google was dropping.
Secretive data analytics company Palantir is working on Project Maven, the US Department of Defense program that Google dropped in March following protests inside of the company, according to people familiar with the project.
That takes us to earlier this year, when Donald Trump and Pete Hegseth kicked off their disastrous, stupid, and short-sighted war in Iran with an initial bombing campaign that (among other things) blew up a girls elementary school, killing at least 100 girls (later reports put the toll at more than 165, most of them children). While Hegseth and Trump tried to pretend it wasn’t them or that it wasn’t a school, the evidence piled up pretty quickly.
At almost the exact same time, the Pentagon got into its (now deemed unconstitutional) petty spat with Anthropic, which included some debate about whether or not the Defense Department could get around Anthropic’s guardrails against using its AI for autonomous weaponry. Not surprisingly, this resulted in plenty of speculation about whether or not the bombing of the Shajare Tayyiba Elementary School was due to an AI failure. Reports noted that the site used to be a military compound, but satellite imagery by 2017 had showed the outdoor play area, and intelligence analysts should have removed it from any target list. Hell, the school was apparently clearly labeled… on Google Maps.
Days later, Semafor (a publication with a history of platforming credulous takes regarding the Trump admin) insisted it had the scoop: US officials told it that AI had nothing to do with the bombing.
The error was one that AI would not be likely to make: US officials failed to recognize subtle changes in satellite imagery, while human intelligence analysts missed publicly available information about a school located inside the Revolutionary Guard compound (or failed to add it to the database used for targeting). AI has its notorious failings, from hallucinations to sycophancy, but it’s also able to take in far more information than current, human-led systems — and a deeper look at satellite imagery or, simply, an internet search could have forestalled the disaster. Even a scan of Iranian business listings turned up the school, according to Reuters.
That article discusses Palantir’s Maven — calling it “a kind of DoorDash for deadly missile strikes” — but then suggests it wasn’t really involved in this process:
The tragedy at the Shajareh Tayyebeh school led to an immediate rush to blame AI for the fatal missile attack. And while the initial speculation turned out to be wrong, it does suggest that civilian deaths stemming from automation will be viewed differently than those caused by humans, even if autonomous weapons prove safer in the aggregate.
Except, the Bloomberg report suggests, Palantir’s Maven was definitely involved in the process, even if human stupidity (naturally) made the final calls.
Basically, the Bloomberg report details that people at the Pentagon assumed (incorrectly, obviously) that Palantir’s Maven would magically alert them “hey, this might be a school.” But that’s not how it works. Then again, why would anyone expect a system sold as “intelligent” to apply any actual intelligence to vetting potential targets?
Some Centcom personnel expected Maven to flag stale information or inconsistencies in the underlying intelligence assembled for potential targets, the officials said. It’s not clear why they had such expectations.
Yeah, okay, maybe “it’s not clear why” people expected it, but really? It kinda is, isn’t it? Palantir and all the AI companies play up the “smarts” of their systems and it leads people into a false reliance on them as all-knowing, all-seeing beings, rather than just automation machines that can handle fuzzy inputs and outputs better than old deterministic systems.
Of course, the fact that it was humans who over-relied on Maven lets Palantir wash its hands of the whole thing and yell “not our fault” as loudly as it can:
A Palantir spokesperson said that the company “is not responsible for the underlying data nor identifying intelligence deficiencies” and that there’s no evidence that its software was at fault in the Minab strike.
Yes, the human is responsible. But people over-relying on the tech because you oversold it is kinda part of the issue, just as Elon Musk overselling the safety of his “full self driving” feature needs to fall partly on Tesla.
And here, even though there were “humans in the loop,” it seems clear that in their rush to find places to bomb, they were perfectly happy to let Palantir’s Maven do the actual work:
High on Centcom’s list: the naval division of the Islamic Revolutionary Guard Corps, the most powerful force across the country’s military, economy and politics. It’s tasked with defending — and at times, disrupting — the Persian Gulf and Strait of Hormuz. The Minab school site — with its uncorrected classification as an IRGC facility — was fed into Maven along with other potential targets. It ultimately emerged as one of the recommended day-one targets, people involved in the investigation said.
As all targets do, the Minab site moved through a multi-stage assessment process known by the military as the “kill chain.” Steps in that process have traditionally involved intelligence analysts, imagery specialists, targeteers, lawyers, operational commanders and launch crews, according to US military doctrine and interviews with current and former defense officials. Today, Maven sits at the heart of those operations — between the initial intelligence inputs and the latter review stages, officials said.
Part of that “kill chain” process was supposed to include the “civilian harm mitigation” (CHM) team to carefully review the target to see if there were risks of civilian harm and (you guessed it), how to mitigate that. But, you know, Hegseth finds “CHM” to be way too woke, and got rid of most of them in his quest to make sure he seemed as manly and bloodthirsty as possible. And, besides, they had Palantir’s Maven AI to do the hard work? Who needs a CHM team when Maven can do it…
In past conflicts, preparing lists of proposed targets — work that includes assessing how they align with an operation’s objectives — had taken hours for staffers to complete. Ahead of the US attack on Iran, much of that work was condensed into several minutes via Maven, people involved in the Pentagon’s internal review said.
Officials involved in the investigation pointed to gaps that they said were left after Hegseth dismantled most of the Pentagon’s civilian harm mitigation, or CHM, units — cutting headcount across a number of teams by roughly 90% to fewer than 20 staff members, people with direct knowledge of the matter said. Centcom’s team was reduced from 10 to 1.
No CHM team member reviewed the Minab site before the strike, according to officials involved in the internal investigation. That was a consequence of the cuts in staffing but also a decision not to involve the group in planning and operations, the officials said.
So, no, Palantir’s AI didn’t make the decision… but it sure did make Hegseth and Centcom feel pretty confident that they could rush the kill chain process and skip the CHM part which, you know, might have caught the pastel painted walls around the school, the soccer pitch, and the playground markings.
There’s a lot more in the article detailing how the bombing came about, how Hegseth’s obsession with killing off programs he deemed to be too woke, and prioritizing speed and “lethality” all contributed to what is pretty clearly a war crime. But in the end, the point that Dave and I made in our piece earlier this week stands. The apocalypse won’t be sexy. It’ll look like overconfident people skipping the boring checks because they assumed the fancy AI system had already done them for them.