There are some things we shouldn't do...

Taking a look at Gospel and what it means to let AI "target".

Thanks to Lena for a great piece on the IDF’s Gospel system. If you’re a subscriber, Lena treated you to two pieces on Gospel, the second of which is still hanging on the Substack. All that effort to click on the subscribe button really pays off! You get two for one!

I’ve been percolating on this for a while. There are several new ideas that need some expansion here, but this isn’t a journal article so let’s just get the discussion going. It’s long. Brace yourself. Also, this essay takes no sides in this conflict, it simply analyzes a component of it.

What concerns me about the information in Lena’s piece and other reports about Gospel and its supporting systems (Fire Factory and Alchemist) is that the IDF might be allowing Gospel to nominate targets, not just identify them. If that is true, Gospel is creating a tremendous amount of strategic risk for the IDF, and they need to be more forthcoming about what it’s doing. Frankly there are some things that AI just shouldn’t do, and nominating targets is one of those things.

AI Nominating Targets is Bad.

In my view, if Gospel is just identifying targets in the battlespace, that is pretty much ok. I’ll include tracking and custody in that. There are some engineering challenges, but it is possible to build an AI-enabled target identification system that operates with reasonable performance under specific circumstances. However, nominating targets should be avoided because of the cognitive and ethical hurdles that machines cannot surmount, which will accrue unacceptable risk for everyone involved.

The difference between nominating targets and identifying targets is significant. In every piece of reporting I’ve seen on Gospel, the term “targeting” is used to cover just about every phase of a targeting cycle. I don’t want to be too pedantic but using that term to cover so many functions obscures what the system might be doing. I assume journalists just don’t know any better, but I suspect the IDF is being deliberately imprecise to avoid some tough questions. So let’s unpack this a bit.

Brief Description of a Targeting Process.

To start, a brief description of the targeting process might help. When militaries are trying to figure out how to accomplish their mission, they use a decision process that includes the nomination, approval, identification, actioning, and analysis of targets. This occurs in a continuing cycle that can take anywhere from hours to months to years. The critical function is to ensure that actions are in service of a greater purpose, are aligned with an ethical framework, and will not waste limited resources.

When nominating targets, analysts are recommending that the higher purpose can be reached by affecting some entity in the environment. As you might guess, nominating (and approving) targets requires reasoning through complex concepts like context, values, trade-offs, risk, strategy, and culture. Analysts must consider immediate and long-term implications of their recommendations while balancing immediate pressures. At best a bad nomination will just waste time and resources. At worst, bad nominations can divert military effort from a winning strategy and cause the collapse of further meaningful pursuit of desired goals. The analysts who nominated the U.S. Pacific Fleet in 1941 made an existential mistake.

Once nominated and approved, finding and identifying the target comes next. In many ways this is an engineering problem with many processes, though not completely devoid of reasoning. Identifying mostly consists of sensors (including people) bringing in data for analysts to confirm that an object is indeed an approved target. AI enabled computer vision can be very helpful in culling through data quickly resulting in faster, and hopefully more accurate, engagements but we know misidentifications will happen.

While misidentifying targets can have serious, sometimes lethal, consequences, in most cases the error is realized quickly and corrected. This is not the case in the complex, sometimes nebulous process of nominating targets. Because the effects from engaging targets is not always immediately known, deciding whether a target is the correct one or not takes time and analysis.  And the risk for getting target nominations incorrect is much higher than misidentifying targets, to the point where it seems obvious that humans would want to keep nominating as a human function only. To stay with our Pearl Harbor example, how long did it take the Japanese military to realize that Pearl Harbor was a strategic mistake. Months? Years? However, using AI-enabled systems to nominate targets is not irrational and there are real incentives to do so.

Why would militaries let AI nominate targets?

First, there is so much data available for analysts to consider when nominating targets that sifting through it all might take so long that an adversary could act more quickly and decisively. This is coupled with the assumption that considering all the data will necessarily improve target nomination quality. This quest for speed and quality in nominating the right targets is key to gaining what the U.S. military terms “decision-advantage”. That is an ability to reason and act at a tempo that overwhelms the adversary.

But there is more to it than just tempo, there is also a quest for Move 37. Those familiar with DeepMind’s AlphaGo will recognize Move 37 as the point in AlphaGo’s 2016 match against the human Go champion, Lee Sodol, that suggested the machine saw a strategy that no human would have seen, much less implemented. When AlphaGo made Move 37, Lee Sodol had to leave the room, he was so taken aback. That move, unforeseeable to the human adversary, appeared to be a critical point for AlphaGo to win the match. Militaries of the world took note. The idea of finding a Move 37 in war is tantalizing to militaries leaders because it provides an almost magical tool against a talented resourceful adversary limited by human cognition. What could go wrong?

Recovering from mistakes.

If AI is employed for tasks where, if it makes a mistake, we can see it, understand it, and recover from it, we can probably consider integrating it safely. This is why AI is well suited for helping humans with process-oriented tasks such as target identification. If the model gets the identification wrong, the mistake may be significant, but we will know and we can then work a way to recover and improve. When introducing AI to reasoning processes, the nature of the technology means that we may not recognize that mistakes are being made until we have a catastrophe. I want to stay this side of saying an existential catastrophe, but I wouldn’t rule it out.

Let’s use an example here. If I use AI to identify tanks on the battlefield and it makes a mistake, we will know within a reasonable span of time, and we’ll be able to refine the model or suspend its actions. Could noncombatants be killed? Yes of course, but unfortunately in most cases this is not catastrophic. However, if AI is used to nominate tanks as a target and the organization efficiently and ruthlessly starts destroying tanks, the cycle will continue until enough data and analysis comes in to suggest that there is a problem. How long will that take? How much ordnance will be expended on tanks and how much damage will be done pursuing tanks? If the AI is also doing the post-effects analysis, how will it ever conclude that it is on the wrong path and that its actions have detached from its purpose? How long would it take human oversight to see the problem?

I suspect that it would take a long time for humans to notice that an AI’s target nominations were not supporting the higher purpose. In that time a lot of damage to the mission could occur. What might that look like? It could look like 28,000 Palestinian deaths in Gaza causing the world to turn its back on Israel and costing Israel a strategic victory in favor of rapid “targeting”. The problem is that the nature of AI technology and the cognitive limitations that humans bring to any interaction with machines makes control of an AI target nomination system perilous.

How would AI target nomination detach action from purpose?

Why would AI target nomination spin action so far from purpose and why is it hard for humans to see? First AI technology is not capable of reasoning. There I said it. It isn’t reasoning at any level that humans find worthy of the term. Second, AI tech has an explainability problem. Meaning the whys and the hows of its outputs remain uncertain. Third, adaptive models will adjust to reinforce positive feedback humans love to outsource hard things to machines and they tend to be comfortable letting the machine take the lead. We call this automation bias. Lastly, many times what we see when machines give humans lots of options is that there is a bias to take action. This idea is a little new. We’ll start to explore it here but I think it deserves some serious research on its own. Some colleagues and I are building this study right now. In the meantime, let’s take a quick look at these four things.

Inability to reason.

First, the obvious. AI is not reasoning. There is no system out there that is reasoning, including LLMs. There is also no system that can understand the complexities that inform human purpose, including constitutional AI from Anthropic (though that is a great idea we tried to build at the JAIC in 2020 and were told it was impossible, so hats off to the Anthropic peeps). So even the most advanced AIs are using probability, token matching, etc to predict which outputs might satisfy the user. This is not the same as reasoning, but I’m not going to do a philosophy summary on reasoning here. Just press the “I believe” button for now.

Transparency/Opacity.

Second, we don’t really know why an AI makes the recommendations it makes. This is the explainability/transparency/opacity problem. When a computer vision model identifies something like a tank, we are not completely sure what features the model is using to determine tank or not tank. Is it the turret, the gun, the tracks, the color? To address this opacity, we box in the model’s deployment parameters. We test the model to determine where and when the model can be used, under what conditions, and where it will fail. With solid T&E data we can try to employ the model well within its performance zone, depending on our risk tolerance. With a target identification model we can test how well it works under given conditions and then mandate that it is employed only in those conditions. How would this work with a target nomination model?

Adaptive models giving you more of what you want.

In military circles we have measures of performance and measure of effectiveness. In shorthand we ask two questions: 1) are we doing things right? And 2) are we doing the right things? There AI models, called adaptive models, that can do the first.  By “observing” their environment the models tweak their software, and thus their actions to increase the probability of reaching the desired output. The feedback mechanism for this is effective but crude so I’m skeptical about applying it to the reasoning problem required for question number two. I don’t think it will work and I think experience shows that it constrains reasoning, creativity, and initiative.

Adaptive models are designed to learn during use so that they get better at giving you what you want. This presumes that what you want now is the same as what you wanted before. Much like your Netflix or Amazon accounts. What can happen is an AI pushes you down a reasoning tunnel that prevents you from seeing or thinking anything new, because it is only showing you stuff based on reinforcement from the user. This is why Facebook’s like and dislike buttons are so important to their models. The model will base its recommendations on previous success rather than on fresh reasoning. How good is it? My accounts think I’m a 70 year-old white lady from Alabama. Please someone make it stop showing me baking videos. Imagine an AI trying to please you with target nominations. AI: “you really liked when we blew up that missile truck last week, shall we blow up some more missile trucks?” You: “I guess. Missile trucks seem bad.”

Automation bias.

That’s crazy, you’d obviously stop that nomination and want a more refined analysis, right? Would you though? There are two AI-human biases that suggest you might let it ride. First, several studies suggest that humans tend to favor outputs from automated sources. At its simplest this means that humans will tend to trust machine outputs even if they receive external input that the machine is wrong. Here is a recent study where students continued following a robot in a “burning” building even though the robot was leading them away from the fire exits, which the students could clearly see.

Automation bias seems related to confidence (or trust, if you must) in the machine’s output and lack of confidence in human reasoning. After all, machines consider more data, and aren’t clouded by emotions, prejudices, and other bias. The confidence created by automation bias can result in increased action, and it seems possible that Gospel could be creating that environment for IDF decision-makers. Particularly when we introduce a second human cognitive limitation: action bias.

Action bias.

In this study, professional soccer goalies were shown to prefer to move rather than stay still during penalty kicks, even if staying still was the better strategy according to the data. Reasons for this are not clearly understood but let’s speculate. There could be pressure to act in the face of action by an adversary. In the case of the goalie, the guy kicking the penalty is acting, what is the goalie doing in response? Just sitting there? To many that would seem unacceptable. The goalie needs to do something. In the goalie’s mind, is it then better to appear to be doing something? Goalies usually do not save penalty kicks. Saving one is very, very hard. If you know you will likely fail, is it better to fail while doing something or doing nothing? If you act, at least you tried right? Or perhaps you do block the penalty kick. If you block it by just standing there, are you rewarded with the same accolades as if you were to block it by making an incredible dive? Were you just lucky?

When we start looking at Gospel we know that it is “producing” many more targets than a human staff would. If that means identifying targets, then action bias might have the IDF striking more targets because there are more identified targets to strike. You can imagine the IDF strike authority being asked why they are not striking all these perfectly good targets. And you certainly would not want to be the one who fails to defeat Hamas because you didn’t action all the available targets. And striking those identified targets would still be validated as being within the overall strategy. The cost for inaction may be worse than the cost of action. Gospel’s productivity might be enabling IDF personnel to do more, but it may also be creating pressure to act far more than may be in their interests.

If Gospel is nominating targets, and action bias is in play, the IDF could be running its strategy off a cliff because it doesn’t know why it’s doing what it’s doing. But it’s doing a lot of it. The confidence and action encouraged by an AI allowed to affect human reasoning through the target nomination process could mean that the IDF is entering a strategic hallucination cycle from which they can’t escape until it’s too late.

The hallucination cycle.

When I say hallucination cycle, I mean that an organization that allows AI into its reasoning processes linking action with purpose no longer knows what is and is not in the service of human goals. Your understanding of the situation becomes so tied to what the AI is feeding you, its interpretation of what you want, and its interaction with your own biases that you can no longer tell what reality is. And the worst part is that you will not even know anything is wrong.

I have no special knowledge of Gospel or what is going on in Gaza, but the sheer quantity of strikes the IDF is actioning makes me worried that their actions have become untethered from a coherent strategy. If that is in part a result from integrating AI systems too far into human reasoning and target nomination cycles, the IDF needs to know that so it can recover and get back on track. The rest of the world needs to know that too so we can avoid a similar fate.