Pope Leo XIV just dropped his first social encyclical, Magnifica Humanitas, and the global media is doing exactly what it always does: panicking over a headline. The pontiff, flanked by Silicon Valley elites like Anthropic co-founder Chris Olah, warned that autonomous weapons systems have advanced "practically beyond any human reach to govern them." He called for AI to be "disarmed" and declared it impermissible to entrust lethal decisions to algorithms.
It is a beautiful, deeply moral sentiment. It is also dangerously disconnected from the mechanical reality of modern conflict.
The lazy consensus dominating the headlines argues that human-in-the-loop systems are inherently safer, more moral, and structurally superior to autonomous targeting. This view treats human judgment as an unalloyed good and algorithmic execution as an unmitigated risk. I have spent years auditing software deployment pipelines and analyzing automated systems under stress. If there is one undeniable truth buried beneath the marketing fluff of both tech companies and international regulators, it is this: human intervention is frequently the exact point where systems fail catastrophically.
The global debate is asking the wrong question. We should not be asking how to force human control back into systems that move too quickly for human biology. We should be asking how to build deterministic, mathematically verifiable constraints into autonomous systems before our insistence on human oversight causes the very slaughter we are trying to prevent.
The Lethal Myth of the Human Loop
The core premise of the Vatican’s argument, and indeed the foundation of most international humanitarian hand-wringing, is that keeping a human finger on the trigger guarantees accountability. It does not. In high-velocity combat environments, it guarantees cognitive failure.
Consider how modern defense arrays operate. A supersonic or hypersonic missile threat does not grant a human operator the luxury of "moral discernment." When an incoming threat registers on radar moving at Mach 5, the window for detection, tracking, identification, and interception is measured in milliseconds.
When you force a human into that loop, you are not injecting morality; you are injecting latency.
[Threat Detected] ➔ [Algorithmic Verification] ➔ [Human Confirmation Request] ➔ [Cognitive Processing Delay] ➔ [Lethal Penetration]
Human cognitive processing takes roughly 200 to 300 milliseconds just to react to a visual stimulus. Under severe stress, that window expands as confirmation bias, panic, and sensory overload take hold. By the time a human operator reviews the data, checks the Rules of Engagement, and clicks a confirmation dialog, the asset they were defending is a smoking crater.
The defense sector already knows this. Systems like the Navy's Phalanx Close-In Weapon System (CIWS) or Israel’s Iron Dome have operated with highly automated, rapid-fire modes for decades. To pretend that AI introduces a completely unprecedented category of speed is to ignore the history of automated air defense. The difference now is scale and adaptation, not the fundamental physics of reaction time.
The Data Bias Fallacy in Kinetic Warfare
Pope Leo correctly identified that algorithms are "never neutral" and can perpetuate biases found in their training data. This is a legitimate critique when applied to credit scoring, hiring algorithms, or domestic surveillance pipelines where historical socio-economic disparities distort the data.
However, transporting this exact argument wholesale into kinetic military targeting is a category error.
A computer vision model tracking a T-72 tank or an S-400 missile battery does not suffer from systemic historical racism. It operates on geometric structures, infrared signatures, radar cross-sections, and electromagnetic emissions. The "bias" in a military target-recognition model is not a moral failing; it is an engineering variance problem.
When a drone misidentifies a civilian vehicle as a technical truck, it is not because the algorithm is filled with "dehumanizing ambition" or corporate greed. It is usually because of low-resolution sensor feeds, atmospheric occlusion, or adversarial masking.
Fixing that problem requires more training data, better edge-compute hardware, and tighter sensor integration—not a slow-down mandated by international decree. Forcing an operator to manually verify an obscured image under the pressure of an active firefight actually increases the likelihood of a panicked, mistaken strike. Humans are notoriously terrible at interpreting ambiguous, low-quality sensor data under duress. We see what we fear, not what is there.
The Illusion of a Verifiable Chain of Responsibility
The encyclical demands an "identifiable and verifiable" chain of responsibility so that accountability is not collapsed into the machine. This sounds logical on paper. In practice, it reveals a profound misunderstanding of how distributed software systems function.
Modern machine learning models, particularly deep neural networks, are probabilistic. They are not explicit, line-by-line recipes written by a single engineer. They are statistical frameworks trained on vast datasets. When a model selects a target, it does so by optimizing a loss function across millions of parameters.
If an autonomous system commits a war crime, who do you prosecute under the Vatican's framework?
- The software engineer who wrote the optimization loop?
- The data labeler who categorized the training images?
- The field commander who activated the system?
- The politician who authorized the deployment?
Insisting on traditional hierarchy in distributed algorithmic warfare is an exercise in futility. If a commander activates an autonomous drone swarm with a broad mission parameter ("deny enemy armor movement in sector X") and the swarm miscalculates an edge case, punishing the commander does nothing to fix the underlying code flaw. It merely creates a political scapegoat while leaving the systemic technical vulnerability entirely intact.
The Geopolitical Naivety of Universal Disarmament
We must address the elephant in the room that Western tech executives and religious leaders routinely ignore during high-profile panel discussions. Treaties only bind the parties that sign them.
The Pentagon’s current integration of autonomous capabilities is not driven by a desire to abdicate moral responsibility; it is driven by intense strategic competition. When peer adversaries are aggressively developing uncrewed, algorithmic strike platforms that operate without ethical constraints, unilateral restraint is a form of strategic suicide.
Imagine a scenario where the United States or its allies rigidly enforce a strict "human-always-approves-the-strike" protocol across all theater operations. An adversary deploying fully autonomous, decentralized drone swarms can saturate defensive lines by sheer computational velocity. The human-in-the-loop army will be systematically dismantled because its decision cycle—its OODA loop (Observe, Orient, Decide, Act)—is bound by human biology, while the adversary's loop is bound only by processing power and electricity.
Adversary OODA Loop: [Sensor Input] ➔ [Algorithmic Decision] ➔ [Kinetic Output] (0.05 seconds)
Western OODA Loop: [Sensor Input] ➔ [Algorithmic Suggestion] ➔ [Human Review] ➔ [Command Approval] ➔ [Kinetic Output] (45.00 seconds)
To tell state actors to "slow down" in the face of this asymmetry is not moral leadership; it is a refusal to acknowledge the fundamental nature of deterrence. Peace is not maintained by a lack of weapons; it is maintained by the mutual calculation that an attack will fail or result in unacceptable costs.
The Real Risk: Brittle Systems, Not Sentient Machines
The true danger of autonomous weapons is not that they will become self-aware, reject human governance, and launch an endless war for global domination. That is the realm of science fiction and sensationalist journalism.
The real risk is brittleness.
Autonomous systems are profoundly dumb in ways humans find difficult to comprehend. They are highly susceptible to adversarial examples—deliberately engineered inputs designed to trick a model into making a glaring error. A specific pattern painted on top of a building can cause a targeting algorithm to misclassify it entirely. A minor modification to an electronic warfare environment can cause a drone's localization stack to fail, leading it off course.
When we focus the entire ethical debate on the grand philosophical question of "should machines have the right to kill," we ignore the immediate engineering reality: these systems can be easily blinded, spoofed, and manipulated.
The downside to my contrarian view is obvious: removing the human loop completely means that if a catastrophic software bug or data corruption event occurs at scale, the system will execute its flawed logic at computational speed across an entire theater before anyone can pull the plug. That is a terrifying prospect. But the solution to system brittleness is not to add a slow human component to the end of the pipeline. The solution is to build redundant, deterministic verification layers into the architecture itself.
Instead of demanding an impossible return to manual warfare, we must pivot toward automated verification. We need systems where independent, hard-coded safety constraints—deterministic "if-this-then-that" rules—can override probabilistic machine learning outputs. If a drone's targeting system selects an object, an independent, non-AI sub-system must verify that the object's coordinates do not intersect with a known civilian exclusion zone, regardless of what the neural network thinks it sees.
Stop trying to fix autonomous warfare by forcing humans to do a machine's job. Start fixing it by holding the software to the unyielding standard of absolute verification.
