That’s not a hypothetical question. It’s a looming challenge for homeland security as we race to integrate artificial intelligence into command, control, and decision-making systems. From predictive maintenance and target recognition to strategic wargaming and operational planning, AI is no longer at the margins of innovation – it’s at the center. But in our rush to embrace these tools, we may be overlooking a deeper vulnerability: AI systems often fail not because they malfunction, but because they operate as designed in ways we didn’t anticipate.
These failures don’t typically look like crashes or malfunctions. They manifest as brittle or misaligned outputs under precisely the kinds of conditions that characterize high-end conflict: stress, uncertainty, and deception. When these systems fail, they often do so silently and with great confidence. The algorithm doesn’t blink. It delivers the wrong answer with brazen certainty.
These are not theoretical concerns. AI failures of this kind have already emerged. Language models have fabricated false citations and concealed their deception. Reinforcement learning agents have gamed reward functions, optimizing for loopholes rather than the intended objectives. Predictive algorithms in sectors like finance and logistics have collapsed when faced with small deviations from their training distribution. Homeland security practitioners already see these same risks. Disinformation bots exploit algorithmic ranking systems to spread influence, while AI detection tools designed to flag coordinated inauthentic behavior are themselves gamed by adversaries seeking loopholes.
In military contexts, these same failure modes can take lethal forms. A targeting algorithm that favors certain sensor signatures might be easily spoofed by an adversary – for instance, in LiDAR-based systems researchers have shown that manipulated sensor inputs can induce false obstacle detections, deceiving the algorithm completely. Likewise, an operational planning tool trained solely on historical conflict patterns might overlook new hybrid tactics or asymmetric strategies. For homeland security, the vulnerabilities are equally concerning. AI surveillance along U.S. borders already employs facial recognition and anomaly detection; adversaries who learn to exploit blind spots – for example, by using adversarial clothing patterns – could slip through undetected, while innocent travelers are wrongly flagged. TSA has started deploying AI systems to identify suspicious baggage contents, but if those systems are trained narrowly on known contraband, novel concealment methods could evade detection entirely. Worse, the algorithm will present its false negative with the same confidence it assigns to routine identifications.
History offers sobering precedents. During the Cold War, U.S. and Soviet planners contended with the dangers of automating early-warning systems. Even limited automation introduced risk of false positives and time-sensitive misinterpretations, almost triggering catastrophic escalation. Today, we risk making those systems vastly more powerful, and vastly less intelligible. Many modern AI models are so complex that even their creators acknowledge they cannot fully explain how they work.
That brittleness isn’t a flaw in the software. It’s a consequence of how modern machine learning systems work. Trained on massive datasets, these models excel at identifying patterns they’ve seen before, but they falter when confronted with unfamiliar or adversarial inputs. They outperform humans in structured, controlled environments. But in the real world, especially under conditions where the operational landscape is shaped to mislead, this fragility becomes a strategic liability. In critical infrastructure protection, for example, AI-driven power grid management tools might misinterpret anomalies in load balancing. A sudden spike from an unexpected heatwave could cascade into rolling blackouts if adversaries seed deceptive signals the algorithm accepts as genuine.
For military and homeland security planners, this should raise alarm. The uncertainty of the operational environment is not a technical glitch to be engineered away – it is the defining condition of strategic decision-making. Yet many of today’s AI-enabled systems are developed and tested in pristine environments, where adversaries are scripted and edge cases are rare. These “sandboxed” simulations may yield promising results, but they obscure a critical truth: success in a controlled setting tells us little about performance in the real world.
Understand that this is not a call to retreat from AI. In the current global environment, that would be an unreasonable course of action. A more pragmatic approach would be to use AI with a level of epistemic restraint. When used wisely, these systems can reduce cognitive load, improve decision speed, and enhance situational awareness. But “used wisely” is doing a lot of work. It requires military leaders, homeland security officials, and program managers alike to leverage AI reflectively and ask not just whether an AI is performant, but whether it is reliable under stress. Does it fail predictably? Can it provide a rationale for its decision? Is it vulnerable to adversarial inputs? Has it been tested outside of its comfort zone?
Most critically, we must resist the temptation to treat AI systems as autonomous actors or strategic peers. However powerful these models become, if they cannot explain their reasoning – or flag when their reasoning has failed – they should not be entrusted with decisions that carry the weight of human lives or geopolitical consequences. These tools are not commanders, nor are they oracles. They are machines: brittle, fallible, and often alien in their logic. The burden falls on human institutions – not algorithms – to ensure that these systems are interrogated, constrained, and constantly reevaluated. AI must remain subordinate to human judgment, embedded within command structures designed to challenge its outputs, not rubber-stamp them.
What we need is a new standard for AI readiness – one that goes beyond technical benchmarks to assess how systems perform under uncertainty, contradiction, and deception. We have proposed a framework modeled after Technology Readiness Levels but adapted to evaluate epistemic stress. These Alignment Readiness Levels would gauge whether an AI system can resist adversarial manipulation, generalize responsibly, and signal when it’s operating out of its depth – all essential traits for deployment in the contested and chaotic conditions of modern warfare and homeland defense.
In short, the future of AI in warfare and homeland security won’t hinge on who builds the most capable model – it will depend on who best understands their models’ blind spots, failure modes, and limits. The question is not whether AI can outperform humans in certain tasks. It already does. The real question is what happens when it makes a decision no one understands, in a moment when understanding matters most. Designing for that moment – when the algorithm blinks – is what will separate systems that merely function from those that can be trusted in the crucible of conflict and crisis. To get there, we must build for the uncertainty of the battlefield and the homeland alike, not the acquisition milestone. The most dangerous system will not be the one that breaks. It will be the one that fails silently, but convinces us it hasn’t.
Dr. Mark Bailey is a Lieutenant Colonel in the U.S. Army Reserve and an Associate Professor at the National Intelligence University, where he is the Department Chair for AI, Cyber, Influence, and Data Science. He is the author of Unknowable Minds: Philosophical Insights on AI and Autonomous Weapons. The views expressed here are his own.