TLDR : Face aux risques d'attaques sur l'IA dans le domaine militaire, le Commandement de la cyberdéfense (COMCYBER) et l'Agence de l'innovation de Défense (AID) ont lancé un défi pour sécuriser l'IA, avec des propositions notables de PRISM Eval et du CEA-List. PRISM Eval se concentre sur l'analyse des dérives comportementales des systèmes d'IA, tandis que le CEA-List vise à sécuriser les modèles de classification visuelle contre les modifications malveillantes de données.
Table of contents
Predictive maintenance, intelligence analysis, conflict simulation, cyber defense: AI is now a major issue for the armed forces and an indispensable information system. At the same time, it has introduced unprecedented attack surfaces: exploitable models, manipulable data, alterable responses... To anticipate these vulnerabilities and develop solutions to counter them, the Cyber Defense Command (COMCYBER) and the Defense Innovation Agency (AID) launched the "AI Security" challenge.
AI, as an information system, is exposed, vulnerable, and potentially divertible. Adversarial attacks, extraction of sensitive information, or generation of malicious content are no longer theoretical hypotheses but active aggression vectors.
Its deployment in the military domain requires rigorous security, integrating a solid technical framework, algorithmic resilience, and enhanced operational control.
The challenge received over a dozen applications from laboratories, startups, SMEs, ETIs, or large groups. Two stood out: those of PRISM Eval and CEA-List.
PRISM Eval: Testing the Behavioral Flaws of LLMs
Founded in 2024, the Paris-based startup PRISM Eval specializes in red teaming, behavioral interpretability, and alignment of advanced AI systems. It aims to develop a fine understanding of the cognitive mechanisms of LLMs to control large-scale deviations. This scientific approach materializes in the BET (Behavior Elicitation Tool) suite, winner of the challenge.
Its first product, BET Eval, directly addresses the robustness needs of LLMs powering ChatGPT, Le Chat, or GenIAl, the Ministry of Defense's AI assistant. The tool operates as a battery of behavioral intrusion tests, combining semantic and contextual attack primitives to evaluate:
- the model's capacity to generate malicious or dangerous content (toxicity, incitements);
- its vulnerability to the exfiltration of sensitive information;
- the ease with which its safeguards can be bypassed (prompt injection, jailbreak).
CEA-List: Securing Visual Models through Verification and Trust
For its part, CEA-List targets the security of visual classification models against data modification attacks. Here, the risk is more insidious: a slightly altered image by an adversary can lead an AI to identify a civilian vehicle as a hostile vehicle — or vice versa.
Its solution relies on two complementary tools:
- PyRAT, which applies formal verification to neural networks. It provides mathematical guarantees against subtle attacks, such as imperceptible pixel modifications intended to deceive automatic classification (a well-documented technique but difficult to detect in real-time);
- PARTICUL, which calculates a confidence score based on the detection of regularities in datasets. It allows the detection of more visible intrusions (like the addition of patches) by measuring the anomaly degree of an entry.
These two tools address both upstream (formal model robustness) and downstream (operational data trust), combining symbolic logic and statistical empiricism.
Translated from PRISM Eval et le CEA-List, lauréats du défi Sécurisation de l'IA
To better understand
What is formal verification of neural networks, and how is it applied in the context of military AI security?
Formal verification is a rigorous method using mathematical techniques to prove or disprove the correct functioning of systems, including neural networks. In the military context, tools like PyRAT apply this technique to ensure visual classification models are not deceived by subtle and intentional data modifications, thus ensuring the reliability of AI decisions.
What are the regulatory challenges associated with the use of AI in the armed forces?
Regulatory challenges include the need to establish robust security standards to prevent AI misuse and ensure compliance with international humanitarian laws. There are also ethical considerations, such as respecting human rights and liability issues for errors or unintended harm caused by autonomous AI systems.