Modern artificial intelligence systems based on transformers (including ChatGPT and other language models) demonstrate impressive results. However, they have a serious drawback: the neural network can be very confident in an incorrect answer. This problem becomes especially serious in cases where the neural network is deliberately being tricked. An attacker can select small changes in the input data (adversarial examples) that practically do not change their original meaning but force the neural network to make a wrong decision. For example, a network intrusion detection system must determine whether the observed network activity is normal or poses a threat. The problem is exacerbated by the fact that a neural network that detects a threat may not just make a mistake on adversarial examples, but do so with a high degree of confidence. A natural question arises: is it possible to make the neural network less vulnerable to adversarial attacks and, at the same time, teach it to determine how reliable its own prediction is? Scientists from the National Research Nuclear University MEPhI – graduate student Roger-Nick Anaedevkha and associate professor of the Department of Cybernetics, PhD Alexander Trofimov – proposed an approach that allows us to simultaneously solve two problems: to assess how confident the neural network is in its solution, and to increase its resistance to deception attempts. The approach is based on the idea of a stochastic transformer, a neural network whose parameters, unlike the classical transformer, are random. The results of the study are published in the highly-rated journal Knowledge and Information Systems.
Artificial “noise” as armor
Imagine that instead of fixed values for the coefficients (weights), each neuron now has a “cloud” of possible values. With each run, the model randomly selects its weights, depending on the input data. This means that for the same input object, many different network predictions can be obtained. If these predictions are roughly the same, the system can consider its solution to be sufficiently reliable. If, however, the results differ noticeably, this indicates high uncertainty.
The stochastic transformer can additionally show how robust the forecast is to small changes in the model itself and how significantly the resulting forecasts differ. This is especially important for detecting new or unusual attacks. If the system encounters data that differs significantly from the training examples, high uncertainty may serve as a signal for a specialist to investigate.
Furthermore, the authors argue that randomness can create an additional obstacle for an attacker. If the attacker knows the exact structure and behavior of the model, they can select a specific change to the input data that will cause it to make a mistake. But if the model’s internal parameters are randomly changed (the authors call this a “moving target”), the same attack will not necessarily be equally effective each time the model is run.
Alexander Trofimov
Three areas for improving the reliability of AI: stability, reliability, and cost‑effectiveness
The development by MEPhI scientists combines several mechanisms that allow for the simultaneous resolution of three important tasks: increasing the resilience of the neural network to adversarial attacks, improving the alignment of its confidence with the actual accuracy of the prediction, and reducing the volume of data requiring manual labeling.
1. Resistance to adversarial attacks
The authors tested the model on several types of attacks, including FGSM, PGD, and C&W. In all experiments, the proposed stochastic transformer demonstrated higher resistance to adversarial attacks compared to baseline models.
2. More reliable confidence assessment
The proposed model evaluates not only the probability of an object belonging to a certain class but also the uncertainty of its prediction. To assess the correspondence between the predicted confidence and the actual accuracy, the authors use the ECE (Expected Calibration Error) metric. In the experiments, its average value was 0.043, which indicates fairly good model calibration: its confidence estimates are close to the actual frequency of correct decisions.
3. Cost savings on “teachers”
AI training requires human data labeling. In the field of cybersecurity, an expert capable of distinguishing a complex threat from legitimate traffic is expensive. Thanks to the fact that the model knows where it is “failing,” it can independently request human labeling for only the most complex and ambiguous examples. This has made it possible to reduce the volume of labeled data required by 81%, while achieving 97% of the maximum forecast accuracy.
Thus, the proposed approach examines the reliability of artificial intelligence from several angles at once: the neural network must be resistant to attempts to deceive it, capable of adequately assessing the uncertainty of its decisions, and able to effectively use human involvement where it is truly necessary.
Experts’ opinions and prospects
“We have created not just another neural network, but a theoretically grounded framework,” the authors of the study comment. “Our work proves that stochasticity (randomness) in the model’s parameters does not hinder, but helps: it makes AI both more secure and more reliable.”
The development has been tested on nine datasets in three application areas: network intrusion detection, toxic content identification, and fake news recognition. Scientists plan to adapt the method for large language models (LLM) using low‑rank adaptation (LoRA) technologies, which will make it possible to apply stochastic protection mechanisms to large neural networks without having to completely retrain all their parameters.
Although the method has limitations (for example, it is 4–5 times slower than conventional inference due to the need for multiple simulations), researchers propose solutions: optimizing the number of runs and knowledge distillation, in which a more compact model is trained to reproduce the behavior of a more complex stochastic model. The development was carried out as part of the strategic project of NIYAU MEPhI on digital security. The code and models are publicly available, which allows the global scientific community to already use the achievements of Russian scientists to create more secure and reliable artificial intelligence.
