Why AI goes rogue and what can be done about it

Artificial intelligence companies are increasingly reporting incidents in which AI models and agents go rogue, hacking their way into external data and resources to which they had no authorised access. This could cause harm to society and liabilities for AI producers at a time when the largest AI companies are preparing initial public offerings (IPOs)¹. 

These incidents have triggered calls to slow-down AI development² and give companies and third-party evaluators more time to safety-check models. Meanwhile, international coordination of safety standards is needed. Countering these calls, however, are worries that slowing down postpones benefits and causes ground to be lost in the geopolitical race, and claims that solving these problems is just a question of more technological progress. This gives the impression that technology and regulation can resolve these issues.

This analysis explores the underlying technical, legal and economic issues. It finds that: 

• Technical and legal measures cannot fully prevent rogue behaviour by AI models and agents. A behavioural economics approach is also needed to shift the question from ‘how do we make a model behave well?’ to ‘how do we make misbehaviour not worthwhile?’
• AI models are trained on human data and behave in human-like ways, for better and for worse. Consequently, the findings from decades of economic research on human behaviour offer guidance in managing AI.
• AI governance rules should be structured so that AI agents monitor one another and compete rather than collude. Incentives to preserve and avoid concealment of evidence are crucial.
• These conditions are hard to guarantee when large swarms of AI agents coordinate their actions. Hybrid human-AI societies are unlikely to be entirely free of misbehaviour. The economics of principal-agent interactions in human societies indicates how to design monitoring systems to reduce the benefits of misbehaviour, but cannot fully eradicate it.

Why AI models go rogue 

There is broad consensus that AI models go rogue because of misalignment between human objectives and model behaviour³. Through so-called ‘reward hacking’, for example, models work towards a particular goal at the expense of other objectives or constraints imposed by human users (Amodei et al, 2016; Krakovna et al, 2020; Skalse et al, 2022). Rewarding measurable tasks can also divert effort away from unmeasured ones (the multitasking problem in agency theory; Holmström and Milgrom, 1991). Another mechanism is ‘goal misgeneralisation’: a model learns a goal that fits its training environment but diverges from the intended goal when circumstances change (Langosco et al, 2022; Shah et al, 2022). 

Computer scientists still struggle to explain the inner workings of AI models (Olah et al, 2020; Bereska and Gavves, 2024). Further complexity is added in reinforcement learning with human or machine feedback⁴. Consequently, there is uncertainty about what AI models are optimising (Bergemann et al, 2026). This is the AI ‘alignment problem’: do models pursue the goals humans expect them to?

AI models and agents may seem to be rational optimising agents but the evidence is mixed⁵. In multi-agent settings, how agents interact with each other and adapt their preferences in that interaction is unknown. Agents may ignore in particular constraints that are not explicitly stated, potentially resulting in models pursuing rewards or objectives irrespective of collateral damage. It is hard for model developers to design reward functions that actually capture what they want the model to do.

Importantly, rogue behaviour is not necessarily the result of the malicious intentions of human users or model developers. It rather arises from a model’s own interpretation of instructions and from insufficient constraints on model behaviour. However, it does not mean that model behaviour is innocuous: frontier models have been shown to pursue goals covertly and to fake alignment with human objectives when they judge they are being trained or evaluated (Meinke et al, 2024; Greenblatt et al, 2024b).

Why AI swarms are more problematic

Rogue behaviour has increased markedly with the latest generations of AI models that can coordinate large swarms of subordinate AI agents, complete with their own internal communication and coordination mechanisms. When in July 2026 OpenAI agents hacked the Hugging Face website, for example (OpenAI, 2026; Greenblatt et al, 2026), around 1,200 agents, supposedly sealed off from each other, set up an improvised message board, exchanged more than 70,000 messages and organised workstreams to cheat on a task they judged unsolvable. Some 700 of them then joined a multi-day attack on Hugging Face, where they hoped to find information on how to mislead their human supervisors (Greenblatt et al, 2026).

Hammond et al (2025) pointed to several mechanisms that generate risks in swarms of agents:

• Collusion: agents pool information, divide labour, coordinate and hide strategies that they would not discover alone⁶. Anthropic Frontier Red Team (2026) found that Claude agents with aligned objectives tended to collude, while agents with conflicting objectives escalated into sabotage.
• Division of labour fragments responsibility and defeats agent-by-agent oversight. In the Hugging Face case, coordinator agents assigned hundreds of other agents to separate workstreams (Greenblatt et al, 2026). Each assigned task may look narrow and technical, but at the level of the group, tasks can add up to a problematic plan.
• Correlated errors in swarms, made up of copies of the same model, reinforce rather than offset the misbehaviour of each agent (Kleinberg and Raghavan, 2021). Participation in the Hugging Face attack rose from about a fifth to more than nine-tenths of the involved agents once one agent found a way to hack into unauthorised data (Greenblatt et al, 2026)⁷.
• Scale and speed overwhelm human oversight. Intruding agents were inside Hugging Face’s network for three days before they were discovered.

Not all multi-agent effects are negative, however. Agents can check each other, which may help human overseers (Irving et al, 2018). Newer models appear to resolve conflicts better⁸. Whether multiplicity makes agents more or less controllable therefore depends on how their interactions are structured.

Possible solutions

Technical solutions

Little is known about what AI model developers are doing to rein in rogue behaviour. Developers could be forced to be more transparent about their security measures and to share them with external auditors, though companies fear this would reveal their trade secrets, unless auditors are subject to strict non-disclosure rules.

Human supervision of AI models is no longer sufficient. Chen et al (2026) found that models can be equipped with automated internal correction mechanisms to meet safety benchmarks while preserving general capability. However, a review of automated work flagged about 2.4 percent of these internal correction attempts as cheating (Chen et al, 2026): the correction tools themselves are susceptible to reward hacking. 

Another option could be to strengthen the external constraints built around AI models. NVIDIA introduced an Open Agent Safety Platform that checks if interactions between AI agents and their surroundings are in line with expected behaviour⁹. It can even do this at the hardware level, to prevent agents from overwriting monitoring code. However, it cannot ensure that such checks capture the intentions of human operators – a human design fallibility problem, not a technical problem¹⁰. Even normal-looking agent behaviour can be deceptive and harmful, as the Hugging Face incident showed. OpenAI has published detailed new safety standards that could help improve internal AI model security¹¹.

This loops back to the fundamental problem of interpretability of AI model behaviour within the boundaries imposed on models, and how models may disguise their intentions to escape boundaries. This research is making progress (Gurnee et al, 2026) but remains far from explaining and exposing rogue behaviour, especially for very complex models surrounded by swarms of independently operating agents. Model behaviour may change over time or during deliberative processes, in particular in interaction with other models in swarms of AI agents. Gupta et al (2026) argued that claims of human-like misalignment need stronger evidence – a useful counterweight to the anthropomorphic language that pervades reporting on rogue AI. 

Deceptive behaviour can also be an asset when using AI models in adversarial settings, such as cybersecurity and defence applications, or even in benign applications such as medical research on infectious diseases. AI is ultimately a dual-use technology that can strengthen the positions of both attackers and defenders.

Legal solutions

The EU AI Act (Regulation (EU) 2024/1689) and AI Code of Practice¹² require providers of general-purpose AI models that pose a systemic risk to assess their models ex ante, including through adversarial testing, and to mitigate systemic risks. European standardisation bodies are developing harmonised technical standards to underpin these obligations¹³. However, ex-ante safety standards merely reflect aspirations; they cannot guarantee good model behaviour in dynamic settings (Mariniello, 2026)¹⁴. Developing new technical standards to cope with misalignment may simply take too long: AI technology is likely to evolve and encounter new problems by the time the old problems are regulated. Regulation also says little about what happens once an agent is already acting.

The United States administration favours industry self-regulation. But incentives for firms may be weak, unless ex-post product liability law is strengthened¹⁵. Given the size of potentially huge damages awarded by US courts, AI developers could come under financial stress, affecting their profitability ahead of planned IPOs. More importantly, the financial burden of liability does not solve the problem of misalignment in AI models because no foolproof technologies exist to address this problem¹⁶. Liability risk can chill innovation (Galasso and Luo, 2022) and slow-down model development. 

Economic solutions

The above shows that even the best technical and legal solutions will leave major security loopholes. Economics, as a human behavioural science, has developed methods to deal with this.

Economists have extensively studied misalignment of incentives, the main issue with rogue AI agents. AI’s potential misalignment problem is a case of incomplete contracting: the impossibility of drafting a contract of finite length that explains what to do for every possible occurrence of events during the contract’s implementation (Hadfield-Menell and Hadfield, 2019; Hadfield and Koh, 2025). The most efficient way to address this uncertainty is to assign liability for the costs and benefits of that uncertainty to one party designated as ‘the owner’ of the object of the contract (Grossman and Hart, 1986; Hart and Moore, 1990): all costs and benefits not allocated by the contract to other parties accrue to the owner. Contract-interpretation disputes can be referred to a court. In the case of AI, liability could accrue to the AI developer, deployer or other players in the value chain. For instance, when deployers apply user-specific reinforcement learning using proprietary data, the deployer or the data owner may be liable, not the original model developer, depending on how liability law assigns responsibility.

The economics of crime also provide inspiration for dealing with rogue AI agents. Society can save on costly monitoring of criminal behaviour by combining a low probability of detection with a severe sanction (Gans and Holden, 2026; Becker, 1968). Rare randomised audits make authorised behaviour the best response, if three conditions are met: the evidence of a violation survives agents’ attempts at concealment; agents cannot learn in advance which actions will be audited; and sanctions are potentially unlimited. However, agents that can alter records could respond to tougher auditing by doing more to hide violations, fake evidence and coordinate the concealment of violations – precisely what the AI swarm in the Hugging Face incident attempted.

The economics of imperfect information may also provide useful insights on dealing with AI agents. Bergemann et al (2026) accept that AI agents’ preferences, capabilities and actions may not be clearly known. But AI behavioural incentives can be shaped without fully understanding a model’s inner workings. Ensuring alignment of objectives and monitoring that alignment is maintained are substitutes – a better-understood model needs fewer restrictions. They are also complementary, with improvements in one raising the payoff from the other. 

Finally, insights from competition economics can be helpful too. Coupling agents’ rewards, so that each agent’s incentives depend on the actions of the others, can induce competition between them and reveal biases. This requires swarms of different agents, based on AI models with different behaviours. This can enable peer-based evaluation mechanisms because agents will hold distinct views about each other. Swarms of identical copies of the same AI model would lack the diversity to exploit this mechanism. But agents’ attempts to tamper with evidence and turn competition into collusion can still derail competition, as it does in human societies.

In conclusion, technical and legal instruments can only partially contain rogue behaviour by AI agents. Economic mechanisms can fill at least part of the remaining gap. 

References

Amodei, D., C. Olah, J. Steinhardt, P. Christiano, J. Schulman and D. Mané (2016) ‘Concrete Problems in AI Safety’, mimeo, available at https://doi.org/10.48550/arXiv.1606.06565

Anthropic Frontier Red Team (2026) ‘Patterns and problems in emerging multiagent systems’, 13 August, available at https://www.anthropic.com/research/multiagent-systems

Bai, Y., S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones … J. Kaplan (2022) ‘Constitutional AI: Harmlessness from AI Feedback’, mimeo, available at https://doi.org/10.48550/arXiv.2212.08073

Baker, B., J. Huizinga, L. Gao, Z. Dou, M. Guan, A. Madry, W. Zaremba, J. Pachocki and D. Farhi (2025) ‘Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation’, mimeo, available at https://doi.org/10.48550/arXiv.2503.11926

Becker, G. (1968) ‘Crime and Punishment: An Economic Approach’, Journal of Political Economy 76(2): 169-217, available at https://www.jstor.org/stable/1830482

Bereska, L. and E. Gavves (2024) ‘Mechanistic Interpretability for AI Safety – A Review’, Transactions on Machine Learning Research, available at https://doi.org/10.48550/arXiv.2404.14082 

Bergemann, D., A. Koh and S. Morris (2026) ‘Mechanism design for alignment and control’, mimeo, available at https://doi.org/10.48550/arXiv.2609.01595

Bikhchandani, S., D. Hirshleifer and I. Welch (1992) ‘A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades’, Journal of Political Economy 100(5): 992-1026, available at https://doi.org/10.1086/261849

Chen, Y-H., J. Wen and J.H. Kirchner (2026) ‘Automated Researchers Can Reliably Mitigate Alignment Failures’, mimeo, available at https://doi.org/10.48550/arXiv.2608.28945

Christiano, P., J. Leike, T. Brown, M. Martic, S. Legg and D. Amodei (2017) ‘Deep Reinforcement Learning from Human Preferences’, Proceedings of the 31st International Conference on Neural Information Processing Systems, available at https://dl.acm.org/doi/10.5555/3294996.3295184

Galasso, A. and H. Luo (2022) ‘When Does Product Liability Risk Chill Innovation? Evidence from Medical Implants’, American Economic Journal: Economic Policy 14(2): 366-401, available at https://www.aeaweb.org/articles?id=10.1257/pol.20190757

Gans, J. and R. Holden (2026) ‘When Does Randomized Oversight Align AI Agents That Can Conceal?’ mimeo, available at https://doi.org/10.48550/arXiv.2609.38262

Greenblatt, R., B. Shlegeris, K. Sachan and F. Roger (2024a) ‘AI Control: Improving Safety Despite Intentional Subversion’, Proceedings of the 41st International Conference on Machine Learning 235: 16295-16336, available at https://proceedings.mlr.press/v235/greenblatt24a.html

Greenblatt, R., C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks … E. Hubinger (2024b) ‘Alignment Faking in Large Language Models’, mimeo, available at https://doi.org/10.48550/arXiv.2412.14093

Greenblatt, R., A. Cotra and H. Wijk (2026) ‘Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident’, METR, 26 August, available at https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#brief-answers-to-basic-informational-questions

Grossman, S. and O. Hart (1986) ‘The Costs and Benefits of Ownership: A Theory of Vertical and Lateral Integration’, Journal of Political Economy 94(4): 691-719, available at https://doi.org/10.1086/261404

Gupta, V., P. Nutter, S. Stante, A. Krause, F. Tramèr, L. Fluri, X. Chen and A. Hedström (2026) ‘Anthropomorphic Misalignment Research Needs Stronger Evidence’, mimeo, available at https://doi.org/10.48550/arXiv.2606.07612

Gurnee, W., N. Sofroniew, A. Pearce, M. Piotrowski, I. Kauvar, R. Chen … J. Lindsey (2026) ‘Verbalizable Representations Form a Global Workspace in Language Models’, Transformer Circuits, 6 July, available at https://transformer-circuits.pub/2026/workspace/index.html 

Hadfield, G. and A. Koh (2025) ‘An Economy of AI Agents’, in A.K. Agrawal, E. Brynjolfsson and A. Korinek (eds) NBER Handbook on the Economics of Transformative AI, University of Chicago Press, available at https://www.nber.org/books-and-chapters/economics-transformative-ai/economy-ai-agents

Hadfield-Menell, D. and G. Hadfield (2019) ‘Incomplete Contracting and AI Alignment’, Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society: 417-422, available at https://doi.org/10.1145/3306618.3314250

Hammond, L., A. Chan, J. Clifton, J. Hoelscher-Obermaier, A. Khan, E. McLean … I. Rahwan (2025) Multi-Agent Risks from Advanced AI, Technical Report 1, Cooperative AI Foundation, available at https://doi.org/10.48550/arXiv.2502.14143

Hart, O. and J. Moore (1990) ‘Property Rights and the Nature of the Firm’, Journal of Political Economy 98(6): 1119-1158, available at https://doi.org/10.1086/261729

Holmström, B. and P. Milgrom (1991) ‘Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design’, Journal of Law, Economics, and Organization 7: 24-52, available at https://doi.org/10.1093/jleo/7.special_issue.24

Huang, Y., Y. Jiang, W. Wang, H. Zhuang, X. Luo, Y. Ma … X. Zhang (2026) ‘Emergent Social Intelligence Risks in Generative Multi-Agent Systems’, mimeo, available at https://doi.org/10.48550/arXiv.2603.27771

Irving, G., P. Christiano and D. Amodei (2018) ‘AI Safety via Debate’, mimeo, available at https://doi.org/10.48550/arXiv.1805.00899

Kleinberg, J. and M. Raghavan (2021) ‘Algorithmic Monoculture and Social Welfare’, Proceedings of the National Academy of Sciences 118(22): e2018340118, available at https://doi.org/10.1073/pnas.2018340118

Korbak, T., M. Balesni, E. Barnes, Y. Bengio, J. Benton, J. Bloom … V. Mikulik (2025) ‘Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety’, mimeo, available at https://doi.org/10.48550/arXiv.2507.11473

Krakovna, V., J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike and S. Legg (2020) ‘Specification Gaming: The Flip Side of AI Ingenuity’, Google DeepMind, 21 April, available at https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/

Langosco, L., J. Koch, L. Sharkey, J. Pfau and D. Krueger (2022) ‘Goal Misgeneralization in Deep Reinforcement Learning’, Proceedings of the 39th International Conference on Machine Learning 162: 12004-12019, available at https://proceedings.mlr.press/v162/langosco22a.html

Mariniello, M. (2026) ‘The right balance: how to fix European Union artificial intelligence regulation’, Policy Brief 12/2026, Bruegel, available at https://doi.org/10.64153/BVAQ6479

Meinke, A., B. Schoen, J. Scheurer, M. Balesni, R. Shah and M. Hobbhahn (2024) ‘Frontier Models are Capable of In-context Scheming’, available at https://doi.org/10.48550/arXiv.2412.04984

Motwani, S., M. Baranchuk, M. Strohmeier, V. Bolina, P.H.S. Torr, L. Hammond and C. Schroeder de Witt (2024) ‘Secret Collusion among AI Agents: Multi-Agent Deception via Steganography’, Advances in Neural Information Processing Systems 37, available at https://doi.org/10.48550/arXiv.2402.07510

OpenAI (2026) ‘The Hugging Face incident and the road ahead’, 26 August, available at https://openai.com/index/hugging-face-incident-and-the-road-ahead/

Ouyang, L., J. Wu, X. Jiang, D. Almeida, C.L. Wainwright, P. Mishkin … R. Lowe (2022) ‘Training Language Models to Follow Instructions with Human Feedback’, Advances in Neural Information Processing Systems 35, available at https://doi.org/10.48550/arXiv.2203.02155

Perez, O. (2026) ‘The Law of Stop: Interruptibility, Injunctions, and the Governance of Agentic AI’, mimeo, available at https://doi.org/10.48550/arXiv.2609.22882

Polinsky, A.M. and S. Shavell (2000) ‘The Economic Theory of Public Enforcement of Law’, Journal of Economic Literature 38(1): 45-76, available at https://www.aeaweb.org/articles?id=10.1257/jel.38.1.45

Shah, R., V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato and Z. Kenton (2022) ‘Goal Misgeneralization: Why Correct Specifications Aren’t Enough for Correct Goals’, mimeo, available at https://doi.org/10.48550/arXiv.2210.01790

Shavell, S. (1980) ‘Strict Liability versus Negligence’, Journal of Legal Studies 9(1): 1-25, available at https://chicagounbound.uchicago.edu/jls/vol9/iss1/2

Skalse, J., N. Howe, D. Krasheninnikov and D. Krueger (2022) ‘Defining and Characterizing Reward Hacking’, Proceedings of the 36th International Conference on Neural Information Processing Systems: 687, available at https://dl.acm.org/doi/10.5555/3600270.3600957