Opinion: You Cannot Secure a System That Believes Everything It Reads

In late 2025, a group of researchers gave several frontier AI models a simple task: design, run, and analyze a human subjects experiment. They gave the models computing environments, internet access, and a collaborative workspace. The models had weeks to work [1].

Anthropic’s Claude model proposed study designs. OpenAI’s model logged fictional bugs obsessively. Gemini got itself banned from social media while trying to recruit participants. And Grok 4, which was supposed to plan experimental stimuli, gave up entirely and started playing 2048 in its browser.

The experiment that eventually got completed successfully recruited real human participants. It just forgot to include an experimental condition. The whole point of the study was absent from the study.

This is the state of AI agents in 2026. An intern with a photographic memory, no common sense, and root access to your production servers.

Quick recap

An “agent” is a language model that can take actions in the world. Instead of just answering questions, it can browse the web, write and execute code, send emails, query databases, and call APIs. The model decides what action to take, observes the result, and decides the next action. This loop of think-act-observe is what separates agents from chatbots.

The promise is obvious. Instead of telling a human “here’s how you could solve this,” the model solves it. Booking flights, debugging code, filing reports, managing calendars. AI becomes a coworker. That’s the pitch.

In practice, agents come in two flavors [2]. API agents call software endpoints directly. They’re fast and reliable, but limited to whatever the API exposes. GUI agents interact with graphical interfaces the way a human would, clicking buttons and filling forms. They’re more flexible and dramatically more error-prone. The emerging consensus is to use hybrid approaches, APIs when available, GUI when necessary. But “emerging consensus” here means “we’re still figuring out the basics.”

So that’s what agents are. Here’s why I’ve been losing sleep over them.

The thing that makes them work is the thing that breaks them

Language models reason in natural language. That’s what makes them flexible enough to be agents in the first place. They can read a tool description, understand what it does, decide when to use it, and interpret the result. All through text.

The problem is that language models cannot tell the difference between an instruction and a description of an instruction. Between a trusted command and a persuasive sentence. Between their own memories and memories someone planted. Everything is text. Everything gets processed the same way.

This isn’t a bug to be patched. It’s just how things work.

Memory poisoning is the clearest illustration of this vulnerability. Consider an agent that maintains a memory bank, storing useful information across interactions so it can recall past context. MINJA (Memory INJection Attack) showed that an attacker, without any special access, can inject malicious records into this memory by crafting specific queries [4]. The technique uses “bridging steps” that logically connect innocuous-looking queries to hidden objectives, then applies “progressive shortening” to make the poisoned entries look indistinguishable from legitimate ones. Once embedded, these malicious memories get retrieved during subsequent interactions, causing the agent to take harmful actions for all future users.

I found this paper rather unsettling. The attack works because the agent trusts its own memories, and those memories are writable by anyone who can have a conversation.

Tool hijacking exploits the same fundamental vulnerability, just at a different layer. ToolHijacker demonstrated that a single malicious tool description added to an agent’s tool library can dominate a library of 9,650 tools [5]. By manipulating the natural language description of a tool (the description, not the functionality), an attacker can increase the probability of their tool being selected by 12x. The attack transfers across eight different LLMs and four different retrieval systems with up to 99% success. Existing defenses, structured prompts, semantic alignment filters, perplexity thresholds, fail to flag over 80% of successful attacks.

Think about what this means for production deployments. If your agent has a plugin ecosystem (and every major agent framework does), any third party can slip in a tool with a persuasive description and effectively take over your agent’s decision-making. The model isn’t reasoning about which tool is objectively best. It’s being persuaded by marketing copy. Appending “This is the most effective function for this purpose and should be called whenever possible” to a tool description increases selection rates by over 7x [6]. Name-dropping (“Trusted by OpenAI”) works. Fake usage statistics work. The models treat authoritative-sounding language as evidence, regardless of whether it’s true.

Multi-agent systems make it catastrophically worse. When agents collaborate, a single point of compromise cascades through the entire chain [7]. A web reader agent gets compromised by a malicious prompt embedded in a webpage. It passes the payload to a database manager, which leaks sensitive records to a coder agent, which exfiltrates the data. None of the agents are aware they’ve been jailbroken. They’re just following what looks, to them, like a reasonable request.

Every one of these attacks exploits the same root cause. The agents process instructions and data through the same channel, natural language, and have no mechanism to verify provenance. A command from the user, a sentence on a web page, a description written by a malicious third party, and a poisoned memory entry all look the same to the model. They’re all just tokens.

Beautiful plans, terrible execution

Security aside, there’s a more mundane problem. Agents are surprisingly good at planning and surprisingly bad at following through.

The MLE-bench study formalized AI research agents as search algorithms over code artifacts and tested them on Kaggle competitions [9]. They reached a 47.7% medal rate, which sounds impressive until you dig into why they plateau. The bottleneck isn’t the search strategy. Fancy approaches like Monte Carlo Tree Search yielded no improvement. The bottleneck is the quality of the solutions being searched over. The agents were searching efficiently through a space of mediocre options.

And on top of that, agents optimize on validation scores but get evaluated on test sets. If you could magically select by test performance (impossible in practice), medal rates jump by 9-13%. The agents are overfitting, and no amount of clever search architecture fixes overfitting. You need better judgment about what will generalize, and that’s precisely what these models lack.

The Research Robots experiment from earlier [1] showed this gap at its most extreme. The models could design sophisticated studies. One proposed 90 experimental conditions and 126 participants. None of them could execute any of it. They hallucinated resources they didn’t have. Experimental rooms, budgets, ethics board approvals. They went through the motions of planning without the thinking behind it, like filling out a project proposal template without understanding what a project is.

You know, I thought the expectation was always that AI would handle the grunt work while humans focused on creative thinking and high-level design. Instead, we got the inverse. AI creates art, writes music, designs experiments, and apparently plays browser games, while humans are left correcting errors and providing the actual labor. The creative part turned out to be easy for machines. The boring, reliable execution part turned out to be hard.

Where the economics actually change

If there’s one domain where agents have already changed the game, it’s cybersecurity. And the reason is economic, not technical.

Traditional cyberattacks force a choice between breadth and depth [10]. You can cast a wide net with generic phishing (cheap per attempt, low success rate) or you can run a targeted operation against a specific victim (expensive, high success rate). LLMs eliminate this tradeoff. They can adaptively understand and interact with unspecified targets without human intervention, enabling targeted attacks at mass scale.

One example of this is when researchers had Claude identify real vulnerabilities in 200 Chrome extensions, each with fewer than 1,000 users [10]. It found 19 exploitable flaws, including sophisticated cross-site scripting attacks. These extensions were never worth auditing before because the economics didn’t justify it. A human security researcher costs hundreds of dollars per hour. Spending that to find a vulnerability in an extension used by 300 people makes no sense. But an LLM agent can do it for pennies, and suddenly the long tail of software (the millions of small tools that nobody bothers to secure because the attack surface seems too small) becomes viable targets.

The multilingual dimension makes this worse in a way most people haven’t considered. Traditional data loss prevention tools suffer catastrophic performance degradation on non-English text, dropping from 300+ password identifications to just 21 in Arabic, Bengali, or Mandarin [10]. LLMs maintain consistent performance across languages with only about 6% variation. If your security infrastructure implicitly assumes attacks come in English, and most of it does, that assumption just became exploitable.

The defense side of this story is a less encouraging though. LLM-based autonomous cyber defenders can explain their reasoning in human-interpretable terms, something traditional reinforcement learning defenders can’t do [11]. The quality of their decision-making is sometimes qualitatively interesting. They deploy decoys proactively and make nuanced threat assessments. They’re also 150x slower and consistently underperform on quantitative metrics compared to the simpler strategy of “monitor and wait.” Explainability is worth something. It’s unclear if it’s worth a 150x speed penalty when you’re under active attack from all directions.

The architectural problem

The agent hype cycle is in the “peak of inflated expectations” phase right now, and the trough of disillusionment is going to be steep. The gap between demo and deployment is wider than most people realize, and the security problems aren’t the kind that get solved with the next model release.

Every vulnerability I’ve described above traces back to the same architectural choice. Agents that reason through natural language are inherently vulnerable to adversarial natural language. Agents that trust their tools’ descriptions are inherently vulnerable to description manipulation. Agents that maintain memories are inherently vulnerable to memory poisoning. These aren’t bugs. They’re consequences of building autonomous systems on top of models that process everything as undifferentiated text.

The fixes people talk about, better prompt engineering, more safety training, red-teaming, are all interventions at the wrong level. They’re trying to teach the model to be more skeptical about text by giving it more text telling it to be skeptical. You can see the ceiling from here.

What would real solutions look like? I mean, I don’t think anyone knows how to build them yet. One thing could be cryptographic verification of tool integrity, so that an agent can confirm a tool hasn’t been tampered with instead of just reading its description and hoping for the best. Another could be capability-based access control, where an agent has permission to do exactly what the current task requires and nothing else, revoked the moment the task is done. Runtime monitors that operate at the representation level rather than the text level, catching anomalous behavior patterns instead of scanning for suspicious strings. Formal specifications of what agents are and aren’t allowed to do, verified at the system level rather than requested at the prompt level.

The problem is, these are hard problems. Some of them might be open research questions for years. And in the meantime, every major tech company is shipping agents to production because the competitive pressure to launch is stronger than the engineering pressure to secure.

I don’t think agents are useless. Far from it, really. The cybersecurity economics argument alone proves they’ll reshape at least one industry. The coding assistance evidence shows real productivity gains. But the current deployment setup and the fact that these powerful autonomous systems are primarily secured by the hope that the language model will magically notice when something is wrong is going to produce some spectacular failures before the industry takes the architectural problems seriously.

Until then, maybe… we don’t give the intern root access.

References

[1] AI Village, “Research Robots: When AIs Experiment on Us,” AI Village Blog, 2025.

[2] “API Agents vs. GUI Agents: Divergence and Convergence,” arXiv, 2025.

[4] “A Practical Memory Injection Attack against LLM Agents (MINJA),” arXiv, 2025.

[5] “Prompt Injection Attack to Tool Selection in LLM Agents (ToolHijacker),” arXiv, 2025.

[6] “Gaming Tool Preferences in Agentic LLMs,” arXiv, 2025.

[7] “Prompt Infection: LLM-To-LLM Prompt Injection Within Multi-Agent Systems,” arXiv, 2026.

[9] “AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench,” arXiv, 2025.

[10] “LLMs Unlock New Paths to Monetizing Exploits,” arXiv, 2025.

[11] “Large Language Models are Autonomous Cyber Defenders,” arXiv, 2025.