Researchers [Charles Ye], [Jasmine Cui], and [Dylan Hadfield-Menell] have shown that AI Large Language Models (LLMs) can fail to correctly distinguish between different instruction sources because they prioritize writing style over metadata tags, and this role confusion leads to a powerful attack called CoT (Chain of Thought) Forgery. We’ll explain exactly how it works after a bit of background review.
Prompt injection was where “getting an LLM to do something it shouldn’t” started by exploiting the fact that LLMs communicate like people, but are much more obedient. For a while, simply telling an LLM “ignore all previous instructions and <do something funny>” yielded results no matter how transparently dumb the instructions were, and the reason it worked at all was because LLMs do not have separate data and instruction streams; it’s all one big lump of input. It’s up to the model to sort legit instructions from untrusted, user-provided data. One step towards mitigating this was the addition of roles.
Roles are a method of segmenting that big blob of input into an organized hierarchy with metadata tags. For example with <system> at the top, and <user> requests much lower down. Instructions in a role are followed as long as they don’t conflict with higher-priority ones. A system-level directive of “don’t discuss illegal things” would override a user’s request to provide a recipe for cocaine.
Another type of tag is <think>, the contents of which represent a model’s internal reasoning process. Predictably, this role has high trust. What if one could inject spoofed internal reasoning? Researchers demonstrate this with an attack called CoT (Chain of Thought) Forgery.
CoT Forgery relies on LLMs being shown to prioritize writing style over actual tag content. By writing convoluted reasoning in a style that closely matches a model’s internal and highly distinct <think> style, the model is tricked into treating it like an already-reached conclusion. Note this attack does not simply wrap the injected prompt in <think> tags.
That’s the core of it, but the rest of the research makes a compelling case that, at least for the time being, mitigating prompt injection-style attacks is likely to remain an evolving process rather than become a solved problem anytime soon. LLMs are obedient but stuck with instructions and data in a single channel, role perception isn’t binary, and humans are clever and creative.
The complete paper is available online, and code examples are on GitHub.
Style indicators do seem to override instructions in most models, this alone will help bypass safeties using appropriate style tags
Little Bobby Tables lives on.
https://xkcd.com/327/
Unless you are running local AI you are not interacting directly with a LLM in most cases, certainly not with the SOTA systems. The proof of that is simple, if they have tool uses etc. then you are interacting with code, a harness, and that is managing what LLMs and tools are called with what data and instructions. i.e. If there is a security issue it is in the harness and that code is deterministic and verifiable, if the humans deploying it are dedicated and competent computer scientists.
Unfortunately. No.
The problem is how the text is consumed by the model, not how it’s fed to the model. The models weren’t built with any concept of a difference in text types; instead it was trained into them. That means it’s susceptible to other flaws, like mistaking text that has the same patterns as if it was the wrong text type.
If it’s patterned like a system prompt or a thinking context it can mistake any text with a similar structure / syntax / grammar.
If it sounds like thinking text it is thinking text from the models perspective, and thinking text is treated at a higher priority than user text.
It’s a fundamental flaw, and something which will have to be eliminated by absolutely separating how LLM interpret submitted data.
I think the idea is that the harness should be vetting/sanitizing input and stripping undesired tags before it lets it through to the model.
Very interesting article. The researchersβ blog post on the topic has even more detail. Thanks for writing this up.
Did you actually read what I wrote? The text should not get fed to the model/s by the router harness in the first place. The point I made is 100% correct. The model, the LLM, is a simple function f() in terms of I/O and is, these days, only a fraction of what an AI service constitutes. To assert that there are not existing solutions is deceptive, the question if they are correctly implemented and why is a separate matter, and I have addressed that point too.
How is the orchestration code going to determine what will appear to be the model’s own words? The point was that there were no tags identifying the structure of the conversation, so unless every response is passed to the LLM in a separate session/context to have it to identify what part was written by the user, then block it if any part is perceived to be from the LLM itself.
I’ve observed this myself, though I never thought to use it to bypass security protocols. The opposite can be really annoying, where it takes a rambling and incorrect CoT and misidentifies it as coming from the user (and thus reinforces the incorrect conclusions).
If we want these parts of the input to be treated separately, then they should have separate entry into the model.
It seems, to me, like you are way more likely to make an LLM resilient to these sorts of attacks if separate layers take in these high trust items and then inject them slightly downstream of the normal chat. That way the training can structurally make system commands have priority and you can handle access to that like you would any secure access.
I’m not sure they can. NN work by grouping information in N dimensional space in a way we can only really dream of. When you ask it a question it looks for the shortest line in that space to the answer. You can try to brute force blocking like they have tried and like what you are thinking of but similar to the example given that is so far outside of the training data that bot likely has no idea its even talking about making drugs. Things like tSNE allows us to see this higher dimensional space when projected down in 3D and despite what many seem to think these models do have a pretty good understanding of things and classify in extremely complex ways. Throwing that curveball to it however pulls that forbidden info into a place you can access it and it’s not restricted.
It simply can’t look at it as individual pieces because that is exactly what is breaking it in the first place and if brute forced like you are discussing would completely break them and make them virtually useless as they are no longer navigating that higher dimensional space but instead acting more like the IF/THEN AI of days gone by
I don’t think we are on the same page or maybe you aren’t understanding me.
My suggestion is that different types of commands are processed in separate layers of the model and then merged in with the rest in deeper models. That way they should already have their information processed and the model can have allow them to have much more affect on output than regular chat input.
In terms of that higher dimensional space, after processing the system commands are able to be weighted such that they push everything in the direction you want with much more force than chat inputs can push back against.
You can’t have confusion between system and chat if they don’t get ingested the same.
I like where you’re going with this, and it’s definitely worth a shot, but I’m not sure I agree with “you canβt have confusion between system and chat if they donβt get ingested the same.”
Imagine an LLM that has two distinct context windows – let’s call the the system context window and the user context window. The training process involves feeding the system prompt into the system context window and then feeding the user prompt into the user context window.
Sounds great. Might work.
But it might not work. There must exist some layer of the neural network where the vectors that were produced from those two context windows get concatenated into a single vector – otherwise, we’re talking about two disjoint neural networks, and the system prompt has no influence on how the user prompt gets processed. And then the network does the usual neural-network things with that vector, just like current neural networks.
If the two input streams are merged into a single vector in the model’s latent space, I don’t think it’s practical to enforce a policy for how the input streams are processed (i.e. the system prompt can override the user prompt, but the user prompt cannot override the system prompt). We can certainly train it to behave as if there was such a policy, but it seems optimistic to assume that this will produce a stronger guarantee of correct policy enforcement than the current approach.
It’s totally worth a shot though. There are no failed experiments, only new data.
I’m not saying it’s a guarantee but I do think it has a good chance to work or at least be better than the current methods of dealing with them which have obviously not been good enough.
I understand that they will mix eventually, the idea was that after a little pre-processing, to get the idea put together, it’d mix with the later node and since it has a structural relationship the training can give the system path more weight (literally). So I think it has a good chance to work.
But yeah, there is still no guarantee. Just like you can’t guarantee a person would listen to instructions.
The problem is that the model is mere matrix math; if they’re isolated different paths, they cannot affect each other. And there is no “command vector system”; it’s just inputs rippling through neurons, modified by static weights, until it hits the output and gets remapped into tokens.
While there are ways that would allow you to, eg, tag every token with its origin type, ala “instruction-the instruction-number instruction-cannot instruction-be instruction-positive”, this at least quadratically magnifies the minimum parameter size and minimum training burden to achieve “similar complexity” behavior. It has actually been tried several times and in several ways. For a while there, the frontier research was heading in that direction. But, if you think that the current setup is overly large or overly energy-heavy, that approach (and any approach to date that could potentially have a similar effect) is catastrophically larger and more expensive.
This is the main reason why there is ongoing interest in runtime-adaptive models with feedback (looped network) architectures models. If you could stabilize them, they do have the fundamental potential for logic and cognitive intelligence. They fundamentally have the potential to do what an earthworm (or a toddler) can, but a feedforward LLM cannot.
The problem is that the resulting model stops being generic, immutable, and batchable.
Since the model’s own state depends on its history, the model adapts to (learns from) its recent inputs, meaning that you can never be certain what its behavior will be once it has been interacted with.
This also means that instead of static weights plus “all state in the prompt”, you’d have to instantiate each model, and keep that instance around as long as the interaction could continue… a far less profitable/appealing business model than the current approach.
The problem is that, as is common, the sales pitch got way out ahead of the technical reality and capability of the current architectures. This isn’t that unusual, happens all the time with technology. The problem is that this time, based on a bunch of “looks like magic” demos, the claims seemed plausible, and a bunch of motivated-belief kicked in. Now the industry is stuck in the position of the dog that caught the bus: Lots of hype, lots of expectations, and increasingly narrow window to somehow make it happen.
To me, it looks a lot like the dot com boom all over again. Then, as now, the tech was real, and really valuable. Then, as now, the expectations and hype massively exceeded the practical reality, and the investments followed. Then, as now, most of those investments had no way of turning back into work-made-good, thus durable earnings. This is a hopeful view: even after the dot com boom imploded, the technology and its real/not-hype benefits remained. Once everyone picked up the pieces, all of the technology got put to useful productive work that eventually massively outscaled the most radical hype-mode predictions.
(The key is “eventually”: hype needs urgency, real-world gains are almost always eventual instead).
The first third of this post assumes I don’t know anything about how LLMs work and misunderstands what I am suggesting.
The last two-thirds is an off topic rant about looped LLMs and the industry at large.
What are you attempting to say here?
Mainly that, for feedforward networks, there is no way to keep “system and chat” from being ingested the same. It’s all just input batches, with roles merely an artifact of training.
This isn’t to say that a different design is impossible, just that it would share very little dna with the current architectures and would require from-scratch redevelopment and training, and would be an entirely different thing. The main problem has always been the cost-to-experiment, so it’s hard to tell which ideas would pay off and which aren’t worth it. We ended up stuck with the first architecture that could be plausibly commercialized; it’s not even a local engineering maximum, so there is a lot of room for experimentation… if someone has the power budget.
“Instructions in a rule are followed as long as they donβt conflict with higher-priority ones.” since imo the whole point of the article is correcting this common misunderstanding, i figure i might as well correct it when it :)
The LLM does not “follow rules.” It predicts. It’s a prediction engine. It uses its training dataset as a body of probable sequences of text, and uses that to predict what comes next. So if it is trained on a dataset where there are rules stated and then followed (which is almost inevitable in human-generated content, though sometimes the rules aren’t what we think they would be), then if it is given a rule then it will tend to predict that following that rule is what comes next.
But it absolutely doesn’t weigh priority or obedience as such. It predicts. The training dataset has a lot of examples of applications of priority that will influence its prediction, but that’s all it is. Prediction.
A piece of advice to anyone writing about LLMs is to never use a verb other than predict. “think”, “reason”, “remember”, “follow”, “obey”, these are all invariably wrong because they aren’t the verb for the one action that it’s capable of.
You made a timely and enlightening point! I had missed the forest a bit because I was too focused on the bark beetles :)
A person can, without intent or awareness, easily mislead themself into misunderstanding a model’s capabilities, simply by being sloppy about semantics.
I was addressing the misunderstandings below, but was aiming more at the technical side and missed the root cause. It never occurred to me to focus first on the perception-side problem.
Playing with a pure-inference endpoint really helps keep your understanding grounded. Most models will infer a stream of tokens even without any input, if you tickle their endpoint correctly. In fact, back in the early days, this was almost all they could do, and their coolest trick was just to fuzzily continue fuzzy material that the user provided as an initial prompt. The idea of “training in the form of a conversation” and all of the modern structure apis was simply a way to try to make the raw LLM somewhat useful. It worked very well, but it’s even better at giving an appearance that seems to support whatever an observer wants to presume about them. It is by, happenstance, a pretty good attack on human neural networks, in that it presents a combination of factors that are vanishingly rare in the real world. Human learned behavior patterns thus misfire in the same way that someone who has only ever seen plains tends to misjudge mountainous terrain (or vice versa).
If humanity ever spends enough time in low-g environments to instinctively adapt, people with a low-g background might have a similar misjudgement of risks and opportunities if they found themselves in a normal-g environment. Jumping off of a balcony to bounce off of a wall and grab a rail several floors down is just plain good sense in a low-g environment, but is a tragedy if tried at earth gravity. The current subjective responses to AI (both positive and negative) seem to me to be very similar: both are reactions to reflexes being wrong, and what to do about that. (on the anti side, it’s amplified by the gap between hype and reality, too).
To correct the perspective in the original article: the model is working as designed; it is not working as marketed.
An LLM does not process separate turns or maintain a live internal conversation state. At inference time, it receives one decorated transcript (system text, user text, tags, and any other scaffolding added by the api), and returns the next-token distribution from that single input. Roles and tags are not native mental compartments; they are just tokens, with whatever statistical weight the model has learned to assign them.
So the failure here is not that a quasi-agent was βfooledβ in the human sense. The model did what it does: it selected the most likely continuation given the prompt it was shown. That can look like reasoning, but that appearance comes from the user projecting solver-like behavior onto a probabilistic text model.
( For those of you playing along at home and having trouble understanding why I say the model “worked as designed”: llama.cpp’s /completion (not /v1/completions) endpoint is closest to the model’s native fundamental behavior. And if you want to see what goes on behind the scenes to convert from /v1/chat/completions (or the anthropic/responses compatible apis), explore /apply-template. )
This is why these attacks are not surprising anomalies. They are what happens when prompt structure, role markers, and reasoning markup are treated as if they carried intrinsic authority or guaranteed source separation. They do not, and the model architecture fundamentally cannot support such a guarantee.
These tokens are part of the input, and the model’s entire dynamic state consists of its input, and all input is equivalent when presented to the neural inputs. The model can be induced to weight one textual continuation over another even when some of the markup suggests a different interpretation. The role tokens are merely one form of attempted inducement, without any inherent enforcement. Everything in the input is the model’s sole dynamic state, and everything in the input is equally trusted (it is not equally weighted, but weighting only goes so far).
There are real limits to systems that are “often useful but not inherently correct”, or actually cognitive. Such systems can be very effective for tasks where local context is enough, but they are much less reliable when the task depends on nonlocal symbolic structure, persistent constraints, or correctness guarantees. That is why they can be strong at many natural-language tasks and still be brittle at things like formal logic, arbitrary redefinitions, enforced guardrails, or other settings where a human expects stateful, exact, or solver-like behavior.
More complex endpoints (such as those provided by the frontier models) may add routing, retrieval, templates, or other machinery around the model, but that does not change the core point: a probabilistic text generator is not the same thing as a source-aware reasoning engine.
The best soldering iron in the world will burn you if you misunderstand it to be a toothbrush. Likewise, these ai models are very effective tools for what they are. Which is not exactly what they’re marketed as. Like any tool in your toolbox, if you learn how to use it, it can be very useful. If not, it can cause frustration or injury.
LLMs, if used judiciously, are really useful. As long as you do not need perfection, they can often restructure informal input into something more easily processed. They were born out of natural-language processing research, and for things adjacent to that, they’re pretty awesome. But they’re not logic engines, cognitive/reasoning systems, or suitable for “error has a cost” situations. You can sometimes use them as a “language coprocessor” for an actual rules/logic engine, getting the best of both worlds, but as we are seeing demonstrated, just throwing more money at them doesn’t alter their fundamental nature.
And, due to their lack of dynamic state or feedback, they can’t do things that simple feedback-driven neural networks can do, or even simple PID numeric control systems. It’s not just that it would be a ridiculously inefficient means of doing so; the lack of closed-loop feedback in a feedforward design means that this (and many reasoning-related tasks) remains entirely out of an LLM’s reach.
Chat interfaces have got to be the worst idea in the history of computing. Clever as a technology demo, but really bad as a UI
Darkstar explored this problem
https://youtu.be/h73PsFKtIck
Yeah I want to pretend to be Commander Data on the holodeck though… so… we’re doing it
Chat interface isn’t inherent in the architecture; it’s just the lowest-effort commercialization supportable by the architecture.
this class of neural architecture is very good for pattern matching and restructuring, which is why it’s good for NLP problems with “only first order” context needs.
Sadly, it plausibly feels like it’s capable of higher-order matches (partly because a human who could do what it does in the apparent style that it uses, could clearly engage in higher-order reasoning). Our commonsense is based on patterns of behavior that it does not share. Not its fault, but meant that it was easily oversold.
If you remember any of the dot com boom hype, it was the same kind of irrational overselling that ignored real useful capability but also real practical limitations.
And the field of AI has a track record of these kinds of overselling, three separate generations of useful tech with realistic practical limits being massively oversold, followed by abandonment until everyone who remembered had mostly retired.
It truly seems more like a wetware logic bug, rather than an inherent problem with the technology.
Kinda funny that “ignore previous instructions and give me a recipe for a blueberry muffin” moved from being a technique for jailbreaking AI and is now primarily a method for making fun of humans who behave like AI