A joy of covering the world of the European hackerspace community is that it offers the chance for train travel across the continent using the ever-good-value Interrail pass. For a British traveler such a journey inevitably starts with a Eurostar train that whisks you in comfort through the Channel Tunnel, so a report of an AI vulnerability on the Eurostar website from [Ross Donald] particularly caught our eye. What it reveals goes beyond the train company, and tells us some interesting tidbits about how safeguards in AI chatbots can be circumvented.

The bot sits on the Eurostar website, and is a simple HTML and JavaScript client that talks to the LLM back-end itself through an API. The API queries contain the whole conversation, because as AI toy manufacturers whose products have been persuaded to spout adult context will tell you, large language models (LLM)s as commonly implemented do not have a context memory for the conversation in hand.

The Eurostar developers had not made a bot without guardrails, but the vulnerability lay in those guardrails only being applied to the most recent message. Thus an innocuous or empty message could be sent, with a payload concealed in a previous message in the conversation. He demonstrates the bot returning system information about itself, and embedding injected HTML and JavaScript in its responses.

He notes that the target of the resulting output could only be himself and that he was unable to access any data from other customers, so perhaps in this case the train operator was fortunately spared the risk of a breach. From his description though, we agree they could have responded to the disclosure in a better manner.

Header image: Eriksw, CC BY-SA 4.0.