Moore, the chief research officer of Dragos, sent nearly two million AI prompts containing unique, trackable secrets to AI providers to see where those secrets surfaced. He wrote that the findings revealed security breaches, prompt theft, and widespread failures to protect customer data. The investigation found that at least three AI providers were compromised. Attackers siphoned prompts. Multiple resellers in the inference marketplace logged the prompts and passed them to people who use the credentials. Several AI providers did not follow end-user privacy settings like Zero Data Retention or "no data collection". A privacy flag does not help when an attacker is already inside.
Compromised hosting and routing infrastructure
Prompts are encrypted in transit, but they are decrypted at each hop inside the large inference reseller network. An AI router reads the plaintext in a prompt to determine which inference providers to handle the request. Prompts move to a pool of available models. One router, such as Vercel, might send a prompt to another router, such as OpenRouter. Each player can log the prompt.
The breach of three separate hosting providers was clearly the most concerning finding. One provider was compromised by attackers who reused "a couple hundred" AWS keys. In one instance, attackers who reused a couple hundred AWS keys copied requests at the provider’s front door before a model ever ran to access S3 buckets, secrets managers, and parameter stores. Each key was sent in exactly one request to one provider in late October, and every single request sent during those windows was stolen. Two other breaches involved stolen requests marked as ZDR. Parties tied to the hack tested each key to see what resources it gave access to, sometimes within hours of the keys being seeded. These attackers looked through S3 buckets, secrets managers, and parameter stores in search of still more credentials. The stolen bait keys were used "tens of thousands" of times in AWS API calls to enable hosted models in AWS Bedrock and add IAM users to maintain access to the systems.
