The audit your AI workload is missing is not the one your vendor demo covers. Most teams I have looked at recently treat AI security as a separate discipline, hand it off to a model safety vendor, and move on. The dashboard comes back green. The production stack behind the model still has the same gaps it had the day the workload was stood up, and those gaps are where the data lives, the secrets live, and the blast radius lives. The interesting question is not whether the model is safe. The interesting question is whether the seven layers underneath the model are safe, because that is what an attacker who never touches the weights file will go after.
Below is the short version of how I think about reviewing an AI workload. It is not a framework for prompt injection. It is the same framework I would use for any new HTTP service I had to deploy next week.
The graph underneath the model
Most teams picture their AI stack as a chat interface sitting on top of a model. The actual production graph is closer to seven layers, and only the seventh one is the model itself. If you only review the model, you have reviewed about one seventh of the surface and told yourself you are done.
In the order a request actually moves through the system, those layers are the chat surface where untrusted input arrives and where output moderation runs, the gateway that handles auth and rate limits and the WAF (Web Application Firewall, a layer that filters malicious requests before they reach your service), the orchestration code that retrieves context and invokes the model and post processes the response, the retrieval store where the proprietary documents and the user history sit, the runtime plus weights where the inference work happens and where supply chain risk enters through a tampered artifact, the underlying platform of identity and secrets and networking and CI, and the humans who can reach any of the above. Skip any of those and the model being safe does not save you, because the attacker will find the path that does not go through the model.
Why this matters in practice is that the model is the easiest layer to think and talk about, which is exactly why it gets all the attention. The layers underneath are harder, slower to review, and look like every other production service you already have. That is also exactly why they are where the gaps end up, and why a security team that has not been told AI is special usually finds more real problems in a week than a specialist AI team finds in a quarter.
Two questions I have stopped skipping
After enough of these reviews I have settled on two questions that consistently surface the worst problems. Neither one is about the model.
The first is whether the weights and the inference server you shipped today are the ones you think you shipped. Pull a public model from a registry, skip the hash check, ship it, and you have given an attacker a path into your inference pipeline. The fix has existed in the open source world for years. Pin a hash, sign the artifact, verify the signature on deploy. It just does not get applied to model artifacts by default, which is why supply chain incidents on model artifacts keep happening in production stacks that passed every other review.
The second is what the model is allowed to do when it calls a tool. A model that can send email, query a database, or push to a code repository is a model that an attacker who controls the prompt can use to do all of those things. The honest fix is to give each tool the smallest possible scope, audit the chain, and treat every integration as a service account whose compromise has to be modeled. That thinking is not new. It is the same thinking you would apply to a CI job with write access to production. It just tends to get skipped on AI projects because everything about the project feels new and the team lets the novelty become an excuse for skipping the boring parts.
What a real review looks like for a week
If I had a week to look at an existing AI workload, this is the rough shape of the time. None of it is glamorous and none of it requires an AI background.
Days one and two go to the perimeter. Several specific things to chase down:
- Is the inference endpoint reachable from the public internet, and if so what auth sits in front of it. Is that auth a real auth layer or a placeholder token that someone pasted into an environment variable six months ago and never rotated.
- Are requests and completions captured by the same logging stack the rest of the fleet uses, or is the model the one workload whose logs nobody reads.
- What does the runtime identity have access to inside the host, and does that identity share secrets with anything else on the box.
- Does the runtime have a path to the metadata endpoint of the cloud it runs on, and if so can a prompt trick the model into reading it.
Days three and four go to the retrieval layer and the tool layer. Who can read the vector store. What identities does the model use to call it. Which tools is the model allowed to invoke, and what is the narrowest scope that still lets each tool function. If the model can hit email, calendars, code repos, and billing, then a single prompt injection can hit all four. That is not theoretical. It is the kind of thing that turns a paper vulnerability into a real incident in a weekend.
On day five you turn to supply chain. Where did the weights come from, and is there a hash pin. Where did the inference server image come from, and is it patched on the same cadence as your other images. Where did the orchestration code come from, and is the CI path that ships it locked down the same way your other CI paths are locked down. Most AI base images are six months out of date the moment they ship, because the GPU driver and CUDA (the GPU compute toolkit used to run model inference) version pins the image to whatever was current when someone installed it.
Day six goes to the humans. Who has production access to the AI workload, and is that access reviewed on a cadence. Who can reach the inference endpoint from inside the corporate network, and do they need to.
Day seven is writing up what you found. Half the value of a review is the document at the end, because the gaps you found will get reintroduced the next time someone ships a feature unless the findings are written down somewhere the next reviewer can find them.
Trade-offs
What is worth being honest about, before you start: hiring a separate AI security org is usually a signal that the existing team has been told the problem is not theirs, and the actual fix is to point the security team you already have at AI workloads on the same terms as everything else. A model safety vendor is a useful additional defense, not a substitute for the underlying work, and the failure mode I see most often is buying the vendor and skipping the rest. Open weights do not automatically mean safer. An open weight model you can audit has a different supply chain profile from a closed API, but the rest of the stack is identical, and the license is a separate question from the security posture. Prompt and completion logging is a privacy decision as well as a security decision, because capturing user prompts is the cheapest way to spot prompt injection campaigns but the same logs become a privacy liability the moment they are retained past their usefulness. Finally, a compromised AI workload usually has a larger blast radius than a compromised web app, because a web app might read one database while a compromised AI workload can reach every system it has a tool for, which in a well-instrumented stack is a lot of systems and worth treating as a real architectural decision rather than a footnote.
The honest summary: the first week of an AI workload should look like the first week of any other public service, and if you are inheriting one, the most useful thing you can do today is the same walkthrough applied to what is already running, with the working assumption that the answers are worse than the team thinks. In my experience they almost always are.