AI Security and Privacy Checklist for Production Systems

Most AI privacy incidents are not exotic attacks — they are ordinary data leaks through a new path. Before choosing controls, write down what data enters the model (prompts, retrieved documents, user context), what leaves it (responses, tool calls, logs), and where copies persist (caches, vector stores, provider logs). If you cannot draw this map on one page, you are not ready to ship.

Classify every field on that map: public, internal, confidential, regulated. The classification drives retention rules, redaction, and which providers may process it. Skip this step and every later control is guesswork.

Treat the model as an untrusted intermediary between users and your systems. API keys for tools should be scoped to the minimum actions the workflow needs, separated per environment, and rotatable without a redeploy. Retrieval should enforce the same document permissions your source systems already have — an assistant that bypasses your access control model is a privilege-escalation path.

For anything the system can do — send, delete, purchase, publish — require an explicit permission boundary. Read-only by default; write access only where the workflow genuinely needs it, with approval steps for irreversible actions.

User input, retrieved content, and tool responses all become part of the model's instructions. Prompt injection works because systems treat these as trusted text. Constrain what the model can be asked to do with system-level instructions, strip or quote untrusted content, and validate structured outputs against a schema before acting on them.

Rate-limit and authenticate every endpoint the feature exposes. An unauthenticated prompt endpoint is an invitation to run up your provider bill and probe your system prompts.

You need enough logging to answer "what did the system do, with what inputs, and why" for any single action — and no more. Log inputs, outputs, tool calls, latency, and cost per request; hash or redact personal data in logs; set retention windows and honor them.

Finally, rehearse the stop button. Every production AI feature needs a documented way to disable it, fall back to a manual path, and communicate the change to users. The first time you need it is the wrong time to design it.