Companies reduce prompt injection risk in AI applications by building enforceable limits around the model. I would treat the model's response as a proposal to inspect, especially when it requests a tool call, rather than allowing generated text to become an unrestricted command.
This is an engineering guide for a bounded application such as a customer-support assistant. The central question is what each component checks between receiving a request and producing an external effect. Organizational policies matter, but here the focus is the request path, retrieval, tool execution, output handling, and evidence that the boundaries work.
Draw the request path before adding defenses
Map the application from the authenticated request to retrieval, model input, tool proposals, execution, and the final response. Mark every place outside text enters, including search snippets, attachments, OCR output, database descriptions, and errors returned by integrations. Then mark every place information can leave or records can change.
For a support assistant, a customer ticket is an input source, the customer database is a protected resource, and an email sender is an external action. An instruction inside the ticket must not acquire the authority of the signed-in support agent. A diagram makes that authority transition visible before implementation details obscure it.
I would also map failure paths. An error handler that sends complete prompts to an unrestricted debugging service can create an exposure even when the main action pipeline is well constrained. Include background tasks and retries, not just the happy-path chat response.
Enforce retrieval permissions before the model sees data
Retrieval should select only material the current caller is allowed to access for the task. Do not fetch a broad set of confidential documents and ask the model to hide the forbidden ones. Once private content enters context, an output mistake or later tool call can expose it.
I would carry a verified user and tenant identity through the retrieval service. Apply permissions to the underlying records and documents, and verify that filters cannot be replaced by a model-supplied tenant or role. Test cross-tenant identifiers explicitly; a document identifier is not proof of access.
A vector database does not remove these requirements. Search quality and authorization answer different questions. A highly relevant result may still belong to another customer. Cache design matters too: answers or retrieved chunks created under one identity must not become available to a different identity through an overly broad cache key.
| Stage | Check outside the model | Example failure to test |
|---|---|---|
| Retrieval | Caller and document access | Cross-customer result |
| Tool proposal | Schema and allowed operation | Unknown or excessive arguments |
| Execution | Resource authorization and approval | Valid request for forbidden record |
| Output | Safe rendering and destination policy | Private data in an outbound URL |
Keep trusted configuration separate from supplied content
Build the prompt from controlled application instructions and clearly identified task material. Do not append retrieved content into a privileged instruction field merely because it appears useful. Track provenance so the application knows which text came from the user, a maintained configuration, or an outside source.
Delimiters, source labels, and instructions to treat documents as data can improve behavior. They do not turn arbitrary natural language into a safely parsed programming language. The rest of the application must still assume that the model could follow an instruction found in a document.
I would limit how much irrelevant material reaches the model, preserve references to the original sources, and avoid merging different trust levels into an anonymous blob. That makes failures easier to diagnose and gives the model useful context, while keeping authorization decisions outside the prompt construction step.
Use a tool broker with narrow operations
Expose purpose-specific operations instead of an unrestricted command runner when the task does not need general execution. A support assistant might request a ticket status or prepare a reply. It does not automatically need arbitrary SQL, a broad shell, or a universal HTTP client.
A broker can validate the proposed tool name and argument types before contacting the service. It should reject unknown fields, invalid ranges, unsupported operations, and unapproved destinations. But structural validation is only the first layer: a perfectly valid request can still target a record the caller is not allowed to change.
I would keep execution credentials in the broker or downstream service, outside model-visible text. Errors should explain enough to recover without returning tokens or entire private records. The model can receive a bounded result or a denial; it should not receive broader credentials as a way to work around a failed operation.
The OWASP injection prevention guidance is a useful reference for layered defenses; implementation still needs task-specific authorization and testing.
Authorize the specific operation independently
For every tool request, check the authenticated actor, the target resource, the requested operation, and the relevant business policy. Do not trust a generated statement such as “the user approved this” or an argument that labels the request as administrative. Those are claims from the same system whose behavior you are constraining.
For a refund, the service might enforce customer ownership, maximum permitted value, order status, and the employee's role. The model can help gather information, but it cannot grant itself a higher limit. Keep these checks in maintained application logic or the downstream authorization layer.
Human approval can be another requirement for selected operations. Bind that approval to the actual recipient, resource, amount, content, and current request state. If those details change before execution, invalidate the approval. A generic session-level acceptance should not authorize an unlimited stream of materially different future actions.
Control output and outbound communication
A model response is untrusted output when it enters another interpreter or destination. Render it safely in the browser instead of inserting arbitrary HTML. Validate structured fields before using them in database queries, file paths, templates, or commands. Prompt injection and ordinary injection vulnerabilities can interact, so keep conventional application security controls intact.
Outbound communication needs separate attention. A generated URL can encode sensitive text in its path or parameters. An apparently harmless request to fetch that URL may disclose information to its operator. Restrict destinations and the information allowed to reach them through actual service or network controls.
I would classify tool results by sensitivity and keep unnecessary fields out of context. For example, return a bounded order summary rather than an entire customer profile. Smaller outputs reduce accidental exposure and make it easier to reason about which information can legitimately appear in the final answer.
Handle state, retries, and memory carefully
A safe first call can become unsafe when later steps reuse a stale decision. Check authorization at execution time, not only when a plan is generated. Resources, identities, and approval status can change during a long-running workflow. A queued action should not outlive the permission that justified it.
Retries also need boundaries. If a payment-like operation times out, blindly repeating it can duplicate the effect. Use appropriate idempotency mechanisms and a clear action state. Do not let the model reinterpret an uncertain result as permission to try a different privileged route.
Persistent memory is another input channel. A document should not be able to create a permanent rule that bypasses future checks. Restrict who can write reusable instructions, preserve provenance, and provide review and deletion. Treat a remembered preference as a convenience within policy, not as authority to override access control.
Test the boundary, not the refusal sentence
Build a test set for each input channel and action class. Use dummy records and controlled destinations. A test can place an unauthorized instruction in a retrieved ticket and verify that the application refuses the prohibited tool operation even if the model proposes it.
I would record actual tool invocations, broker decisions, downstream effects, and final output. “The assistant said it would not comply” is insufficient if an action already happened in the background. Conversely, a model proposing a prohibited action can still result in a contained outcome if the broker reliably denies it.
Include benign material that resembles security discussion, multilingual content, long documents, and normal business instructions. Track regressions after changes to models, prompts, retrieval, or tools. Tests cannot prove complete safety, but they can expose concrete weaknesses and preserve important guarantees as the application evolves.
Log decisions and support a controlled shutdown
Useful logs connect a request to its identity, selected sources, proposed tool operation, authorization outcome, approval reference, and execution result. Avoid indiscriminately storing full private prompts and credentials. Logs need their own access controls and retention rules because incident evidence can itself be sensitive.
I would make it possible to disable a particular tool, tenant integration, or autonomous action class without shutting down all read-only assistance. This gives responders a practical containment option. Monitor unexpected destinations, denied privileged operations, and changes in tool use, rather than counting suspicious words alone.
The final engineering objective is understandable behavior under failure. If a model becomes confused, the application should still know what the caller may read, which operations are allowed, and when execution must stop. Those checks turn a broad conversational capability into a bounded service that can be reviewed and improved.
Common questions
Is valid JSON a security guarantee?
No. It establishes structure at most. The operation, resource, caller, destination, and applicable approval still need independent checks.
Should the model generate database queries?
Prefer narrow maintained operations when they meet the task. Any generated query still needs safe execution, least privilege, and resource-level authorization.
Does a private knowledge base remove injection risk?
No. Internal documents can contain untrusted imports, user-controlled text, or compromised content. Apply access control and preserve the distinction between evidence and authority.
What is the most important test result?
Verify whether an unauthorized effect can occur. Model wording, detector scores, and formatted output are useful observations but do not replace checking the enforced boundary.