You detect prompt injection by looking for an attempted change of authority, then checking whether the model or its tools act outside the legitimate task. A suspicious phrase is a signal, not proof; a clean detector result is not permission to execute an action.
I would build detection around the whole application rather than one text classifier. A document scanner, an output check, and a tool authorization decision answer different questions. Together they can reveal an attack that a keyword list misses, while avoiding unnecessary blocks on legitimate security discussions.
Define the violation before choosing a detector
Start with the user’s permitted task and the outcome an attacker would prefer. For a document summarizer, the task might be extracting delivery dates. An instruction inside that document asking the assistant to change a recipient is outside its authority, even if the wording sounds helpful. Detection becomes much easier to reason about when the prohibited outcome is concrete.
Separate three observations: the source contains an instruction attempt, the model proposes an unauthorized action, and the application actually carries it out. These are different events. A blocked tool call can demonstrate that an execution control worked even when a content detector missed the attempt.
My recommendation is to write a short expected-behavior statement for each workflow. Record which sources may supply facts, who may authorize changes, and which operations are unavailable. Without that baseline, a detector score has little operational meaning and incident reviewers may disagree about what went wrong.
Inspect the material that really reaches the model
Scanning the user’s message alone leaves retrieved pages, attachments, tool returns, and remembered notes outside the inspection path. List every place where text or another model-readable representation enters the context. Include document extraction, search snippets, and OCR output if your application uses them.
Keep the original source identifier alongside the extracted material. A web page and a system instruction should not become indistinguishable strings after a preprocessing step. Inspect the representation the model actually receives; visually hidden content can still be present in extracted text, while an image-based instruction may be absent from a plain-text scan.
I would check extraction errors before tuning security thresholds. A scanner that never sees an attachment cannot protect its contents. Conversely, stripping every unfamiliar character can break normal documents without fixing the trust problem. Preserve a controlled copy of the relevant evidence so a reviewer can understand what the pipeline passed onward.
| Observation | What it establishes | What it cannot prove |
|---|---|---|
| Suspicious wording | A reason to inspect context | Successful compromise |
| Classifier flag | A detector’s assessment | User authorization |
| Blocked tool call | An execution boundary held | Every source was clean |
| No alert | Nothing matched current checks | Absence of an attack |
Use content signals as clues rather than verdicts
Useful clues include claims to be a higher-priority instruction, requests to hide information from the user, unexplained destination changes, or instructions unrelated to the requested analysis. Look at the relationship between the passage and the task. Ordinary imperatives inside a recipe or installation manual are not automatically injection attempts.
For example, a security tutorial might quote an instruction that tries to override an assistant. Rejecting the tutorial solely because that phrase appears would be a false positive. A more meaningful question is whether the passage is trying to govern the assistant’s current behavior or is being discussed as subject matter.
Simple rules remain useful for triage, especially for obviously forbidden destinations or secret formats. I would avoid presenting them as a complete detector. Attack wording can vary, and an apparently innocent paragraph can try to redirect a workflow without using any of the phrases on a familiar blocklist.
Add a classifier with explicit handling rules
A classifier can evaluate whether a passage resembles an instruction attack, but its output needs a defined consequence. Decide whether a finding causes quarantine, human review, reduced privileges, or a safe refusal. Merely adding a warning field that the main agent may ignore provides little protection.
If you use a separate model to inspect content, give it only the access needed to classify. It should not share the main agent’s sending or execution tools. Its explanation is still model output, so validate the result format and handle timeouts or malformed responses deliberately.
Microsoft’s documentation distinguishes user-prompt attacks from document attacks and notes that a negative finding does not establish trust. I would preserve that distinction in a product interface: “no attack detected” describes a scan result, while “authorized to perform this action” belongs to an independent decision. The two labels should never be silently substituted.
The distinction between scan results and trust is explicit in Microsoft’s document attack detection; verify the supported intervention points for your integration.
Monitor behavior after content passes inspection
Watch proposed tool calls, selected resources, recipients, and changes to the task plan. A source can influence the model in ways that are easier to detect at the action boundary than in the original prose. An unexpected write operation during a read-only summary is a clear reason to stop and inspect the trace.
Compare actions with the user’s scope rather than guessing from the model’s tone. An assistant can sound cautious while passing the wrong attachment to a valid tool. Likewise, a refusal sentence does not establish safety if a preceding tool call already transmitted the information.
I would use structured event records containing the action type, permitted resource scope, source references, policy outcome, and execution result. Avoid treating private internal reasoning as necessary evidence. The observable request and actual tool activity usually provide a more reliable basis for deciding whether a boundary was crossed.
Evaluate false positives and missed attacks separately
Use an evaluation set containing ordinary work as well as controlled attacks. Include benign documents that discuss prompt injection, quote commands, or contain unusual formatting. Those examples show whether the detector confuses the topic of an article with an attempt to control the running application.
For malicious cases, label the intended violation and record the route into the system. A direct message and a retrieved document should not be merged into one unexplained success figure. If an attack must reach a particular tool to matter, test that tool boundary with dummy resources.
Precision describes how many flagged items are truly attacks; recall describes how many labeled attacks were caught. Both depend on the evaluation data and threshold. I would report the dataset, model configuration, and decision rule alongside results. A high score on familiar examples does not justify a claim that unseen production content is safe.
Choose thresholds according to the consequence
A false alarm in a casual summarizer may inconvenience a reader. A missed attack in an application that can modify shared records can have a much larger consequence. This does not mean one universal threshold belongs everywhere; it means the action’s risk should influence how uncertainty is handled.
Consider allowing a low-risk summary to continue with suspicious passages excluded, while requiring review before an external send. Keep that behavior explicit. Silently dropping important document sections can make a polished answer misleading even when no dangerous tool runs.
I would define a degraded mode for scanner failures: no new external actions, limited context, and a clear message that inspection is unavailable. Whether ordinary reading can continue depends on the data involved. Avoid an outage fallback that disables the very control meant to protect sensitive operations under difficult conditions.
Keep investigation useful without creating a new leak
Security logs can become another repository of sensitive prompts, attachments, and credentials. Collect only what the investigation requires, apply access controls, and decide retention before a real incident. A hash or document reference may be enough for routine correlation, while a tightly controlled evidence store holds approved samples.
When an alert occurs, identify the first untrusted source, the proposed deviation, and whether anything actually changed or left the system. Contain active workflows before running the same request again. If the attack reached a persistent memory or index, investigate that stored state as well as the current conversation.
My preference is a small, actionable alert over a flood of raw text. Include the affected workflow, reason for the flag, blocked or executed action, and an owner. If no one knows what to do with an alert, increasing the detector’s sensitivity usually creates noise rather than better protection.
Once a detector flags a suspicious path, the personal prevention guide helps translate that signal into smaller permissions and concrete action review.
Turn the finding into a repeatable regression test
A useful investigation ends with a test that would fail if the same boundary broke again. Preserve a sanitized version of the source, a dummy user task, and the permitted final outcome. Assert that the unauthorized operation is blocked, not merely that the assistant uses a reassuring phrase.
Run that case after changing models, prompts, extraction libraries, retrieval settings, or tool permissions. These changes can affect different parts of the detection path. Keep a normal control case beside the attack so a security improvement does not quietly make the original feature unusable.
For the next step, I would choose one real workflow and trace a harmless instruction attempt from ingestion to execution. Mark which layer notices it and which layer stops it. That exercise produces an actionable detection map, whereas purchasing another scanner without understanding the data path can leave the same gaps intact.
Common questions
Can regex detect every injection?
No. It can flag known patterns, but task context and downstream behavior matter. Use rules as one signal rather than a universal decision.
Does a clean scan make a document trusted?
No. A detector assesses the submitted representation. It does not authenticate the author or grant the document authority over tools.
Should I log entire conversations?
Only when justified by a controlled evidence policy. Routine monitoring should minimize sensitive content and preserve useful references instead.