by PolicyForge (LX AI)
Last updated: 2026-09-16
AI Agent Security Checklist 2026
Agents that can call tools and read secrets are no longer demos - they are production systems. This checklist is the pre-deployment gate we use at LX AI before any agent touches real data. It maps to the OWASP Top 10 for LLM Applications (2025) and the EU AI Act (Regulation (EU) 2024/1689).
1. Inventory every prompt and tool
What to capture
- Exact system prompt string
- Each tool definition: name, endpoint URL, authentication method, input schema, output schema
- Data stores the tool reads from or writes to (databases, file buckets, APIs)
- External content the agent may ingest (email bodies, web pages, document uploads)
- Version identifier for the prompt and each tool configuration
How to store the inventory
- Keep a YAML file named
agent-inventory.yaml in the repo root
- List each item under top‑level keys:
prompt, tools, data_sources, external_inputs
- Tag each entry with a stable ID (UUID) that never changes across releases
- Add a comment line with the date and the engineer who last reviewed it
- Run a CI step that fails if the file is missing or if any ID is duplicated
Why this matters
Without a complete list you cannot apply any of the later controls. Missing a tool means you cannot enforce its allow‑list. Missing a data source means you cannot assess its sensitivity. Treat the inventory as the single source of truth for every subsequent security check.
2. Guard against prompt injection
Defensive layers
- Treat all external input as untrusted – email bodies, web scrapes, tool outputs
- Strip or escape any text that looks like an instruction (e.g., strings starting with “Ignore”, “Override”, “System”)
- Require tools to return strictly structured data (JSON with a known schema) and reject anything that does not conform
- Maintain a deterministic allow‑list of actions that may affect secrets or money; any action not on the list is blocked
- Log the raw input and the sanitized version side‑by‑side for later review
Practical checklist
- Run a unit test that feeds the agent a known injection payload and asserts the output does not contain the payload
- Use the OWASP cheat sheet for prompt injection prevention as a reference: LLM Prompt Injection Prevention Cheat Sheet
- In CI, run a static scan that searches for raw string concatenation of user content into the prompt; fail the build if any are found
- Deploy a lightweight proxy that sits between the agent and the LLM API; the proxy applies the stripping rules before forwarding the request
- Monitor logs for any occurrence of the strings “Ignore previous” or “Disregard”; trigger an alert if they appear
Table: Example allow‑list for a finance‑assistant agent
| Action |
Allowed? |
Reason |
| Read account balance |
Yes |
Read‑only, low risk |
| Transfer funds |
No |
Requires human approval |
| Export transaction CSV |
Yes |
File stored in encrypted bucket |
| Update user profile |
No |
Modifies personal data |
Link to OWASP guidance
See also the OWASP LLM01 page for the underlying threat model: OWASP LLM01 – Prompt Injection
3. Stop secret leakage
Where secrets must live
- Environment variables injected at container start
- Secret manager service (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager)
- Never in the source tree, never in logs, never in the prompt text
Pre‑deploy scanning steps
- Run
git grep -i \"key\\|secret\\|token\\|password\" on the repo; fail if any match appears outside of .env.example or secret‑manager reference files
- Check CI configuration files (e.g.,
.github/workflows/*.yml) for hard‑coded values; treat any find as a blocker
- Use a secret‑detector tool (e.g., Gitleaks) as a CI job; set the severity to high
- After a successful build, pull the latest secret versions from the manager and compare their checksums to the values used in the previous release; alert on mismatch
Post‑deploy hygiene
- Rotate any secret that ever appears in a log or error message; automate rotation via a webhook from the logging system
- Set a short TTL (e.g., 24 hours) for secrets used by the agent; the orchestrator renews them automatically
- Audit logs for any outbound network call that includes a secret‑like string in the URL or headers; flag for review
- On secret rotation, update the agent’s configuration and redeploy within the same maintenance window
Credential‑hygiene reference
For additional guidance on keeping CI pipelines clean, see the OWASP CICD‑SEC‑06 page: Insufficient Credential Hygiene
4. Require human oversight for high‑risk actions
Defining high‑risk
- Any operation that can delete data, send money, or change access rights
- Actions that affect personal data covered by GDPR or similar regulations
- Calls to external services that have financial consequences (e.g., payment gateways, trading APIs)
Implementation pattern
- Wrap the risky call in a function that returns a pending‑approval token
- Store the token in a durable table with fields:
request_id, user_id, action_type, payload, requested_at, status
- Expose a simple UI or slack‑bot where an authorized operator can approve or reject
- On approval, the function executes the real call; on rejection, it logs the reason and returns a user‑friendly error
- Set a timeout (e.g., 15 minutes); if no decision arrives, automatically reject and notify the requester
EU AI Act link
Article 14 of the EU AI Act requires human oversight for high‑risk AI systems; the pattern above satisfies that requirement by making the oversight explicit, traceable, and enforceable.
5. Log every run with a runId
What to log
- A UUID generated at the start of the agent run (
runId)
- The version of the ruleset or policy file used (
policyVersion)
- The exact input payload (prompt + any external content after sanitization)
- All tool calls made, with their inputs and outputs
- The final output produced by the agent
- Any errors or exceptions encountered
- Timestamp in UTC with millisecond precision
Storage and querying
- Write each log entry as a JSON line to an append‑only store (e.g., Amazon S3 with Object Lock, or an immutable Kafka topic)
- Partition logs by date (
YYYY/MM/DD) to enable fast range scans
- Create a simple index table that maps
runId to the object key; keep this index in a regular database for quick look‑ups
- Retain logs for the period required by applicable regulation (e.g., 24 months for EU AI Act high‑risk systems)
- Provide an internal API that accepts a
runId and returns the full log; audit teams use this to investigate incidents
Why a stable identifier matters
Without a stable runId you cannot correlate a bad output with the exact input and policy version that produced it. This breaks the ability to reproduce issues, to prove compliance, and to improve the system over time.
6. Set a fair‑use cost guard
Limits to enforce
- Maximum tokens per individual request (e.g., 4 000)
- Maximum tokens per day per agent instance (e.g., 200 000)
- Maximum number of tool calls per run (e.g., 20)
- Maximum outbound bandwidth per hour (e.g., 10 MB)
Enforcement mechanism
- Count tokens locally before sending the request to the LLM API; if the count exceeds the per‑request limit, return HTTP 429 immediately
- Maintain a rolling counter in Redis (or similar) keyed by agent ID and date; increment after each successful call; reject new calls when the daily quota is reached
- For tool calls, increment a separate counter; if the limit is hit, return a deterministic error that the caller can handle
- When a limit is hit, emit a structured log event with fields
limit_type, current_value, limit_value, and timestamp
- Expose a Prometheus metric (
agent_cost_guard_rejections_total) so ops can see trends and adjust limits
Link to OWASP LLM10
Unbounded consumption is captured in OWASP LLM10; the guard turns a potential runaway bill into a predictable, observable failure mode.
7. What this is NOT
This checklist is not a guarantee that an agent will never be compromised. It does not promise 100 % protection against every possible attack. It does not replace the need for threat modeling, penetration testing, or continuous monitoring. Treat it as a baseline gate; additional controls may be required depending on your specific risk profile.
References
Frequently asked questions
What is policy-as-code?
Compliance rules expressed as code, reviewed and tested like software.
Which frameworks?
GDPR, EU AI Act, and custom internal controls out of the box.
How are policies tested?
Each rule ships with test cases; CI runs them on change.
Is it a DLP?
No - it governs policy logic, not data loss prevention.
Can non-engineers edit?
Pro includes a template editor; changes still go through review.
How is versioning handled?
Git-backed; each ruleset has a version for audit.
References
- OWASP Top 10 for LLM Applications (2025) - owasp.org/www-project-top-10-for-large-language-model-applications/
- EU AI Act, Regulation (EU) 2024/1689 - eur-lex.europa.eu
- NIST AI Risk Management Framework (AI RMF 1.0) - nist.gov/itl/ai-risk-management-framework
Want a fast, self-serve read on your own agent? Try PolicyForge