Where personal data actually leaks in an agent system
Three places, and an agent system touches all three. The prompt: the agent assembles context and ships it to the model, often including raw customer records pulled by a tool call. The logs: every prompt and response is usually logged verbatim for debugging, so the PII in the prompt is now also in your log store, where it gets indexed by your observability vendor and retained for the default 30 to 90 days. The model API: unless you have a no-retention or zero-data-retention agreement with the provider (Anthropic, OpenAI, and Google all offer these on Enterprise tier, but not by default), a third party now holds your customer data and may use it for abuse monitoring for up to 30 days. Vibe-coded systems are worse here because the AI assistant builds the "log everything" version by default. It is the simplest code, so it is the code the model writes. The first audit we run at a client is to print the last prompt sent to the model and read it line by line. Most teams have never done this exercise and are surprised by what is in there.
Fix 1: Data minimisation at the tool boundary
The single highest-leverage fix. When the agent calls a tool that returns a customer record, the tool should return only the fields the agent needs for this task, not the whole row. An agent answering "what is my order status" needs the order ID and the status, not the customer date of birth, payment token, and address history. Enforce this in the tool layer, where the data is fetched, not in the prompt template, where it is too late. GDPR Article 5(1)(c) calls this the data minimisation principle, and Article 5(1)(b) extends it with purpose limitation: data collected for one purpose should not flow to another. The India DPDP Act 2023 contains the equivalent in Section 8. It is also just good engineering: less data in the prompt means lower token cost, less context dilution, and better model focus. A practical worked example: a support tool that previously returned the whole 47-field user record now returns five fields. The agent works as well, the prompt is 80 percent smaller, the audit finding is gone, and the inference cost dropped by two thirds.
Fix 2: Redact PII before the prompt is built
Even with minimisation, some PII has to reach the model (a name, an email the agent needs to act on). For everything else, redact before the prompt is assembled. A redaction pass at the tool boundary swaps emails, phone numbers, card numbers, national IDs, and free-text addresses for typed placeholders (such as <EMAIL_1>, <PHONE_2>) the model can still reason about. Microsoft Presidio is the common open-source choice; it ships with detectors for the common PII classes and is straightforward to extend with custom recognisers for region-specific identifiers (Aadhaar, NHS number, SIN). For higher accuracy, the AWS Comprehend PII detection API and the Google DLP API both do the same job as managed services. The rule: redaction happens before prompt construction, not after logging. Redacting after the data is already in the log defeats the purpose, because the unredacted version is what your observability vendor indexed.
Fix 3: Retention windows and automated deletion
Conversation logs are useful for debugging and useless as a permanent liability. Set a retention window (30 days for low-stakes consumer agents, 60 to 90 days for regulated sectors that need a longer investigation window) and an automated deletion job that enforces it. "We keep everything" is the most common privacy finding we write up, and the fix is a scheduled job: a daily Lambda or a Kubernetes CronJob that prunes anything past the window. Pair it with a documented data subject access and deletion path, because GDPR Article 15 gives individuals the right of access and Article 17 the right of erasure, and you cannot honour either if you do not know where the data lives. The DPDP Act establishes parallel rights for Indian users. A useful drill: ask your engineering team to demonstrate the deletion path for a single named user in under an hour. If they cannot, the audit finding writes itself.
Fix 4: Regional data residency
If your customers are in the EU, their data and the model inference on it should stay in the EU region. If they are in India, the DPDP Act expects the same and Section 16 gives the government power to restrict cross-border transfers further. If they are in the UK, post-Brexit UK GDPR still requires an adequacy decision or appropriate safeguards. Most model providers now offer regional endpoints: Anthropic has EU and US regions, OpenAI offers EU data residency for Enterprise customers, Azure OpenAI lets you pin to specific regions including Frankfurt and Dublin. The gap we find is that the vibe-coded system used whatever default endpoint the SDK pointed at, usually us-east-1. Pin the endpoint to the customer regulatory region in the SDK initialisation, audit it in CI, and surface the region in your observability dashboards so a regression is visible. For high-sensitivity deployments (healthcare, public sector), run the model on infrastructure you provision in-region: Bedrock in Frankfurt, Vertex AI in Mumbai, or self-host with vLLM on EU compute.
Fix 5: Consent-aware activation
An agent that messages, profiles, or makes decisions about a person should check consent at the point of action, not assume it. For native channels (a marketing email through Braze, a push through OneSignal) this is usually a flag the agent reads before it acts and the platform enforces. For custom actions (a webhook to your own systems, a model-driven outbound call) the agent has to check consent itself, because nothing else will. The pattern: a consent service the agent queries before any outbound action, with the consent record versioned and timestamped so you can prove what was true at the moment the action fired. Purpose limitation applies too: data collected for support should not be silently reused to train a sales agent, even if the agent is technically authorised to read the underlying record. Design the consent check in as a tool the agent must call before any send; do not bolt it on after the regulator asks. The ICO has published explicit guidance on this for AI systems since 2023, and it is the clearest plain-language source available.
The privacy-by-design checklist
Run these seven before an agent ships. Data minimisation at the tool layer. PII redaction before prompt construction. A retention window with automated deletion. Regional data residency pinned in the SDK and audited in CI. Consent checked at the point of action through a versioned consent service. A documented data subject access and deletion path that an engineer can execute in under an hour. A no-retention or zero-data-retention agreement with the model provider, signed, on file, and named in the system documentation. Each is small. Together they are the difference between an agent that is privacy-compliant by design and one that is a finding waiting to be written up. The NIST Privacy Framework v1.0 is the cleanest reference for mapping the checklist onto a recognised standard if your buyer needs one.
Further reading
Real, named sources the editor can swap in for specific URLs. We do not auto-link these because the right link changes over time. If you find a great primary source, write us and we will update the note.
- GDPR, Articles 5 and 25 (data minimization, privacy by design). The two articles that govern how an agent should handle personal data. Short, readable, foundational.
- India Digital Personal Data Protection Act (DPDP). India's data protection law. Residency and consent expectations that apply to any agent serving Indian users.
- NIST Privacy Framework. The companion to the NIST Cybersecurity Framework. A practical structure for privacy risk in AI systems.
- ICO guidance on AI and data protection. The UK regulator's working guidance. The clearest plain-language read on AI privacy obligations.
- Microsoft Presidio. Open-source PII detection and redaction. The common choice for the redaction step at the tool boundary.
- r/privacy and r/gdpr. Practitioner threads on AI logging, retention, and data-subject-request handling.
Comments