LLM Guardrail Gateway
An API gateway that intercepts and redacts PII/PHI from LLM prompts and enforces output schemas.
The LLM Guardrail Gateway is a critical security infrastructure component designed for healthcare environments. It acts as a reverse proxy between internal applications and external LLM APIs (like OpenAI), automatically detecting and redacting Personally Identifiable Information (PII) and Protected Health Information (PHI) before it leaves the internal network.
Hospitals (like AIHK) want to use advanced LLMs to summarize patient histories, but sending raw medical records to external API providers is a direct violation of HIPAA and GDPR. Relying on users to manually anonymize data is too risky.
Hospitals (like AIHK) want to use advanced LLMs to summarize patient histories, but sending raw medical records to external API providers is a direct violation of HIPAA and GDPR. Relying on users to manually anonymize data is too risky.
- Automatically identify and redact 18+ types of PII/PHI in milliseconds.
- Detect and block prompt injection attacks.
- Ensure all LLM outputs conform to a strict, safe JSON schema before returning to the client application.
System Flow
- Inbound Sanitization: Prompts are validated against
config.yamlguardrails to strip PII and block instruction-override phrases. - Upstream Governance: The proxy connects asynchronously via
httpx.AsyncClientto the LLM (e.g., Groq API), injecting a strict JSON-mode system prompt. - Outbound Validation: The raw string output is parsed, validated against required schema keys (e.g.,
status,message,data), and auto-repaired if necessary. - Audit Logging: Every lifecycle event is recorded in an append-only
audit_trail.jsonl.
System Flow
- Inbound Sanitization: Prompts are validated against
config.yamlguardrails to strip PII and block instruction-override phrases. - Upstream Governance: The proxy connects asynchronously via
httpx.AsyncClientto the LLM (e.g., Groq API), injecting a strict JSON-mode system prompt. - Outbound Validation: The raw string output is parsed, validated against required schema keys (e.g.,
status,message,data), and auto-repaired if necessary. - Audit Logging: Every lifecycle event is recorded in an append-only
audit_trail.jsonl.
| Layer | Technology |
|---|---|
| Framework | FastAPI (Python) |
| HTTP Client | HTTPX (Async) |
| NLP/Security | Regex Heuristics, MS Presidio, spaCy |
| LLM Provider | Groq API |
| Testing | Custom ASCII Security Dashboard via test_gateway.py |
- Challenge: NLP models (like spaCy) can add 300-500ms of latency to every request.
- Solution: Optimized the spaCy pipeline by stripping out unnecessary components (like the dependency parser) and only running the Named Entity Recognition (NER) module.
- Challenge: Handling false positives in medical terminology.
- Solution: Implemented custom Presidio recognizers using Regex specifically tailored for Pakistani medical ID formats and local terminology.
- Challenge: NLP models (like spaCy) can add 300-500ms of latency to every request.
- Solution: Optimized the spaCy pipeline by stripping out unnecessary components (like the dependency parser) and only running the Named Entity Recognition (NER) module.
- Challenge: Handling false positives in medical terminology.
- Solution: Implemented custom Presidio recognizers using Regex specifically tailored for Pakistani medical ID formats and local terminology.
- "How does Microsoft Presidio actually identify PII, and how did you optimize its latency in your gateway?"
- "Explain the process of Re-identification. How do you return a coherent summary to the user if the names were stripped before sending to the LLM?"