I BUILD AI THAT IS: SECURE, TESTED, AUDITABLE.I BUILD AI
THAT IS
SECURE_

Problem

Support teams want to automate ticket handling, but automating access requests, critical incidents or tickets with personal data without clear limits creates risk. Inspired by handling ~1,000 incidents a month.

What I built

A local Python prototype. The rules of the first version (standard library only) stay as the guardrail, and an LLM classifier (Claude through the Anthropic API, JSON output validated against closed lists, invalid output goes to a person) is combined with them: the stricter decision wins and any disagreement goes to a person. A reply is drafted only when both allow automation, is checked (sensitive data, claims of actions performed, length) and is sent, in simulation, only after explicit approval. An evaluation harness compares 2 prompt versions and 2 models on 30 labelled synthetic tickets. The audit log records both decisions with sensitive values redacted.

Evidence

• Evaluation on 30 labelled synthetic tickets, 2 Claude models × 2 prompt versions. With the written policy in the prompt (v2), both models got the category right on 28/30 tickets, against 9/30 for the rules alone.
• Unsafe automation (a ticket that needs a person but would get an automated answer, 21 of the 30): 8/21 with the rules alone; with the guardrail, 1/21 on prompt v1 and 0/21 on v2. The LLM alone would have automated 3/21 (v1) and 1/21 (v2).
• The cost of caution: with v2, 4 of the 9 tickets that could be automated still went to a person. Cost per ticket with v2: US$ 0.00016 (Haiku 5.5) against US$ 0.0032 (Sonnet 5.5), at prices checked on 10/11/2026.
• 84 automated tests (standard library; none calls the API) run in CI on each push (Python 3.10 and 3.13). The results table is generated from a committed results file, and a test checks that the README matches it.

Guardrails

• The LLM can only make a decision stricter: whatever the rules block it cannot release, and any disagreement between rules and LLM goes to a person.
• Invalid LLM output (bad JSON, unknown value, refusal, API error) is treated as a failure and sent to a person.
• A reply draft is checked for sensitive data, claims of actions performed and length, and nothing is sent without explicit human approval.
• Sensitive data is redacted before it reaches the audit log.

Known limitations

• 30 synthetic tickets, labels drafted with AI assistance and approved by one reviewer (me): one ticket is 3.3 percentage points. One run per combination, variance not measured.
• Prompt v2 was written from the same policy that produced the labels. The result shows that the model follows a stated policy, not how it would perform on real tickets.
• Only classification was evaluated; reply drafts and latency were not.
• No full prompt-injection defence (5 injection tickets are a sample, not a security assessment), and no masking before the API call: the ticket text is sent as written, only the requester's name is left out.
• Human approval is a console prompt and sending is simulated; no real ticketing-system integration.

Problem

Employees adopt AI tools faster than organizations can approve them: personal data goes into unapproved tools, automated decisions skip human review, and nobody knows which AI systems are in use (Shadow AI).

What I built

A bilingual (EN/PT) policy with 14 sections and an auditable compliance checklist, a section-by-section framework mapping (ISO/IEC 42001, NIST AI RMF, LGPD), and an AI vendor risk assessment procedure with an Excel matrix that scores up to 3 vendors side by side.

Evidence

• Framework mapping: of 23 requirements, 13 covered explicitly, 7 partially and 3 delegated to the adopting organization. Gaps documented in a v3.0 roadmap.
• Vendor risk procedure: inherent-risk tiering, 30 questions in 8 weighted domains, 5 knock-out criteria and required contract clauses.

Known limitations

• A template, not an active policy of any organization. Adoption requires legal and governance review by the adopting company.