Logging LLM calls in a way that's useful for debugging AND compliant with GDPR, AI Act and NIS2. Client-side PII redaction, Ed25519-signed evidence packs, and what this isn't.

When your backend calls an LLM — OpenAI, Claude, Mistral, Azure OpenAI — you have a logging problem that's different from traditional API calls. A REST call to a payment provider logs you a request ID and a status code. An LLM call potentially logs the full text an end user typed in, including everything inside it.

GDPR calls this "personal data". AI Act Article 12 calls it "events the AI system generates". NIS2 calls it "audit trail". Three regulations, one problem: how do you log LLM calls in a way that's useful for debugging AND compliant with all three?

This is what the monsys AI observability module does — and what it deliberately doesn't do.

The three hard invariants

Three principles that are not negotiable, regardless of customer or configuration:

1. Passive, never autonomous monsys executes no prompts, never blocks inline, never makes decisions on its own based on what's logged. It's an observability layer — evidence after the fact, not a control plane. If you need inline prompt-injection filtering or guardrails, combine us with Lakera or Protect AI.

2. PII redacted at the source Belgian IBAN, national registry number, BTW-BE, KBO, NL BSN, FR NIR, e-mails and phone numbers are recognised with checksum validation before storage. Raw content never reaches the database.

3. Evidence packs with Ed25519 Every logging session is offline-verifiable through a signed tarball. A DPA inspector without a monsys account can prove the data hasn't been altered.

The wire envelope spec: what goes from your code to the hub

When your SDK closes a span, a single HTTPS POST is sent:

{
  "schema_version": "1",
  "app_token": "aiv_...",
  "trace_id": "0e22f4a1-...",
  "span_id": "b3c1d2e4-...",
  "span_name": "openai.chat",
  "provider": "openai",
  "model": "gpt-4o",
  "prompt_hash": "sha256:a3f2c1...",
  "completion_hash": "sha256:b7e4d2...",
  "prompt_text": "What is the balance of IBAN [REDACTED-IBAN-BE] as of today?",
  "completion_text": "The balance of account [REDACTED-IBAN-BE] is €1,847.32.",
  "input_tokens": 23,
  "output_tokens": 18,
  "cost_eur": 0.000041,
  "pii_hits": [
    {
      "type": "IBAN_BE",
      "offset_start": 28,
      "offset_end": 46,
      "token": "sha256:c1d2e3f4..."
    }
  ],
  "pii_hits_count": 1,
  "started_at": "2026-05-25T09:14:02.341Z",
  "ended_at": "2026-05-25T09:14:03.887Z"
}

Note: prompt_text and completion_text contain [REDACTED-IBAN-BE] — not the original IBAN. The redaction happens in the SDK, client-side, before the POST is sent. The hub never sees the raw value.

prompt_hash and completion_hash are the SHA256 hashes of the original text (before redaction). You can use those hashes later to prove two spans had the same prompt, without keeping the prompt content itself.

PII detection: how the checksum validation works

For IBAN detection, a regex is not enough — BE68539007547034 looks like an IBAN, but BE00123456789012 is not (checksum fails). monsys uses the ISO 13616 mod-97 checksum:

def validate_iban(iban: str) -> bool:
    # Move the first 4 characters to the back
    rearranged = iban[4:] + iban[:4]
    # Replace letters with digits (A=10, B=11, ...)
    numeric = ''.join(
        str(ord(c) - ord('A') + 10) if c.isalpha() else c
        for c in rearranged
    )
    # Mod-97 check
    return int(numeric) % 97 == 1

The same principle applies to national registry number (mod-97), BTW-BE (mod-97), KBO (mod-97), NL BSN (mod-11), FR NIR (mod-97).

This eliminates false positives: a number that happens to look like an IBAN but fails the checksum is not redacted.

What's not yet covered:

DE Steuer-ID
IT Codice Fiscale
ES NIF
US SSN
Credit card numbers (Luhn check available but not yet implemented)

These are on the Q3-Q4 2026 roadmap. We state this explicitly because "PII redaction" is a dangerously vague promise if you don't say which PII types you actually cover.

The SDK: ~150 lines per language, no dependencies

The Python SDK (and Node/Go equivalents) have one external dependency: cryptography for the Ed25519 signing of the span payload. That's it.

# monsys_ai/tracer.py — simplified
import hashlib, json, time
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import Optional
import httpx

@dataclass
class Span:
    name: str
    provider: str
    model: str
    prompt: Optional[str] = None
    completion: Optional[str] = None
    input_tokens: Optional[int] = None
    output_tokens: Optional[int] = None
    _started_at: float = field(default_factory=time.time)

    def _redact_pii(self, text: str) -> tuple[str, list]:
        """Redact PII client-side before transmission."""
        hits = []
        result = text
        import re
        iban_pattern = re.compile(r'\bBE\d{2}[\s]?\d{4}[\s]?\d{4}[\s]?\d{4}\b')
        for match in iban_pattern.finditer(result):
            candidate = match.group().replace(' ', '')
            if self._validate_iban(candidate):
                hits.append({"type": "IBAN_BE", "token": hashlib.sha256(candidate.encode()).hexdigest()})
                result = result[:match.start()] + "[REDACTED-IBAN-BE]" + result[match.end():]
        return result, hits

The SDK never fails: if the HTTP POST to the hub fails (timeout, network error), it's logged but the calling code gets no exception. Your application's LLM functionality is never blocked by the observability layer.

Alerts: when the system wakes you

Four alert types for AI observability:

Cost spike: cost_per_minute > threshold_eur. Default: €1/minute. Configurable per app. Fires via ntfy/webhook, not e-mail (too slow for a prompt bug that burns €500/hour).

Refusal rate: refusal_count / total_spans > threshold over a 15-minute sliding window. The hub detects refusals by checking whether the completion starts with patterns like "I cannot", "I'm sorry, I can't", or equivalents in NL/FR. Not perfect, but good enough to flag model drift.

PII rate: pii_hits_count / total_spans > threshold. If suddenly 40% of your spans have PII hits when it's normally 2%, something has changed in how end users use the application.

Z-score anomaly: same z-score method as the server monitoring (|z| > 2.5). Computed over cost_per_span, input_tokens_per_span, duration_ms per app per model. Catches prompt regressions that drive costs up without crossing an absolute threshold.

Evidence packs for AI Act Article 12

Article 12 of the AI Act requires high-risk AI systems to automatically keep logs of their operation, including input data to the extent it enables identification of the cause of problems.

A monsys evidence pack for AI observability contains per period:

evidence_<tenant>_<period>.tar.gz
├── manifest.json              ← period, tenant, inputs_hash
├── manifest.sig               ← Ed25519 signature
├── signing_key.pub
├── verify.py
└── data/
    └── ai_traces/
        ├── spans.ndjson       ← all spans, PII-redacted
        ├── pii_summary.json   ← aggregate: X hits of type Y
        ├── cost_summary.json  ← cost per app, per model, per day
        └── alerts.ndjson      ← cost spikes, refusals, anomalies

The spans.ndjson contains the redacted texts and the SHA256 hashes of the originals. A regulator can prove that a specific span processed a customer's IBAN (the hash matches) without seeing the IBAN itself.

For the scenario where you do need the original content (e.g. complaint handling): the TOTP-gated "unlock content" function grants one-time access to the unredacted span, logged in the audit trail.

What this is not

We state this explicitly because the market is full of products that promise "AI compliance" without saying what they actually deliver:

No NIS2 certification — that doesn't exist for vendors. We help with Article 21 evidence.
Not a DPIA replacement — we are one input to your impact assessment.
No guardrail or prompt-injection filter — passive audit layer, never blocking inline.
No jailbreak detection — we log refusals when the model refuses; misuse pattern detection is out of scope.
No guarantee about what OpenAI/Anthropic does with your data — that's covered by your contract with them.

The value is evidence, not prevention. For prevention, you need different tools.

AI observability is documented at docs.monsys.ai/en/ai/quick-start. First AI app free per tenant: monsys.ai/en/ai.

AI observability: PII redaction at the source, the three hard invariants and what we deliberately don't do