Scanning container images with Trivy: in CI and on the host, without 400 findings per image

trivy image python:3.12 yields around 150 CVEs on a good day. That's not Trivy's fault — most Debian-based images contain hundreds of packages for which Debian knows vulnerabilities it deliberately doesn't fix yet (no-dsa, low risk). Whoever puts that list unfiltered into a CI pipeline has a || true behind it within a week and therefore no scanner at all. This article turns Trivy into a gate that only fails when it should, and a nightly check of what actually runs on your hosts.

Step 1: install and scan one image

# Ubuntu/Debian: official repo
sudo apt install -y wget apt-transport-https gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt update && sudo apt install -y trivy
trivy --version

# The first scan fetches the vulnerability database (~60 MB); after that it's local
trivy image --severity HIGH,CRITICAL grafana/grafana:latest

The output is per target: the OS layer (alpine 3.24), then every detected application artefact separately — Go binaries, Node packages, Python wheels. That distinction matters: a CVE in a Go binary of a plugin you don't use weighs differently from a CVE in the OS libc.

trivy image --quiet --severity HIGH,CRITICAL --format json grafana/grafana:latest \
  | jq -r '.Results[] | "\(.Target)\t\(.Type)\t\(.Vulnerabilities|length)"' | column -t -s $'\t'
# grafana/grafana:latest (alpine 3.24.1)                          alpine    2
# usr/share/grafana/bin/grafana                                   gobinary  3
# usr/share/grafana/data/plugins-bundled/.../postgresql_datasource gobinary  14

Step 2: the three filters that remove the noise

# 1. --ignore-unfixed: only CVEs the distribution HAS a fix for
# 2. --severity: threshold; MEDIUM you review weekly, not per build
# 3. --ignorefile: deliberately accepted findings, with reason and expiry date
trivy image --ignore-unfixed --severity HIGH,CRITICAL --ignorefile .trivyignore \
  ghcr.io/example/shop:2026.09.1

--ignore-unfixed is the most important. A CVE without a fix in the distribution you can't resolve in the image; at most you can switch base image. That's a decision for a quarterly review, not for every build. Do let MEDIUM/LOW and the unfixed land in a weekly report (step 5), so they don't disappear.

.trivyignore — in the repo, next to the Dockerfile. Trivy only reads the CVE ids, but the comments next to them are for humans and for the auditor:

# .trivyignore — one line per accepted finding, with reason and expiry date.
# Reassess on every base-image change.
CVE-2026-1234   # libxml2 XInclude: not reachable, we parse no external XML. J. Peeters 2026-09-16, valid until 2026-12-31
CVE-2026-5678   # curl in build stage only, not in the runtime image. See Dockerfile line 12. Valid until base-image bump

Better than .trivyignore is a VEX statement: it carries the reason machine-readably and can be read by several scanners. trivy image --vex vex.json --show-suppressed … shows what was suppressed and why.

Step 3: the CI gate — only break on what's new

The classic gate --exit-code 1 --severity CRITICAL fails the build on every critical finding, including the one that was there yesterday with a ticket in progress. That leads to || true. Better: break only on findings that weren't in the previous successful scan.

# .gitlab-ci.yml (GitHub Actions is equivalent; Woodpecker too)
image-scan:
  stage: test
  image: aquasec/trivy:latest
  variables:
    IMG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  script:
    - trivy image --ignore-unfixed --severity HIGH,CRITICAL --ignorefile .trivyignore --format json -o scan.json "$IMG"
    - trivy image --ignore-unfixed --severity HIGH,CRITICAL --ignorefile .trivyignore "$IMG"   # readable in the log
    # Baseline = last scan of main, stored as artifact
    - 'curl -sf -o baseline.json "$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/artifacts/main/raw/scan.json?job=image-scan" || echo "{}" > baseline.json'
    - |
      NEW=$(jq -r '[.Results[]?.Vulnerabilities[]?.VulnerabilityID] | unique | .[]' scan.json \
            | grep -vxFf <(jq -r '[.Results[]?.Vulnerabilities[]?.VulnerabilityID] | unique | .[]' baseline.json) || true)
      if [ -n "$NEW" ]; then echo "NEW HIGH/CRITICAL findings since main:"; echo "$NEW"; exit 1; fi
  artifacts:
    paths: [scan.json]
    expire_in: 90 days

What this does: a merge request introducing a new critical vulnerability (new dependency, new base image) fails. Existing findings don't block, but appear in the log and the artefact. On main itself you add a separate weekly job that reports all HIGH/CRITICAL to a ticket or ntfy — that's the place for the backlog.

Step 4: what actually runs — scanning the host

CI scans what you build. On the host something else may run: an image pulled three months ago, a third party's latest, a container someone started by hand. So scan the running containers nightly, on the host itself:

sudo tee /usr/local/sbin/trivy-running.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
# Scans every unique image of a running container; reports HIGH/CRITICAL with a fix.
NTFY="https://ntfy.example.be/cve"; HOST=$(hostname -s)
D=/var/lib/cve-scan/images; mkdir -p "$D"; OUT="$D/$(date +%F).tsv"; : > "$OUT"
trivy image --download-db-only --quiet 2>/dev/null
for img in $(docker ps --format '{{.Image}}' | sort -u); do
  digest=$(docker image inspect --format '{{index .RepoDigests 0}}' "$img" 2>/dev/null | cut -d@ -f2 | cut -c8-19)
  trivy image --quiet --ignore-unfixed --severity HIGH,CRITICAL --format json "$img" 2>/dev/null \
  | jq -r --arg img "$img" --arg d "$digest" '
      .Results[]? | .Target as $t | .Vulnerabilities[]? 
      | [$img, $d, .Severity, .VulnerabilityID, .PkgName, .InstalledVersion, (.FixedVersion // "-"), $t] | @tsv' >> "$OUT"
done
n=$(wc -l < "$OUT"); c=$(awk -F'\t' '$3=="CRITICAL"' "$OUT" | wc -l)
# Only report what's new since yesterday
prev=$(ls -1 "$D"/*.tsv | grep -v "$(date +%F)" | tail -1)
if [ -n "$prev" ]; then
  new=$(comm -13 <(cut -f1,4,5 "$prev" | sort -u) <(cut -f1,4,5 "$OUT" | sort -u))
  [ -n "$new" ] && printf 'NEW findings in running images on %s (%s total, %s critical):\n%s\n' "$HOST" "$n" "$c" "$new" \
    | curl -s -H "Title: trivy@$HOST" -H "Priority: high" --data-binary @- "$NTFY" >/dev/null
fi
echo "$OUT: $n findings ($c critical)"
EOF
sudo chmod 0755 /usr/local/sbin/trivy-running.sh
echo '45 3 * * * root /usr/local/sbin/trivy-running.sh' | sudo tee /etc/cron.d/trivy-running

The tsv per day is your history: "since when has CVE-X been open in image Y" is a grep over the directory. The ntfy message only comes on new findings — usually meaning: Trivy's database learned a new CVE for a package you've been running for months. That's exactly the moment to look.

Step 5: the weekly overview, including what you filtered

What you filter out in step 2 mustn't disappear. Once a week the full state, sorted by what you can fix:

trivy image --quiet --format json ghcr.io/example/shop:2026.09.1 \
| jq -r '.Results[]?.Vulnerabilities[]? | [.Severity, (if .FixedVersion then "fixable" else "unfixed" end)] | @tsv' \
| sort | uniq -c
#  12 CRITICAL fixable      ← this week
#   3 CRITICAL unfixed      ← base-image decision
#  41 HIGH fixable
#  88 HIGH unfixed
# 140 MEDIUM unfixed        ← accept, document, quarterly review

And the answer to "fixable" is almost always the same: rebuild on a fresh base image. FROM python:3.12-slim without apt upgrade in the Dockerfile picks up the base image's fixes as soon as you rebuild — so rebuild weekly, even without code changes.

Pitfalls

• Scanning latest in CI. You scan what's under that tag today, not what runs in production. Scan the image digest you deploy, and pin base images by digest.
• Build-stage packages in the runtime image. gcc, curl, git in the final image give dozens of CVEs unrelated to your app. Multi-stage builds and --no-install-recommends halve the list.
• Trivy database offline. In a CI runner without internet the DB update fails silently and Trivy scans with an old database. trivy image --skip-db-update is deliberate; an outdated DB by accident isn't. Check trivy --version (shows the DB date) in the log.
• Ignoring secrets and misconfig. trivy image also scans for secrets by default (API keys in layers) and trivy config for Dockerfile mistakes (root user, no healthcheck). Those findings are often more important than the fiftieth libc CVE.
• .trivyignore without expiry. An ignore from 2024 on a CVE that's had an exploit since 2025 is a hole nobody sees anymore. Every line a date, every quarter a review.

What you still don't have

• Correlation with the host. Trivy says "CVE in image X". Which container runs that image, on which port, internet-facing or not, with which other containers next to it — that's what determines priority, and it's not in the scan report.
• Fleet view. Thirty hosts, thirty /var/lib/cve-scan/images directories. "Which hosts still run image digest abc…" is ssh work.
• Evidence. A tsv per day is history; an auditor wants the scan date, the DB version used and the decision per finding in one signed document.

How monsys does it

The monsys agent discovers running containers via the Docker socket and reports the image digest per container; the hub scans every unique digest with Trivy (once per digest, not per host) and links the findings to the container, the host and the network topology — a CRITICAL in a container behind port 443 on an internet-facing host ranks above the same CRITICAL in an internal batch job. Base-image drift ("this image was built on a base 4 months old") is a separate report. VEX statements and accepted risks you record on the finding and they appear in the audit pack; the scan DB version and date accompany every result.

FAQ

Why does Trivy find hundreds of CVEs in an official image?

Because the image contains hundreds of OS packages and the distribution knows vulnerabilities for many of them that it deliberately doesn't fix (yet) — low risk, no-dsa. With --ignore-unfixed what remains is what you can resolve by rebuilding on a fresh base image.

Should I fail the build on every CRITICAL?

No: that leads to || true within a week. Fail a merge request on new HIGH/CRITICAL findings relative to main, and handle the existing backlog via a weekly report and tickets.

How often should I rebuild images?

Weekly, even without code changes, so fixes from the base image come along. On a KEV listing or a critical CVE in your runtime (glibc, openssl, node): immediately.