Inventorying sudo rights and authorized_keys across the whole fleet with one script

The question always comes at the worst moment: a colleague leaves, a laptop is stolen, an auditor asks "who has root on the database server?" — and the answer is a round of ssh across all hosts while everyone waits. The problem isn't that the information is missing; it's in /etc/passwd, /etc/sudoers and thirty authorized_keys files. The problem is that nobody has it in one place, with a date, in a form you can compare with yesterday.

Step 1: what's on one host

# Interactive accounts (uid ≥ 1000, real shell) + root
awk -F: '($3>=1000 || $1=="root") && $7!~/nologin|false/ {print $1, $3, $6, $7}' /etc/passwd

# Who may sudo? Via group...
getent group sudo admin wheel 2>/dev/null
# ...and via explicit rules (without comments, Defaults and @include directives)
sudo grep -rhvE '^\s*(#|@|Defaults|$)' /etc/sudoers /etc/sudoers.d/ 2>/dev/null

# Which keys are authorised, where, and whose?
for d in /root /home/*; do
  f="$d/.ssh/authorized_keys"; [ -f "$f" ] || continue
  echo "== $f"; sudo ssh-keygen -lf "$f" 2>/dev/null       # bits, fingerprint, comment, type
done

Two things ssh-keygen -lf shows you immediately: keys of 1024-bit RSA or of type ssh-dss (both banned in OpenSSH defaults for years, yet surprisingly common), and keys without a comment — whose owner nobody knows.

Don't forget the places outside authorized_keys:

# Alternative locations from sshd_config
sshd -T | grep -iE '^(authorizedkeysfile|authorizedkeyscommand|trustedusercakeys) '
# SSH certificates (CA) — then authorisation lives at the CA, not in a file
# System accounts with a shell AND a key (deploy users, backup users)
awk -F: '$3<1000 && $7!~/nologin|false/ {print $1}' /etc/passwd

Step 2: one script, one tsv per host

The script produces three kinds of lines — user, sudo, key — in a form you can sort, diff and paste into a spreadsheet.

sudo tee /usr/local/sbin/access-inventory.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
# Output: host<TAB>type<TAB>subject<TAB>detail<TAB>extra
H=$(hostname -s)

# Accounts with a shell (incl. system accounts with a shell — deploy users are access too)
awk -F: -v h="$H" '$7!~/nologin|false|sync|halt|shutdown/ {
  printf "%s\tuser\t%s\tuid=%s\tshell=%s\n", h, $1, $3, $7 }' /etc/passwd

# Last SSH login per account, from the journal (lastlog and wtmp are no longer reliable/present on Ubuntu ≥ 24.04)
journalctl -u ssh --since "365 days ago" --no-pager -o short-iso 2>/dev/null \
  | grep -E 'Accepted (publickey|password)' | awk '{print $7, $1}' | sort -k1,1 -k2,2r | awk '!seen[$1]++' \
  | awk -v h="$H" '{printf "%s\tlastlogin\t%s\t%s\t\n", h, $1, $2}'

# Sudo via group membership
for g in sudo admin wheel; do
  getent group "$g" 2>/dev/null | awk -F: -v h="$H" -v g="$g" '$4!="" {n=split($4,u,","); for(i=1;i<=n;i++) printf "%s\tsudo\t%s\tgroup=%s\t\n", h, u[i], g}'
done
# Sudo via explicit rules (without comments, Defaults and @include directives)
grep -rhvE '^\s*(#|@|Defaults|$)' /etc/sudoers /etc/sudoers.d/ 2>/dev/null \
  | awk -v h="$H" '{ subj=$1; $1=""; sub(/^ /,""); printf "%s\tsudo\t%s\trule=%s\t\n", h, subj, $0 }'

# Keys: one line per authorised key, with fingerprint, type and comment
for d in /root /home/*; do
  f="$d/.ssh/authorized_keys"; [ -f "$f" ] || continue
  u=$(basename "$d"); [ "$d" = /root ] && u=root
  ssh-keygen -lf "$f" 2>/dev/null | awk -v h="$H" -v u="$u" '{ fp=$2; type=$NF; gsub(/[()]/,"",type); $1=""; $2=""; $NF=""; sub(/^ +/,""); sub(/ +$/,""); printf "%s\tkey\t%s\t%s\t%s %s\n", h, u, fp, type, $0 }'
done
EOF
sudo chmod 0755 /usr/local/sbin/access-inventory.sh
sudo /usr/local/sbin/access-inventory.sh | column -t -s $'\t' | head -30
# web-01  user       jeroen   uid=1001                         shell=/bin/bash
# web-01  sudo       jeroen   group=sudo
# web-01  sudo       deploy   rule=ALL=(root) NOPASSWD: /usr/bin/systemctl restart webapp
# web-01  key        jeroen   SHA256:7Kq…                      ED25519 jeroen@laptop-2024
# web-01  key        deploy   SHA256:aa1…                      RSA ci-runner
# web-01  key        deploy   SHA256:9f3…                      RSA (no comment)   ← whose?

Step 3: collect fleet-wide and pivot per person

From a management host, over a hosts.txt:

D=/srv/inventory/access; mkdir -p "$D"
while read -r h; do
  ssh -o ConnectTimeout=5 -o BatchMode=yes "$h" sudo /usr/local/sbin/access-inventory.sh 2>/dev/null \
    || echo "$h	error	-	unreachable	" 
done < hosts.txt > "$D/$(date +%F).tsv"

# Turn the question around: per KEY, on which hosts and under which account does it sit?
awk -F'\t' '$2=="key" {print $4, $5, "→", $1":"$3}' "$D/$(date +%F).tsv" | sort | uniq
# SHA256:7Kq… ED25519 jeroen@laptop-2024 → db-01:jeroen
# SHA256:7Kq… ED25519 jeroen@laptop-2024 → web-01:jeroen
# SHA256:9f3… RSA (no comment)           → web-01:deploy   ← one key, no owner: clean up or label
# SHA256:aa1… RSA ci-runner              → web-01:deploy, web-02:deploy, db-01:root   ← CI key on root of the database?

# Per person: on which hosts do they have sudo?
awk -F'\t' '$2=="sudo" {print $3, "→", $1, "("$4")"}' "$D/$(date +%F).tsv" | sort

That reversed list — per key, where it sits — is where the surprises live: the CI key on the database's root account, the key of an ex-colleague under a shared deploy account, and the key without a comment that nobody dares to remove.

Step 4: daily diff — report what was added

A new key or sudo rule is always either a deliberate change (then there's a ticket for it) or an incident. In both cases you want to see it the same day.

sudo tee /usr/local/sbin/access-diff.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
NTFY="https://ntfy.example.be/security"
D=/srv/inventory/access
today=$D/$(date +%F).tsv
prev=$(ls -1 "$D"/*.tsv 2>/dev/null | grep -v "$today" | tail -1)
[ -f "$prev" ] || exit 0
# Compare only keys, sudo rules and users (lastlogin changes daily)
added=$(comm -13 <(grep -E $'\t(key|sudo|user)\t' "$prev" | sort) <(grep -E $'\t(key|sudo|user)\t' "$today" | sort))
removed=$(comm -23 <(grep -E $'\t(key|sudo|user)\t' "$prev" | sort) <(grep -E $'\t(key|sudo|user)\t' "$today" | sort))
if [ -n "$added$removed" ]; then
  { [ -n "$added" ] && { echo "ADDED:"; echo "$added"; }; [ -n "$removed" ] && { echo "REMOVED:"; echo "$removed"; }; } \
  | curl -s -H "Title: access change (fleet)" -H "Priority: high" --data-binary @- "$NTFY" >/dev/null
fi
EOF
sudo chmod 0755 /usr/local/sbin/access-diff.sh
echo '15 6 * * * root /usr/local/sbin/access-collect.sh && /usr/local/sbin/access-diff.sh' | sudo tee /etc/cron.d/access-inventory

(Put the while read loop from step 3 in /usr/local/sbin/access-collect.sh.) The result: every morning a message if something changed, otherwise silence. An ADDED … key … root … line without a ticket is an incident, not a to-do.

Step 5: cleaning up without locking anyone out

The list is one thing; removing keys is the exciting part. Work in this order:

1. Add a comment to every key whose owner you know (ssh-keygen can't; edit the file: the third field is free text).
2. Keys without an owner: log for 30 days first: LogLevel VERBOSE in sshd_config logs the fingerprint on every login (Accepted publickey for deploy … SHA256:9f3…). Not used: remove.
3. Shared accounts (deploy with eight keys): convert to personal accounts + sudo rule, or to SSH certificates with a short TTL. Then every login traces to a person.
4. 1024-bit RSA and DSA: replace with ed25519 — OpenSSH 9+ no longer accepts them by default, so they're already broken on the newest hosts.

Pitfalls

• AuthorizedKeysCommand. If sshd fetches keys from an IdP or a central source, authorized_keys is empty and so is your inventory. sshd -T tells you (step 1); then you inventory at the source.
• Keys in ~/.ssh/authorized_keys2. Deprecated but still read on old hosts if it's in AuthorizedKeysFile. Same check.
• Sudo via %group from LDAP/SSSD. getent group sudo only shows those members if NSS resolves them; on hosts with SSSD that can be slow or empty during an outage. Run it in the maintenance window without cache.
• NOPASSWD rules on commands with arguments. NOPASSWD: /usr/bin/systemctl (without arguments) means any systemctl command, including systemctl --now disable auditd. Restrict to the exact command with arguments.
• Forgetting sudo -l per user. The sudoers rules say what's allowed; sudo -l -U jeroen says what's effectively allowed after all groups and aliases. For the review that's the better source.

What you still don't have

• The link to staff. The tsv says "jeroen has sudo on 12 hosts", not whether Jeroen still works here. That's the quarterly review, with the HR list next to it.
• History per key. When did this key appear, who added it (which sudo session), and when was it last used? Three questions, three sources.
• Evidence. An auditor wants the inventory signed and immutable, and proof that the diff alert works (a test log).

How monsys does it

The monsys agent inventories per host the accounts, sudo rights (effective, after groups and aliases), authorized_keys per user with fingerprint and comment, and the last login — as part of the regular inventory. The hub shows it fleet-wide per person and per key, reports a new key or sudo rule as drift detection, and generates a signed access review report (ISO 27001 A.5.18) every quarter with all accounts, their rights and their last activity — including the SSO configuration without secrets.

FAQ

How do I find out whose SSH key it is when there's no comment?

Set LogLevel VERBOSE in sshd_config; from then on every login logs the fingerprint. After a few weeks you know which keys are used and from which IP. Keys never used you can remove; keys that are used you check with the user of that IP.

What's the difference between the sudo group and a sudoers rule?

Membership of sudo (Ubuntu) or wheel (RHEL) grants ALL=(ALL:ALL) ALL — everything. An explicit rule in /etc/sudoers.d/ can be restricted to one command. Both count for the review; sudo -l -U <user> shows the combined result.

How often should I run this?

The inventory daily (automated, with the diff alert), the human review every quarter. NIS2 and ISO 27001 don't ask for a frequency in days, but for demonstrable periodic review of access — and a quarter is what auditors expect.