Honeypot files on a Linux server: four canary files that give an intruder away

Perimeter detection tells you someone is at the door. This article is about the moment after: someone is inside, with a stolen key, a leaked password or through a vulnerable web app, and starts looking around. What does such a person do in the first five minutes? ls -la ~, cat ~/.bash_history, find / -name "*.pem", ls ~/.ssh, cat .env. That's exactly where you plant files no legitimate process ever touches. Whoever reads them is wrong by definition.

Step 1: the four files

A good canary looks valuable, sits where an attacker looks, and is used by nothing on the server itself.

File	Why an attacker opens it
/root/.ssh/id_rsa.monsys-backup	A private key with "backup" in the name: lateral movement in one step
/home/deploy/.aws/credentials	Cloud credentials are the top target after a break-in
/etc/backup/db-passwords.txt	The name alone
/var/www/.env.bak	Web app intruders look here first, before getting any further

Create them with believable content. An empty file or dummy gives the trap away.

sudo install -d -m 0700 /root/.ssh /etc/backup
sudo install -d -m 0700 -o deploy -g deploy /home/deploy/.aws

# 1. A real (but nowhere-authorised) private key
sudo ssh-keygen -q -t ed25519 -N '' -C 'backup@old-nas' -f /root/.ssh/id_rsa.monsys-backup
sudo rm /root/.ssh/id_rsa.monsys-backup.pub

# 2. AWS credentials in the right format (this key doesn't exist)
sudo tee /home/deploy/.aws/credentials >/dev/null <<'EOF'
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region = eu-west-1
EOF
sudo chown deploy:deploy /home/deploy/.aws/credentials; sudo chmod 0600 /home/deploy/.aws/credentials

# 3. A password file with a history
sudo tee /etc/backup/db-passwords.txt >/dev/null <<'EOF'
# migrated from wiki 2023-11 — TODO move to vault
prod-postgres  postgres  Pg!2023-rotate-me
prod-redis     -         r3d1s-Old-Pass
EOF
sudo chmod 0600 /etc/backup/db-passwords.txt

# 4. A .env backup
sudo tee /var/www/.env.bak >/dev/null <<'EOF'
APP_ENV=production
DB_PASSWORD=Sh0p-2024-legacy
STRIPE_SECRET=sk_live_51HxEXAMPLEEXAMPLEEXAMPLE
EOF
sudo chmod 0640 /var/www/.env.bak

Make the files look old: sudo touch -d '2024-03-18 09:12' <path>. A file from today next to files from two years ago stands out in ls -la.

Step 2: let auditd watch them

auditd is the kernel audit daemon; it sees every open() on the file regardless of which program does it, and logs the user, the process, the session and the source IP of the SSH session.

sudo apt install -y auditd
sudo tee /etc/audit/rules.d/50-canary.rules >/dev/null <<'EOF'
# -p r = read, w = write, a = attribute change. No 'x'.
# -k canary = key to search on.
-w /root/.ssh/id_rsa.monsys-backup -p rwa -k canary
-w /home/deploy/.aws/credentials  -p rwa -k canary
-w /etc/backup/db-passwords.txt   -p rwa -k canary
-w /var/www/.env.bak              -p rwa -k canary
EOF
sudo augenrules --load
sudo auditctl -l | grep canary        # rules active?

Test: sudo cat /etc/backup/db-passwords.txt and then:

sudo ausearch -k canary -ts recent -i

You'll see a PATH record with the file, a SYSCALL record with uid, auid (the original user, even after sudo), exe=/usr/bin/cat, ses= (the session id) and tty. With ausearch -k canary --format csv you get something parseable.

You get the source IP of the SSH session via the session id: journalctl _AUDIT_SESSION=<ses> -u ssh | grep Accepted. Or user auid plus timestamp against last -i.

Step 3: the alert

Every five minutes, push the new canary events with who/what/where:

sudo tee /usr/local/sbin/canary-watch.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
NTFY="https://ntfy.example.be/security"
HOST=$(hostname -s)
# --checkpoint: ausearch remembers how far it has read — no duplicate alerts
CHK=/var/lib/canary-watch.checkpoint

# One line per event: auid→name, exe, file, session
EVENTS=$(ausearch -k canary --checkpoint "$CHK" -i 2>/dev/null \
  | awk '/^type=SYSCALL/ { for(i=1;i<=NF;i++){ if($i~/^auid=/)a=$i; if($i~/^exe=/)e=$i; if($i~/^ses=/)s=$i } }
         /^type=PATH/ && /name=/ { for(i=1;i<=NF;i++) if($i~/^name=/) f=$i; print a, e, f, s }' \
  | sort -u)

if [ -n "$EVENTS" ]; then
  { echo "CANARY TOUCHED on $HOST"; echo "$EVENTS";
    echo; echo "sessions:"; echo "$EVENTS" | grep -oE 'ses=[0-9]+' | sort -u | while read -r s; do
      journalctl "_AUDIT_SESSION=${s#ses=}" --no-pager 2>/dev/null | grep -m1 -E 'Accepted (publickey|password)' ; done; } \
  | curl -s -H "Title: HONEYPOT $HOST" -H "Priority: urgent" -H "Tags: rotating_light" --data-binary @- "$NTFY" >/dev/null
fi
EOF
sudo chmod 0755 /usr/local/sbin/canary-watch.sh
echo '*/5 * * * * root /usr/local/sbin/canary-watch.sh' | sudo tee /etc/cron.d/canary-watch

This is an alert with priority urgent. There is no legitimate reason for a canary to be read; every notification is either a break-in or an exception you should have configured in step 4.

Step 4: the exceptions (or day one is a disaster)

Three kinds of software read all files and trigger your canaries within 24 hours:

File indexers. updatedb (mlocate/plocate) runs daily and stats everything — but doesn't read, so with -p rwa (no x, no stat) that's fine. Exclude anyway, to be safe:

# /etc/updatedb.conf
PRUNEPATHS="/tmp /var/spool /media /var/lib/os-prober /var/lib/ceph /home/.ecryptfs /var/lib/schroot /root/.ssh /etc/backup /home/deploy/.aws"

Backup tools. restic, borg, rsync and Veeam agents read every file. Exclude the canaries in the backup config — you don't want them in your backup either, because then your fake credentials end up in an archive that might leak later:

# restic
restic backup / --exclude-file=/etc/restic/excludes
# in /etc/restic/excludes:
/root/.ssh/id_rsa.monsys-backup
/home/deploy/.aws/credentials
/etc/backup/db-passwords.txt
/var/www/.env.bak

Malware scanners and compliance tools. ClamAV, Lynis, OpenSCAP and rkhunter read files. If you run those, give them their own exclusion or accept that a notification with exe=/usr/bin/clamscan at 03:00 is expected — and filter it in the script on exe=.

Then run 48 hours without a single notification before treating the alert as "urgent". Every notification in those 48 hours is an exception you didn't know about yet.

Step 5: the web server user too

The .env.bak file is deliberately readable by www-data (mode 0640 with the right group). An attacker entering through a PHP flaw runs as www-data and has no sudo; you still want to see them. Verify that auditd logs those reads too:

sudo chgrp www-data /var/www/.env.bak
sudo -u www-data cat /var/www/.env.bak >/dev/null
sudo ausearch -k canary -ts recent -i | grep -E 'uid=www-data'

Pitfalls

• -p x on a file. Executing a text file fails but triggers. Leave x out, or you get noise from tab completion and editors checking executable bits.
• auditd buffer full. On busy servers auditd loses events if -b (backlog) is too small. Put -b 8192 at the top of /etc/audit/rules.d/10-base.rules and check auditctl -s for lost.
• A canary in git. /var/www/.env.bak in a deploy directory cleaned with git clean -fdx disappears on the next deploy. Put it in .git/info/exclude or one directory up.
• Too many canaries. Four well-placed files catch more than forty. Every extra file is an extra exclusion in your backup and your indexer.
• Never tested. Set a quarterly reminder: cat one canary, verify the push arrives within five minutes. Detection you never test doesn't exist.

What you still don't have

• Correlation with the entry path. The canary alert says that someone read; which SSH login or which web request preceded it, you have to piece together from two logs yourself — under pressure, in the middle of the night.
• Fleet view. The same auid reading canaries on three servers within ten minutes is one break-in. Three separate ntfy messages don't show that.
• Evidence. For a NIS2 incident notification (24-hour early warning) you want a timeline you can hand to the CCB, signed, not an ausearch dump.

How monsys does it

The monsys agent plants and watches the canary files itself (read, write and delete), without auditd configuration on the host, and reports a honeypot_trip with user, process and session. The hub links that to the SSH detection: a honeypot trip plus an internal SSH login within ±5 minutes becomes a lateral movement detection with the full chain. Every detection lands in the signed audit trail, so the timeline for an incident notification is already there. Intervening (killing the process, isolating the host) remains a human decision via an Emergency Action Token.

FAQ

Are honeypot files legal?

Yes. They're files on your own server. They do nothing outward, deceive nobody outside your system and collect only what the kernel already logs. This is fundamentally different from an active honeypot server that attracts attackers.

Does this also catch an attacker who is root?

Yes, as long as auditd is running. Root can stop auditd or wipe the rules, but that is itself an event (auditctl -D is logged as CONFIG_CHANGE). Anyone that thorough has usually given themselves away long before.

Why not inotify instead of auditd?

inotifywait works and is simpler, but it doesn't see who read the file — no uid, no process, no session. For an alert you have to act on at 03:00, "someone read it" is too little.