Checking whether a running process is still the original binary: /proc/<pid>/exe, sha256 and dpkg -V

ps shows a process named sshd. That says nothing about what's running. The name is the first argument; the file may have been replaced; the process may have been started from a binary that was deleted afterwards, so there's nothing suspicious left on disk. Whoever takes over a server rarely replaces ls — they replace sshd (to log passwords) or make an existing process load an injected library. This article gives you three independent checks that work on any Linux host without an agent, plus the cron script that combines them.

Check 1: what's really running? — /proc/<pid>/exe

For every process the kernel keeps a symlink /proc/<pid>/exe to the file that was executed. That link follows the inode, not the name: if the file is replaced (cp new /usr/sbin/sshd), the old process's link still points at the old inode and gets the suffix (deleted).

# Per process: pid, name, real path
sudo ls -l /proc/[0-9]*/exe 2>/dev/null | awk '{print $NF, $(NF-2)}' | sed 's|/proc/\([0-9]*\)/exe|\1|' | sort -u | head

# Processes running from a DELETED file
sudo find /proc -maxdepth 2 -name exe -lname '*(deleted)*' 2>/dev/null \
  | while read -r l; do pid=${l#/proc/}; pid=${pid%/exe}; printf '%s\t%s\t%s\n' "$pid" "$(cat /proc/$pid/comm)" "$(sudo readlink "$l")"; done
# 1437  agetty  /usr/sbin/agetty (deleted)   ← util-linux was updated; getty still runs the old one

A (deleted) process is usually innocent: a service not yet restarted after an apt upgrade (see unattended-upgrades). But the combination "deleted" + "no package update in the logs" + "path in /tmp or /dev/shm" is exactly how malware hides.

# Processes whose binary lives in a suspicious directory (even without deleted)
sudo ls -l /proc/[0-9]*/exe 2>/dev/null | awk '{print $NF}' | grep -E '^/(tmp|dev/shm|var/tmp|run/user|home)/' | sort | uniq -c

Check 2: does the binary still match the package? — dpkg -V

Debian and Ubuntu keep an MD5 sum for every installed file in /var/lib/dpkg/info/*.md5sums. dpkg -V compares the files on disk against them.

# Everything deviating from the package (reads EVERY file of EVERY package: expect minutes, not seconds)
sudo dpkg -V 2>/dev/null | grep -vE '^\?\?5\?+ +c ' | head -30
# Reading the output: "??5??????" = MD5 differs; " c " = config file (expected); "missing" = gone
# Only binaries and libraries:
sudo dpkg -V 2>/dev/null | awk '$NF ~ /^\/(usr\/)?(s?bin|lib)/ && $2 != "c"'

A deviating binary in /usr/sbin that is not a config file has exactly two explanations: someone replaced it, or you patched it yourself and forgot. RHEL family: rpm -Va.

The limitation: dpkg -V checks against the md5sums the package itself shipped. Whoever is root can modify those too. Hence check 3.

Check 3: a baseline that doesn't live on the host

Take a sha256 of every running executable and keep it off the server. Then compare daily.

sudo tee /usr/local/sbin/exe-baseline.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
# Prints sha256 of the file behind /proc/<pid>/exe for every unique executable.
# Hashes via /proc/<pid>/exe itself, so a deleted binary gets hashed too.
for l in /proc/[0-9]*/exe; do
  pid=${l#/proc/}; pid=${pid%/exe}
  target=$(readlink "$l" 2>/dev/null) || continue
  [ -z "$target" ] && continue
  h=$(sha256sum "$l" 2>/dev/null | awk '{print $1}') || continue
  printf '%s\t%s\n' "$h" "$target"
done | sort -u -k2,2
EOF
sudo chmod 0755 /usr/local/sbin/exe-baseline.sh

# Create the baseline on a clean host, and keep it AWAY from the host
sudo /usr/local/sbin/exe-baseline.sh > /tmp/exe-baseline.tsv
scp /tmp/exe-baseline.tsv mgmt-host:/srv/baselines/$(hostname -s).tsv

The daily comparison, from the management host (so the baseline isn't on the machine being checked):

sudo tee /usr/local/sbin/exe-verify.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
# Runs on the management host; checks every host in hosts.txt against its baseline.
NTFY="https://ntfy.example.be/security"
B=/srv/baselines
while read -r h; do
  cur=$(ssh -o BatchMode=yes -o ConnectTimeout=5 "$h" sudo /usr/local/sbin/exe-baseline.sh 2>/dev/null) || { echo "$h unreachable"; continue; }
  base="$B/$h.tsv"; [ -f "$base" ] || { echo "$cur" > "$base"; echo "$h: baseline created"; continue; }
  # Same path, different hash = replaced (or patched). New path = new process.
  changed=$(join -t $'\t' -1 2 -2 2 <(sort -t $'\t' -k2,2 "$base") <(echo "$cur" | sort -t $'\t' -k2,2) | awk -F'\t' '$2!=$3 {print $1}')
  new=$(comm -13 <(cut -f2 "$base" | sort) <(echo "$cur" | cut -f2 | sort))
  deleted=$(echo "$cur" | grep -c '(deleted)')
  msg=""
  [ -n "$changed" ] && msg+="CHANGED BINARY: $(echo "$changed" | paste -sd, -)\n"
  [ -n "$new" ] && msg+="new executables: $(echo "$new" | paste -sd, -)\n"
  [ "$deleted" -gt 0 ] && msg+="$deleted process(es) running from deleted files\n"
  [ -n "$msg" ] && printf "$msg" | curl -s -H "Title: integrity $h" -H "Priority: urgent" --data-binary @- "$NTFY" >/dev/null
done < /srv/baselines/hosts.txt
EOF
sudo chmod 0755 /usr/local/sbin/exe-verify.sh
echo '40 * * * * root /usr/local/sbin/exe-verify.sh' | sudo tee /etc/cron.d/exe-verify

After a legitimate update (apt upgrade + service restart) the hash changes, of course. The script reports it; you confirm against dpkg.log and refresh the baseline:

grep "$(date +%F)" /var/log/dpkg.log | grep ' upgrade ' | awk '{print $4}'   # which packages today?
ssh web-01 sudo /usr/local/sbin/exe-baseline.sh > /srv/baselines/web-01.tsv   # refresh baseline AFTER verification

Check 4 (bonus): loaded libraries and LD_PRELOAD

A process doesn't need to be replaced to be hijacked: a library injection via LD_PRELOAD or /etc/ld.so.preload is enough. Two quick checks:

# Does /etc/ld.so.preload exist? On a regular server it shouldn't.
ls -la /etc/ld.so.preload 2>/dev/null && cat /etc/ld.so.preload
# Processes with LD_PRELOAD in their environment
for e in /proc/[0-9]*/environ; do pid=${e#/proc/}; pid=${pid%/environ}; sudo cat "$e" 2>/dev/null | tr '\0' '\n' | grep -q '^LD_PRELOAD=' && echo "$pid $(cat /proc/$pid/comm)"; done
# Libraries loaded from suspicious paths
sudo awk '/\.so/ && $NF ~ /^\/(tmp|dev\/shm|var\/tmp|home)\// {print FILENAME, $NF}' /proc/[0-9]*/maps 2>/dev/null | sort -u | head

Pitfalls

• Forgetting to restart after an update. Most (deleted) reports are services not restarted after a library or binary update. That's not a break-in, but it is a reason to enable needrestart — otherwise the real report drowns in the noise.
• Interpreters. For a Python or Node service, /proc/<pid>/exe is the interpreter (/usr/bin/python3.12), not your application. The interpreter's hash is fine while the app code was modified. Hash the app directory separately (find /srv/app -type f -exec sha256sum {} +), or rely on git rather than dpkg -V.
• Baseline on the host. A baseline in /var/lib/… on the same machine can be modified along by an attacker with root. Hence the management host.
• Prelink and ASLR. On systems with prelink (rare nowadays) binaries on disk change legitimately. Debian/Ubuntu don't use it.
• Containers. /proc/<pid>/exe of a container process points to a path in the container filesystem (/api, /bin/node_exporter); the baseline contains those too, but dpkg -V on the host knows nothing about them. Hash per image, or use Trivy.

What you still don't have

• The first baseline. If the host was already compromised when you created the baseline, every deviation afterwards is "normal". Create it right after installation, or compare against a fresh install of the same package.
• Correlation. A replaced binary and a new SSH key and a login from a new country within one hour: three scripts, three messages, no story.
• Evidence. For a NIS2 incident notification you want the timeline (when the hash changed, which session was active) signed and retained.

How monsys does it

The monsys agent does this continuously as Process DNA: on the first run it takes the SHA256 of /proc/<pid>/exe of every running executable as baseline, and afterwards reports every divergence — replaced binary, process from a deleted file, executable in /tmp or /dev/shm — as process_dna_divergence. The baseline lives in the hub, not on the host. After a legitimate package update the hub sees the matching dpkg inventory change and marks the new hash as expected, so only the unexplained deviation becomes an alert. See integrity monitoring in the docs.

FAQ

How do I see whether a running process has been modified?

Compare the hash of /proc/<pid>/exe with a previously stored baseline, and check whether the link carries the (deleted) suffix. dpkg -V <package> additionally compares the file on disk with the checksum from the package.

Is dpkg -V reliable as evidence?

As a first indication yes; as evidence no, because the md5sums sit on the same host as the binaries and can be modified by root. A baseline outside the host (or an agent that keeps the hashes elsewhere) is the answer.

What if a process runs from a deleted file?

First check today's /var/log/dpkg.log: if the package was just updated, restart the service. If not, and especially if the path is in /tmp, /dev/shm or a home directory: treat it as an incident, isolate the host and preserve /proc/<pid>/exe (cp /proc/<pid>/exe /root/evidence-<pid>) before stopping the process.