Configuring unattended-upgrades properly — and why you must monitor reboot-required

apt install unattended-upgrades and done — that's what most servers have, and it's better than nothing. But the default configuration has three holes: services keep running with the old library in memory after a libssl update (so the CVE is still live), a new kernel gets installed but never booted, and nobody is told what happened. This article closes those three holes and adds a fleet-wide reboot check next to them.

Step 1: install and check the basics

sudo apt install -y unattended-upgrades apt-listchanges needrestart
# Are the periodic timers on?
cat /etc/apt/apt.conf.d/20auto-upgrades
# APT::Periodic::Update-Package-Lists "1";
# APT::Periodic::Unattended-Upgrade "1";
systemctl list-timers 'apt-daily*' --no-legend

If 20auto-upgrades is missing (sometimes on Debian minimal): sudo dpkg-reconfigure -plow unattended-upgrades creates it.

Step 2: the configuration you want

Everything goes in /etc/apt/apt.conf.d/52unattended-upgrades-local — your own file with a higher number overrides the defaults from 50unattended-upgrades without a package update overwriting your changes.

sudo tee /etc/apt/apt.conf.d/52unattended-upgrades-local >/dev/null <<'EOF'
// Ubuntu: security + ESM. Regular -updates deliberately NOT automatic (those contain
// feature changes; you plan those). Debian uses Origins-Pattern, see below.
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
};

// What you NEVER want unplanned: kernel (separate guide), Docker, databases.
Unattended-Upgrade::Package-Blacklist {
    "linux-image-";
    "linux-headers-";
    "docker-ce";
    "containerd.io";
    "postgresql-";
    "mysql-server";
    "mariadb-server";
};

// Cleanup
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";

// Reboot: NOT automatic in production. Do report (step 4).
Unattended-Upgrade::Automatic-Reboot "false";
// On dev/staging it's fine, in a fixed window, and not while someone is logged in:
// Unattended-Upgrade::Automatic-Reboot "true";
// Unattended-Upgrade::Automatic-Reboot-Time "03:30";
// Unattended-Upgrade::Automatic-Reboot-WithUsers "false";

// Mail on changes or errors (requires a working local mailer or msmtp)
Unattended-Upgrade::Mail "ops@example.be";
Unattended-Upgrade::MailReport "on-change";

// To syslog/journal, so it's in your logs
Unattended-Upgrade::SyslogEnable "true";

// Complete half-done installs instead of stopping
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::InstallOnShutdown "false";
EOF

On Debian 12 replace the Allowed-Origins block with:

Unattended-Upgrade::Origins-Pattern {
    "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
    "origin=Debian,codename=${distro_codename},label=Debian-Security";
};

Always dry-run first:

sudo unattended-upgrade --dry-run --debug 2>&1 | grep -E 'Allowed origins|Packages that will be upgraded|blacklist|No packages'

Step 3: restarting services after a library update (needrestart)

This is the hole most often left open. After apt upgrade libssl3t64, nginx keeps running with the old libssl in memory until someone restarts it. needrestart detects that and can do it automatically — but defaults to list mode when there's no terminal.

sudo tee /etc/needrestart/conf.d/50-auto.conf >/dev/null <<'EOF'
# a = restart automatically, l = list only, i = ask (useless without a tty)
$nrconf{restart} = 'a';
# Services you do NOT want restarted automatically (regex on unit name)
$nrconf{override_rc}{qr(^postgresql)} = 0;
$nrconf{override_rc}{qr(^mariadb)}    = 0;
$nrconf{override_rc}{qr(^docker)}     = 0;
$nrconf{override_rc}{qr(^ssh)}        = 1;   # ssh is fine: existing sessions stay open
EOF
# What would it restart now?
sudo needrestart -r l

needrestart runs automatically after every apt run via the hook in /etc/apt/apt.conf.d/99needrestart. With restart = 'a' it restarts everything running on an outdated library, except what you excluded. Those excluded services (databases) show up in the reboot check of step 4 as "needs restart".

Step 4: making reboot-required visible fleet-wide

After an update the new kernel sits on disk and /var/run/reboot-required appears. Nobody sees that file. This script reports every morning which hosts are waiting, and for how long:

sudo tee /usr/local/sbin/reboot-check.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
NTFY="https://ntfy.example.be/patching"
HOST=$(hostname -s)
MSG=()

if [ -f /var/run/reboot-required ]; then
  days=$(( ( $(date +%s) - $(stat -c %Y /var/run/reboot-required) ) / 86400 ))
  pkgs=$(sort -u /var/run/reboot-required.pkgs 2>/dev/null | paste -sd, -)
  MSG+=("reboot-required since ${days}d (${pkgs:-?})")
fi

# Services running on outdated libraries (needrestart in list mode, batch)
stale=$(needrestart -b 2>/dev/null | awk -F': ' '/^NEEDRESTART-SVC/{print $2}' | paste -sd, -)
[ -n "$stale" ] && MSG+=("services on old libs: $stale")

# Last successful unattended-upgrade run
last=$(grep -h 'Packages that were upgraded' /var/log/unattended-upgrades/unattended-upgrades.log 2>/dev/null | tail -1 | cut -c1-19)
[ -n "$last" ] && MSG+=("last auto-upgrade: $last")

# Pending security updates NOT installed (blacklist or error)
open=$(apt list --upgradable 2>/dev/null | grep -c security)
[ "$open" -gt 0 ] && MSG+=("$open security updates pending")

if [ ${#MSG[@]} -gt 0 ]; then
  printf '%s\n' "${MSG[@]}" | curl -s -H "Title: patch@$HOST" -H "Priority: default" --data-binary @- "$NTFY" >/dev/null
fi
EOF
sudo chmod 0755 /usr/local/sbin/reboot-check.sh
echo '30 7 * * * root /usr/local/sbin/reboot-check.sh' | sudo tee /etc/cron.d/reboot-check

The ntfy topic patching becomes your to-do list every morning: hosts waiting three days for a reboot, and databases not yet restarted after a libssl update.

Step 5: controlling the timing

apt-daily-upgrade.timer fires at 06:00 with up to 60 minutes of random delay. On a fleet that means: all hosts patch at the same time, in the middle of some customers' morning peak. Move it:

sudo systemctl edit apt-daily-upgrade.timer

[Timer]
OnCalendar=
OnCalendar=*-*-* 03:00
RandomizedDelaySec=45min

Giving different groups (web, db, batch) a different hour is the simplest form of staged rollout: if something breaks on the web group at 03:00, the db group isn't due until 04:00 and you can intervene.

Pitfalls

• -updates in the origins on Ubuntu. Some guides add ${distro_id}:${distro_codename}-updates. That also installs non-security updates with behavioural changes (new PHP minor, new systemd). Only do that on hosts where you deliberately want it.
• Phased updates. Ubuntu rolls some updates out in phases; apt list --upgradable shows them as "deferred". That's normal, not an error. Want them immediately anyway: APT::Get::Always-Include-Phased-Updates "true"; — but then you're volunteering as a guinea pig.
• dpkg lock conflicts. Ansible, Puppet or a manual apt at 03:00 collides with the lock and fails with "Could not get lock". Schedule your config management outside the window from step 5.
• Mail that never arrives. Mail "ops@example.be" does nothing without an MTA. Install msmtp-mta with a relay, or drop the mail and rely on step 4.
• Automatic-Reboot "true" on a host with DKMS modules. After the reboot a network or storage module may be missing. See kernel updates in production before enabling automatic reboots.
• Docker containers don't see host updates. A patched libssl on the host does nothing for the libssl in your python:3.12 image. Container images are a separate patch stream.

What you still don't have

• Overview. Thirty ntfy messages per morning aren't a dashboard. "Which hosts have been waiting longer than seven days?" requires a script across the messages.
• Evidence of lead time. An auditor wants "days between USN publication and installation, p95". That sits scattered in unattended-upgrades.log and dpkg.log per host.
• Rollback. A security update that breaks a service is installed at 03:00 and discovered at 08:00. unattended-upgrades has no "stop on errors on other hosts" mechanism.

How monsys does it

The monsys agent reports per host the pending updates, reboot_required (with since when) and the services running on outdated libraries; the hub shows that as one fleet-wide list and measures patch lead time against the publication date of the USN or DSA. Those who want to go further enable auto-patch: per host and per category (OS packages, app dependencies) explicit opt-in with TOTP, executed via signed Emergency Action Tokens, with a rollback gate that pauses the loop across the whole fleet as soon as one batch fails. The "observes and reports, never intervenes on its own" principle stands: no action without a human having permitted it in advance.

FAQ

Should I enable automatic updates in production?

Security updates: yes, with the blacklist from step 2 for kernel, databases and Docker. The chance that a security update breaks something is small; the chance that an uninstalled security update is exploited is bigger. Feature updates (-updates) you plan deliberately.

Why is a security update still pending while unattended-upgrades runs?

Three causes, in order of likelihood: the package is on the blacklist (kernel, database), the update is "phased" and not yet assigned to your host, or the update requires a package from an origin you haven't allowed (e.g. -updates). unattended-upgrade --dry-run --debug tells you which.

Does needrestart replace a reboot?

For library updates: yes, it restarts the services using the old library. For kernel, glibc and systemd updates: no, a reboot remains necessary, and that's exactly what /var/run/reboot-required indicates.