9.9 KiB

Raw Blame History

Proxmox VE 8 to 9 Upgrade Execution Record

Tracking issue: https://github.com/CCOSTAN/Home-AssistantConfig/issues/1745

This records the 2026-06-23 Proxmox VE 8 to 9 cluster upgrade and keeps the reusable validation notes for future maintenance.

Official References

Proxmox VE support lifecycle: https://pve.proxmox.com/wiki/FAQ#faq-support-table
Proxmox VE 8 to 9 guide: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9
Proxmox VE 9 known issues: https://pve.proxmox.com/wiki/Roadmap#9.0-known-issues
Debian 13 release notes: https://www.debian.org/releases/trixie/release-notes/

Execution Summary

Completed on 2026-06-23.

Both Proxmox nodes upgraded from Proxmox VE 8.4.19 to pve-manager/9.2.3/d0fde103346cf89a.
Both nodes rebooted successfully into kernel 7.0.12-1-pve.
Final pve8to9 --full output on both nodes had FAILURES: 0; the only warning left was the expected two-node quorum warning.
apt update completed cleanly on both nodes after disabling the PVE enterprise source and using the PVE 9 no-subscription repository.
Final simulated apt dist-upgrade on both nodes showed no pending installs or removals.
Shared storage was active on both nodes after the upgrade.
HA-managed VMs were restored to their normal node, started, and visible in HA status.
The camera/NVR VM was started again after the node upgrades; the Frigate container reported Docker health healthy.

Changes Applied During Upgrade

Captured host configuration backups before package changes.
Applied the pending Proxmox VE 8.4 package updates and rebooted both nodes into kernel 6.8.12-30-pve before changing repositories.
Removed the unused systemd-boot meta-package after pve8to9 flagged it and verified the nodes were booting through GRUB EFI.
Configured GRUB to maintain the removable EFI path and reinstalled grub-efi-amd64 on both nodes.
Switched Debian repositories from Bookworm to Trixie with non-free-firmware enabled.
Added the Proxmox VE 9 no-subscription deb822 source.
Installed intel-microcode on both nodes; final TSX status was Mitigation: TSX disabled.
Changed VM CPU model from Skylake-Client to Skylake-Client-noTSX-IBRS for all cluster VMs. This was required because the upgraded/microcode-protected node no longer exposed TSX flags used by the previous VM CPU model.
Disabled the generated PVE enterprise deb822 source with Enabled: no so no-subscription APT updates remain clean.
On 2026-06-24, enrolled the updated Microsoft UEFI 2023 certificates for the camera/NVR VM EFI disk with qm enroll-efi-keys, then started the VM with no EFI certificate warning and verified Frigate Docker health healthy.

Residual Follow-Ups

Full retained VM backup archives were not found during inventory. Host config backups were captured, but this should be improved with a real enabled VM backup target and restore test.
pve8to9 reported old-format RRD files as informational post-upgrade data. Keep them if historical graphs are useful; delete only if old history is not needed.

Initial Evidence

Captured on 2026-06-23 through the Proxmox API and the official Proxmox documentation.

Cluster health was healthy with both nodes online.
Cluster release endpoint reported Proxmox VE 8.4.19 / release 8.4.
Both nodes reported pve-manager/8.4.19/a68fb383814bb1e6.
Both nodes were booted on kernel 6.8.12-29-pve.
Both nodes had pending packages: proxmox-kernel-6.8 to 6.8.12-30, proxmox-kernel-6.8.12-30-pve-signed, and libhttp-daemon-perl.
Both nodes used EFI boot mode with Secure Boot disabled.
Root filesystems had more than 10 GB free.
Shared VM-image storage was online on both nodes and lightly used.
Local Proxmox backup storage on both nodes had no backups.
The only Proxmox cluster backup job found was disabled.
All current VM disks were on shared storage, which keeps migration feasible.
The high-load camera/standby-DNS VM was powered off during the upgrade work to reduce load.
Repository drift cleanup completed on 2026-06-23: the duplicate Proxmox no-subscription install repo and inactive Ceph Quincy repo were disabled on both nodes through the Proxmox repository API. apt update then completed OK on both nodes with no warnings or errors.

Readiness Checklist

Read issue #1745 and official Proxmox lifecycle/upgrade documents.
Capture cluster health and live resource posture.
Capture current Proxmox VE release, node kernel, boot mode, storage, VM, repository, and pending-package state.
Identify initial load-management target for planning work.
Disable duplicate/stale Proxmox and Ceph APT repository entries, then confirm clean apt update output through the Proxmox API.
Apply pending Proxmox VE 8.4 updates on both nodes.
Reboot each node as needed and confirm it returns healthy.
Run pve8to9 --full on each node and save the output.
Verify full VM backups and at least one restore path for every critical guest.
Confirm recovery access that does not depend on a running guest.
Confirm an operator console path that can run host shell commands: SSH, out-of-band/local console, or an approved logged-in Proxmox UI console.
Confirm final guest move/shutdown order for each node.
Execute the Proxmox VE 9 upgrade in a maintenance window.

Guest Handling Strategy

Upgrade the quieter node first after the camera/standby-DNS VM is stopped. This keeps the first host upgrade low-risk and proves the upgrade path on a node that does not currently carry the main always-on workloads.

For the busier node, keep the camera/standby-DNS VM stopped and migrate or stop guests deliberately. The current memory footprint suggests not every active VM can be moved to the other node at once. Treat the home-automation VM as the highest-priority guest to keep running, then decide which internal tooling or edge/public VMs can be stopped temporarily.

Suggested order:

Update and upgrade the quieter node while the camera/standby-DNS VM remains stopped.
Validate the quieter node after reboot.
Move only the required critical guest set to the upgraded node.
Stop non-critical guests that do not fit safely during the busier-node maintenance window.
Upgrade the busier node.
Validate the busier node after reboot.
Restore normal guest placement and power state.

Preflight Commands

Run from a durable console session such as tmux or screen on the node being worked. Capture output for the issue before changing repositories.

pveversion -v
uname -a
apt update
apt list --upgradable
apt policy
pvesm status
pvesh get /cluster/status
pvesh get /cluster/resources --type vm
pve8to9 --full

Review:

No node is degraded.
No storage used by guests is unavailable.
No unexpected Bookworm, test, stale Ceph, or duplicate repository entries are present.
No guest depends on local-only disks unless intentionally handled.
pve8to9 --full warnings are understood before continuing.

Bring Nodes Current on 8.4

Do this before any Debian Trixie or Proxmox VE 9 repository changes.

apt update
apt dist-upgrade
pveversion

If the kernel changes, reboot the node and verify it returns cleanly before updating the next node.

Upgrade Procedure Template

Follow the official Proxmox guide for exact repository content and any changes made after this document was written.

For each node, after backups, recovery access, and pve8to9 --full are clean or explicitly accepted:

tmux new -s pve9-upgrade
pve8to9 --full
apt update
apt dist-upgrade
pve8to9 --full
reboot

For the actual 8 to 9 step, update Debian and Proxmox repositories from Bookworm to Trixie, use the Proxmox VE 9 repository format documented by Proxmox, then run:

apt update
apt policy
apt dist-upgrade
pve8to9 --full
reboot

Do not accept any apt transaction that removes the proxmox-ve package unless the official guide and pve8to9 output explicitly explain the situation.

Known-Issue Checks

Check these before the maintenance window:

cgroup v1 removal and any legacy container assumptions.
Network interface naming changes.
EFI, GRUB, LVM, and systemd-boot warnings.
LVM/LVM-thin autoactivation warnings from pve8to9.
Third-party repositories and storage plugins.
Ceph repository state. If Ceph is not actually used, remove stale Ceph repo entries before the upgrade; if it is used, follow the official Ceph Squid prerequisite first.
Kernel 6.14 compatibility with older hardware and any PCI/USB passthrough.
systemd-sysctl no longer reading /etc/sysctl.conf; move persistent sysctl settings to /etc/sysctl.d/.
Debian 13 /tmp behavior changes.

Post-Upgrade Validation

Validate after each node, not only at the end.

pveversion -v
uname -a
pvesh get /cluster/status
pvesm status
pvesh get /cluster/resources --type vm
apt update
apt list --upgradable
pve8to9 --full

Also verify:

Proxmox UI/API reachable.
Cluster shows both nodes online after each reboot.
Shared storage is mounted on both nodes.
Migrated guests start and report healthy.
Home Assistant, MQTT, DNS, camera/NVR, edge apps, backups, and monitoring are healthy according to their normal validation surfaces.
Home Assistant Proxmox telemetry and repair automations return to normal.

Rollback and Recovery Notes

A failed major upgrade is not a simple package downgrade. Treat recovery as restore-from-backup or repair-in-place using the official troubleshooting guide.
Keep one node unchanged while upgrading the other.
Keep guest backups and config exports outside the node being upgraded.
Preserve /etc/pve, network config, storage config, and any non-default host config before the upgrade.
If apt reports that proxmox-ve would be removed, stop and fix repository configuration before proceeding.
If a node fails to boot, use host-independent recovery access and the official GRUB/recovery guidance.

9.9 KiB Raw Blame History