# Proxmox VE 8 to 9 Upgrade Execution Record Tracking issue: https://github.com/CCOSTAN/Home-AssistantConfig/issues/1745 This records the 2026-06-23 Proxmox VE 8 to 9 cluster upgrade and keeps the reusable validation notes for future maintenance. ## Official References - Proxmox VE support lifecycle: https://pve.proxmox.com/wiki/FAQ#faq-support-table - Proxmox VE 8 to 9 guide: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 - Proxmox VE 9 known issues: https://pve.proxmox.com/wiki/Roadmap#9.0-known-issues - Debian 13 release notes: https://www.debian.org/releases/trixie/release-notes/ ## Execution Summary Completed on 2026-06-23. - Both Proxmox nodes upgraded from Proxmox VE `8.4.19` to `pve-manager/9.2.3/d0fde103346cf89a`. - Both nodes rebooted successfully into kernel `7.0.12-1-pve`. - Final `pve8to9 --full` output on both nodes had `FAILURES: 0`; the only warning left was the expected two-node quorum warning. - `apt update` completed cleanly on both nodes after disabling the PVE enterprise source and using the PVE 9 no-subscription repository. - Final simulated `apt dist-upgrade` on both nodes showed no pending installs or removals. - Shared storage was active on both nodes after the upgrade. - HA-managed VMs were restored to their normal node, started, and visible in HA status. - The camera/NVR VM was started again after the node upgrades; the Frigate container reported Docker health `healthy`. ## Changes Applied During Upgrade - Captured host configuration backups before package changes. - Applied the pending Proxmox VE 8.4 package updates and rebooted both nodes into kernel `6.8.12-30-pve` before changing repositories. - Removed the unused `systemd-boot` meta-package after `pve8to9` flagged it and verified the nodes were booting through GRUB EFI. - Configured GRUB to maintain the removable EFI path and reinstalled `grub-efi-amd64` on both nodes. - Switched Debian repositories from Bookworm to Trixie with `non-free-firmware` enabled. - Added the Proxmox VE 9 no-subscription deb822 source. - Installed `intel-microcode` on both nodes; final TSX status was `Mitigation: TSX disabled`. - Changed VM CPU model from `Skylake-Client` to `Skylake-Client-noTSX-IBRS` for all cluster VMs. This was required because the upgraded/microcode-protected node no longer exposed TSX flags used by the previous VM CPU model. - Disabled the generated PVE enterprise deb822 source with `Enabled: no` so no-subscription APT updates remain clean. - On 2026-06-24, enrolled the updated Microsoft UEFI 2023 certificates for the camera/NVR VM EFI disk with `qm enroll-efi-keys`, then started the VM with no EFI certificate warning and verified Frigate Docker health `healthy`. ## Residual Follow-Ups - Full retained VM backup archives were not found during inventory. Host config backups were captured, but this should be improved with a real enabled VM backup target and restore test. - `pve8to9` reported old-format RRD files as informational post-upgrade data. Keep them if historical graphs are useful; delete only if old history is not needed. ## Initial Evidence Captured on 2026-06-23 through the Proxmox API and the official Proxmox documentation. - Cluster health was healthy with both nodes online. - Cluster release endpoint reported Proxmox VE `8.4.19` / release `8.4`. - Both nodes reported `pve-manager/8.4.19/a68fb383814bb1e6`. - Both nodes were booted on kernel `6.8.12-29-pve`. - Both nodes had pending packages: `proxmox-kernel-6.8` to `6.8.12-30`, `proxmox-kernel-6.8.12-30-pve-signed`, and `libhttp-daemon-perl`. - Both nodes used EFI boot mode with Secure Boot disabled. - Root filesystems had more than 10 GB free. - Shared VM-image storage was online on both nodes and lightly used. - Local Proxmox backup storage on both nodes had no backups. - The only Proxmox cluster backup job found was disabled. - All current VM disks were on shared storage, which keeps migration feasible. - The high-load camera/standby-DNS VM was powered off during the upgrade work to reduce load. - Repository drift cleanup completed on 2026-06-23: the duplicate Proxmox no-subscription install repo and inactive Ceph Quincy repo were disabled on both nodes through the Proxmox repository API. `apt update` then completed OK on both nodes with no warnings or errors. ## Readiness Checklist - [x] Read issue #1745 and official Proxmox lifecycle/upgrade documents. - [x] Capture cluster health and live resource posture. - [x] Capture current Proxmox VE release, node kernel, boot mode, storage, VM, repository, and pending-package state. - [x] Identify initial load-management target for planning work. - [x] Disable duplicate/stale Proxmox and Ceph APT repository entries, then confirm clean `apt update` output through the Proxmox API. - [x] Apply pending Proxmox VE 8.4 updates on both nodes. - [x] Reboot each node as needed and confirm it returns healthy. - [x] Run `pve8to9 --full` on each node and save the output. - [ ] Verify full VM backups and at least one restore path for every critical guest. - [x] Confirm recovery access that does not depend on a running guest. - [x] Confirm an operator console path that can run host shell commands: SSH, out-of-band/local console, or an approved logged-in Proxmox UI console. - [x] Confirm final guest move/shutdown order for each node. - [x] Execute the Proxmox VE 9 upgrade in a maintenance window. ## Guest Handling Strategy Upgrade the quieter node first after the camera/standby-DNS VM is stopped. This keeps the first host upgrade low-risk and proves the upgrade path on a node that does not currently carry the main always-on workloads. For the busier node, keep the camera/standby-DNS VM stopped and migrate or stop guests deliberately. The current memory footprint suggests not every active VM can be moved to the other node at once. Treat the home-automation VM as the highest-priority guest to keep running, then decide which internal tooling or edge/public VMs can be stopped temporarily. Suggested order: 1. Update and upgrade the quieter node while the camera/standby-DNS VM remains stopped. 2. Validate the quieter node after reboot. 3. Move only the required critical guest set to the upgraded node. 4. Stop non-critical guests that do not fit safely during the busier-node maintenance window. 5. Upgrade the busier node. 6. Validate the busier node after reboot. 7. Restore normal guest placement and power state. ## Preflight Commands Run from a durable console session such as `tmux` or `screen` on the node being worked. Capture output for the issue before changing repositories. ```bash pveversion -v uname -a apt update apt list --upgradable apt policy pvesm status pvesh get /cluster/status pvesh get /cluster/resources --type vm pve8to9 --full ``` Review: - No node is degraded. - No storage used by guests is unavailable. - No unexpected Bookworm, test, stale Ceph, or duplicate repository entries are present. - No guest depends on local-only disks unless intentionally handled. - `pve8to9 --full` warnings are understood before continuing. ## Bring Nodes Current on 8.4 Do this before any Debian Trixie or Proxmox VE 9 repository changes. ```bash apt update apt dist-upgrade pveversion ``` If the kernel changes, reboot the node and verify it returns cleanly before updating the next node. ## Upgrade Procedure Template Follow the official Proxmox guide for exact repository content and any changes made after this document was written. For each node, after backups, recovery access, and `pve8to9 --full` are clean or explicitly accepted: ```bash tmux new -s pve9-upgrade pve8to9 --full apt update apt dist-upgrade pve8to9 --full reboot ``` For the actual 8 to 9 step, update Debian and Proxmox repositories from Bookworm to Trixie, use the Proxmox VE 9 repository format documented by Proxmox, then run: ```bash apt update apt policy apt dist-upgrade pve8to9 --full reboot ``` Do not accept any apt transaction that removes the `proxmox-ve` package unless the official guide and `pve8to9` output explicitly explain the situation. ## Known-Issue Checks Check these before the maintenance window: - cgroup v1 removal and any legacy container assumptions. - Network interface naming changes. - EFI, GRUB, LVM, and systemd-boot warnings. - LVM/LVM-thin autoactivation warnings from `pve8to9`. - Third-party repositories and storage plugins. - Ceph repository state. If Ceph is not actually used, remove stale Ceph repo entries before the upgrade; if it is used, follow the official Ceph Squid prerequisite first. - Kernel 6.14 compatibility with older hardware and any PCI/USB passthrough. - `systemd-sysctl` no longer reading `/etc/sysctl.conf`; move persistent sysctl settings to `/etc/sysctl.d/`. - Debian 13 `/tmp` behavior changes. ## Post-Upgrade Validation Validate after each node, not only at the end. ```bash pveversion -v uname -a pvesh get /cluster/status pvesm status pvesh get /cluster/resources --type vm apt update apt list --upgradable pve8to9 --full ``` Also verify: - Proxmox UI/API reachable. - Cluster shows both nodes online after each reboot. - Shared storage is mounted on both nodes. - Migrated guests start and report healthy. - Home Assistant, MQTT, DNS, camera/NVR, edge apps, backups, and monitoring are healthy according to their normal validation surfaces. - Home Assistant Proxmox telemetry and repair automations return to normal. ## Rollback and Recovery Notes - A failed major upgrade is not a simple package downgrade. Treat recovery as restore-from-backup or repair-in-place using the official troubleshooting guide. - Keep one node unchanged while upgrading the other. - Keep guest backups and config exports outside the node being upgraded. - Preserve `/etc/pve`, network config, storage config, and any non-default host config before the upgrade. - If apt reports that `proxmox-ve` would be removed, stop and fix repository configuration before proceeding. - If a node fails to boot, use host-independent recovery access and the official GRUB/recovery guidance.