Fix Duplicati verification timing and Proxmox cleanup

master
Carlo Costanzo 1 month ago
parent 8670d3892e
commit 8e9ea094bf

@ -31,6 +31,7 @@ This directory contains the `homeassistant-yaml-dry-verifier` skill and the CLI
- Detects duplicate entries within a single block (`INTRA`).
- Detects package-defined scripts called from multiple files (`CENTRAL_SCRIPT`).
- Collapses noisy ENTRY reports when they are already fully explained by an identical `FULL_BLOCK` finding.
- Adds workflow guardrails for automation refactors that rename/remove entity references or introduce cleanup behavior: stale-reference checks, dry-run/preview expectations, explicit confirmation, and audit/backup output.
## CLI Usage
@ -70,6 +71,8 @@ Exit codes:
- This verifier intentionally keeps text output and a small CLI surface.
- It does not implement suppression files, severity scoring, JSON output, or diff-only mode.
- It is not an orphan entity cleaner and should not delete Home Assistant registry entries during normal DRY runs.
- Treat generic `unavailable`, disabled, or no-`config_entry_id` entities as audit signals only; YAML, helper, REST, command-line, MQTT, finance, YouTube, and local infrastructure telemetry can be intentional.
- Use it as a fast pre-refactor signal and pair with Home Assistant config validation before restart/reload.
**All of my configuration files are tested against the most stable version of home-assistant.**

@ -1,6 +1,6 @@
---
name: homeassistant-yaml-dry-verifier
description: "Verify Home Assistant YAML for DRY and efficiency issues by detecting redundant trigger/condition/action/sequence structures and repeated blocks across automations, scripts, and packages. Use when creating, reviewing, or refactoring YAML in config/packages, config/automations, config/scripts, or dashboard-related YAML where duplication risk is high."
description: "Verify Home Assistant YAML for DRY and efficiency issues by detecting redundant trigger/condition/action/sequence structures and repeated blocks across automations, scripts, and packages. Use when creating, reviewing, or refactoring YAML in config/packages, config/automations, config/scripts, or dashboard-related YAML where duplication risk is high. Include a read-only entity/reference safety pass when automation changes rename/remove entities or introduce maintenance cleanup behavior."
---
# Home Assistant YAML DRY Verifier
@ -13,13 +13,15 @@ Use this skill to lint Home Assistant YAML for repeat logic before or after edit
- Resolve the findings in the same task by refactoring YAML to remove duplication.
- Re-run the verifier after refactoring and iterate until targeted findings are cleared.
- If a finding cannot be safely resolved, explicitly document the blocker and the smallest safe follow-up.
- If touched YAML performs cleanup, registry hygiene, purge, deletion, or other destructive maintenance, require preview/dry-run behavior, explicit confirmation, and audit/backup output before any destructive action is considered complete.
## Quick Start
1. Run the verifier script on the file(s) you edited.
2. Review repeated block findings first (highest confidence).
3. Refactor into shared scripts/helpers/templates where appropriate.
4. Re-run the verifier and then run your normal Home Assistant config check.
4. If the change renames/removes entity references or adds maintenance cleanup behavior, perform the read-only entity/reference safety pass.
5. Re-run the verifier and then run your normal Home Assistant config check.
```bash
python codex_skills/homeassistant-yaml-dry-verifier/scripts/verify_ha_yaml_dry.py config/packages/life360.yaml --strict
@ -53,15 +55,25 @@ python codex_skills/homeassistant-yaml-dry-verifier/scripts/verify_ha_yaml_dry.p
- Repeated triggers: consolidate where behavior is equivalent, or split by intent if readability improves.
- For cooldown/throttle behavior, prefer automation-local `this.attributes.last_triggered` with custom event handoff before adding new helper entities, unless shared persistent state is required across automations.
5. Validate after edits:
5. Entity/Registry Safety Pass:
- Keep this pass read-only during normal DRY work. Report possible stale references or risky cleanup behavior; do not delete Home Assistant registry entries as part of this skill.
- When refactors rename, remove, or consolidate automations, scripts, helpers, entities, service calls, or dashboard targets, search touched and adjacent YAML for stale references before closing the task.
- Use live Home Assistant context or registry/state evidence when available and in scope, especially before changing entity IDs or automations that depend on device/integration state.
- Do not treat generic `unavailable`, disabled, or no-`config_entry_id` entities as safe deletion candidates. In YAML-heavy setups these are often intentional.
- Treat these platforms as common false-positive sources unless stronger evidence proves otherwise: `automation`, `script`, `scene`, `template`, helpers (`input_*`, `group`, `timer`, `counter`, `schedule`, `zone`, `person`, `tag`), `command_line`, `rest`, `mqtt`, `yahoofinance`, `youtube`, and local infrastructure telemetry.
- For cleanup or maintenance automations, prefer a preview/report action first, persistent ignore rules where repeated noise is expected, and an audit artifact that records what would change or did change.
- Destructive cleanup must be gated by explicit operator confirmation and should have backup/audit output. A dry-run-only recommendation is acceptable when the evidence is not strong enough.
6. Validate after edits:
- Re-run this verifier.
- Run Home Assistant config validation before reload/restart.
6. Enforce closure:
7. Enforce closure:
- Treat unresolved `FULL_BLOCK`/`ENTRY` findings in touched files as incomplete work unless a blocker is documented.
- Prefer consolidating duplicated automation triggers/conditions/actions into shared logic or a single branching automation.
- Treat unresolved `CENTRAL_SCRIPT` findings in touched scope as incomplete unless documented as deferred-with-blocker.
- Move shared package scripts to `config/script/<script_id>.yaml` when they are used cross-file.
- Treat unresolved stale-reference or cleanup-safety concerns in touched scope as incomplete unless documented as `deferred-with-blocker`.
## Dashboard Designer Integration
@ -77,6 +89,8 @@ Always report:
- Script caller detection should include direct `service: script.<id>` and `script.turn_on`-style entity targeting when present.
- Concrete refactor recommendation per group.
- Resolution status for each finding (`resolved`, `deferred-with-blocker`).
- Entity/reference hygiene status when this pass is in scope (`checked`, `not-in-scope`, or `deferred-with-blocker`).
- For cleanup or destructive maintenance YAML, preview/dry-run, confirmation, and audit/backup status.
Strict behavior:
- `--strict` returns non-zero for any reported finding (`FULL_BLOCK`, `ENTRY`, `INTRA`, `CENTRAL_SCRIPT`).

@ -120,10 +120,6 @@
name: HASS CPU
- entity: sensor.qemu_carlo_hass_105_memory_used_percentage
name: HASS MEM
- entity: sensor.qemu_wireguard_104_cpu_used
name: WireGuard CPU
- entity: sensor.qemu_wireguard_104_memory_used_percentage
name: WireGuard MEM
- type: custom:vertical-stack-in-card
card_mod:

@ -53,35 +53,28 @@
- entity: person.carlo
state_not: home
row:
type: attribute
entity: sensor.carlo_place
entity: sensor.carlo_location_display
name: Carlo Location
- type: conditional
conditions:
- entity: person.stacey
state_not: home
row:
type: attribute
entity: sensor.stacey_place
attribute: place_name
entity: sensor.stacey_location_display
name: Stacey Location
- type: conditional
conditions:
- entity: person.justin
state_not: home
row:
type: attribute
entity: sensor.justin_place
attribute: place_name
entity: sensor.justin_location_display
name: Justin Location
- type: conditional
conditions:
- entity: person.paige
state_not: home
row:
type: attribute
entity: sensor.paige_place
attribute: place_name
entity: sensor.paige_location_display
name: Paige Location
- show_state: true
show_name: true

@ -66,36 +66,28 @@
- entity: person.carlo
state_not: home
row:
type: attribute
entity: sensor.carlo_place
attribute: formatted_place
entity: sensor.carlo_location_display
name: Carlo Location
- type: conditional
conditions:
- entity: person.stacey
state_not: home
row:
type: attribute
entity: sensor.stacey_place
attribute: formatted_place
entity: sensor.stacey_location_display
name: Stacey Location
- type: conditional
conditions:
- entity: person.justin
state_not: home
row:
type: attribute
entity: sensor.justin_place
attribute: formatted_place
entity: sensor.justin_location_display
name: Justin Location
- type: conditional
conditions:
- entity: person.paige
state_not: home
row:
type: attribute
entity: sensor.paige_place
attribute: formatted_place
entity: sensor.paige_location_display
name: Paige Location
state_color: false
- type: custom:mushroom-template-card

@ -7,9 +7,9 @@
# Related Issue: 1632, 1584
# APT results and container down repairs.
# -------------------------------------------------------------------
# Notes: Hosts run weekly Wed 12:00 APT job and POST JSON to webhooks.
# Notes: Hosts run daily read-only APT pending checks plus Mon/Thu 12:00 APT jobs.
# Notes: Reboots are handled directly on each host by apt_weekly.sh.
# Notes: Reboot staggering: docker_14 first, docker_69 second, docker_10 third.
# Notes: Reboot staggering: docker_14, docker_69, docker_17, docker_10.
# Notes: Container monitoring is dynamic with binary_sensor status preferred over switch state.
# Notes: Weekly Joanna reconcile checks discovered container switches vs configured group members.
# Notes: Includes Portainer stack status repairs, 20-minute Joanna dispatch for persistent container outages, and scheduled image prune.
@ -899,6 +899,7 @@ automation:
updated: "{{ payload.get('updated', false) | bool }}"
reboot_required: "{{ payload.get('reboot_required', false) | bool }}"
packages: "{{ payload.get('packages', 0) | int(0) }}"
security_packages: "{{ payload.get('security_packages', 0) | int(0) }}"
message: "{{ payload.get('message', '') | string }}"
helpers:
docker_10:
@ -919,17 +920,20 @@ automation:
last_result: input_text.apt_docker_69_last_result
host_helpers: "{{ helpers[host_id] if host_id in helpers else none }}"
result: >-
{% set security = ' (' ~ security_packages ~ ' SEC)' if security_packages > 0 else '' %}
{% if not success %}
ERROR{% if (message | trim) != '' %}: {{ message | trim }}{% endif %}
{% elif updated %}
UPDATED {{ packages }} PKGS{% if reboot_required %} (REBOOT REQ){% endif %}
UPDATED {{ packages }} PKGS{{ security }}{% if reboot_required %} (REBOOT REQ){% endif %}
{% elif packages > 0 %}
PENDING {{ packages }} PKGS{{ security }}{% if reboot_required %} (REBOOT REQ){% endif %}
{% elif reboot_required %}
NO UPDATES (REBOOT REQ)
{% else %}
NO UPDATES
{% endif %}
log_message: >-
{{ host_id }} updated {{ packages }} package{% if packages != 1 %}s{% endif %}{% if reboot_required %}; reboot required{% endif %}.
{{ host_id }} updated {{ packages }} package{% if packages != 1 %}s{% endif %}{% if security_packages > 0 %} ({{ security_packages }} security){% endif %}{% if reboot_required %}; reboot required{% endif %}.
condition:
- condition: template
value_template: "{{ host_helpers is not none }}"

@ -9,7 +9,7 @@
# Related Issue: 1584
# Notes: Home dashboard consumes `infra_*` entities for exceptions-only alerts.
# Notes: Domain warning threshold is <30 days; critical threshold is <14 days.
# Notes: Nightly Duplicati verification is performed by codex_appliance after the Duplicati retry window because HA backup entities are not available.
# Notes: Nightly Duplicati verification runs at 08:00 after the 05:30 Duplicati job and docker_14 reboot window.
# Notes: Monthly HA log hygiene review requests Telegram + GitHub issue follow-up only; Joanna must wait for approval before any changes.
# Notes: Numeric WAN telemetry exposes state_class so recorder can keep long-term statistics.
# Notes: Docker host root disk usage uses Glances-backed normalized sensors; raw Glances sensors are recorder/logbook-filtered.
@ -578,7 +578,7 @@ automation:
mode: single
trigger:
- platform: time
at: "06:45:00"
at: "08:00:00"
action:
- variables:
trigger_context: "HA automation infra_backup_nightly_verification (Infrastructure - Backup Nightly Verification)"
@ -612,6 +612,10 @@ automation:
continue_on_error: true
data:
issue_id: infra_duplicati_backup_failure
- service: repairs.remove
continue_on_error: true
data:
issue_id: user_infra_duplicati_backup_failure
default:
- service: repairs.create
data:
@ -633,7 +637,7 @@ automation:
entity_ids:
- "switch.duplicati_container"
diagnostics: >-
scheduled_time=06:45:00,
scheduled_time=08:00:00,
duplicati_container={{ duplicati_state }},
verifier_http_status={{ verify_http_status }},
verifier_status={{ verify_status }},

@ -12,6 +12,7 @@
# Notes: Read more https://www.vcloudinfo.com/2018/01/going-green-to-save-some-green-in-2018.html | Existing Issue #272
# Tesla Powerwall added via UI Integration
# Notes: Live outage tracking derives its chronometer anchor from binary_sensor.powerwall_grid_status.last_changed.
# Notes: Camera PoE restore requires grid online and Powerwall charge at least 50%.
######################################################################
# Binary Sensors:
# - binary_sensor.powerwall_charging ............. battery_charging (on=charging)
@ -375,7 +376,7 @@ automation:
- alias: "Restore PoE ports when grid returns"
id: 1ae8b5c5-8627-4a44-8c8a-5bf8ca5e1bf5
description: "Turn camera PoE ports back on after grid has been stable"
description: "Turn camera PoE ports back on after grid has returned and battery is at least 50%"
mode: single
trigger:
- platform: state
@ -386,7 +387,7 @@ automation:
minutes: 60
- platform: numeric_state
entity_id: sensor.powerwall_charge
above: 90
above: 49.9
condition:
- condition: or
@ -403,9 +404,12 @@ automation:
- condition: state
entity_id: switch.poe_garage_port_6_poe
state: 'off'
- condition: state
entity_id: binary_sensor.powerwall_grid_status
state: 'on'
- condition: numeric_state
entity_id: sensor.powerwall_charge
above: 90
above: 49.9
action:
- service: switch.turn_on
target:
@ -417,7 +421,7 @@ automation:
- service: script.notify_engine
data:
title: "Grid restored - PoE ports re-enabled"
value1: "Power is stable. Camera PoE ports 3-6 were turned back on automatically."
value1: "Grid is online and Powerwall charge is at least 50%. Camera PoE ports 3-6 were turned back on automatically."
who: 'family'
group: 'information'

@ -49,7 +49,6 @@ exclude:
- sensor.*_since
- sensor.*uptime*
- sensor.sun_next_*
- sensor.vpn_client_*
- sensor.*_linkquality
- sensor.*_link_quality
- sensor.*_lqi

@ -27,7 +27,8 @@ Longer-running shell helpers referenced by automations, packages, or cron. Anyth
| File | Why it matters |
| --- | --- |
| [HAUpdate.sh](HAUpdate.sh) | One-command Home Assistant update helper. |
| [apt_weekly.sh](apt_weekly.sh) | Weekly APT updater that posts webhook status and can schedule reboot when needed. |
| [apt_pending_check.sh](apt_pending_check.sh) | Read-only APT pending-count reporter for the Docker host maintenance dashboard. |
| [apt_weekly.sh](apt_weekly.sh) | Twice-weekly APT updater that posts webhook status and can schedule reboot when needed. |
| [apt_reboot_report.sh](apt_reboot_report.sh) | Boot-time webhook status reporter that clears/keeps reboot-required state in HA. |
| [gitupdate.sh](gitupdate.sh) | Pull the latest config changes on demand. |
| [basketball.yaml](basketball.yaml) | ESPN stat scraping helper used by sensors. |

@ -0,0 +1,8 @@
[Unit]
Description=Daily APT pending check (Home Assistant webhook)
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/apt_pending_check.sh "http://YOUR_HOME_ASSISTANT:8123/api/webhook/YOUR_APT_WEBHOOK" "docker_10"

@ -0,0 +1,62 @@
#!/usr/bin/env bash
set -euo pipefail
# Read-only APT pending check for Docker hosts.
# Posts current package counts to the same Home Assistant webhook as apt_weekly.sh.
WEBHOOK_URL="$1"
HOST_NAME="${2:-$(hostname -s)}"
if [[ -z "$WEBHOOK_URL" ]]; then
echo "Usage: $0 <webhook_url> [host_name]" >&2
exit 1
fi
post_result() {
local success="$1"
local packages="$2"
local security_packages="$3"
local reboot_required="$4"
local message="${5:-}"
payload=$(cat <<JSON
{
"event": "pending_check",
"host": "${HOST_NAME}",
"success": ${success},
"updated": false,
"packages": ${packages},
"security_packages": ${security_packages},
"reboot_required": ${reboot_required},
"message": "${message}"
}
JSON
)
curl -sS -X POST -H 'Content-Type: application/json' -d "$payload" "$WEBHOOK_URL" || true
}
log() { echo "[$(date --iso-8601=seconds)] $*"; }
APT_OPTS=(-o Acquire::ForceIPv4=true)
REBOOT=false
if [[ -f /var/run/reboot-required ]]; then
REBOOT=true
fi
log "Refreshing package lists"
if ! apt-get "${APT_OPTS[@]}" update -qq; then
post_result false 0 0 "$REBOOT" "apt-get update failed"
exit 0
fi
UPGRADABLE="$(apt list --upgradable 2>/dev/null | tail -n +2 || true)"
PACKAGES=0
SECURITY_PACKAGES=0
if [[ -n "$UPGRADABLE" ]]; then
PACKAGES="$(printf '%s\n' "$UPGRADABLE" | sed '/^[[:space:]]*$/d' | wc -l)"
SECURITY_PACKAGES="$(printf '%s\n' "$UPGRADABLE" | grep -Ec '(^|,|-)(security|esm-apps-security|esm-infra-security)(,|/|[[:space:]]|$)' || true)"
fi
log "Posting pending package count to Home Assistant"
post_result true "$PACKAGES" "$SECURITY_PACKAGES" "$REBOOT" ""

@ -0,0 +1,11 @@
[Unit]
Description=Run APT pending check daily
[Timer]
OnCalendar=*-*-* 07:30:00
Persistent=true
RandomizedDelaySec=10m
Unit=apt_pending_check.service
[Install]
WantedBy=timers.target

@ -1,7 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
# Weekly APT maintenance for docker hosts (runs Wednesdays at 12:00 local via systemd timer)
# Twice-weekly APT maintenance for Docker hosts (Mon/Thu 12:00 via systemd timer).
# Posts results to Home Assistant webhook and optionally schedules reboot when required.
WEBHOOK_URL="$1"
@ -25,21 +25,28 @@ fi
log() { echo "[$(date --iso-8601=seconds)] $*"; }
APT_OPTS=(-o Acquire::ForceIPv4=true)
UPDATED=false
REBOOT=false
MESSAGE=""
log "Updating package lists"
if ! apt-get update -qq; then
if ! apt-get "${APT_OPTS[@]}" update -qq; then
MESSAGE="apt-get update failed"
curl -sS -X POST -H 'Content-Type: application/json' -d "{\"success\":false,\"updated\":false,\"packages\":0,\"reboot_required\":false,\"message\":\"$MESSAGE\"}" "$WEBHOOK_URL"
exit 0
fi
PACKAGES=$(apt list --upgradable 2>/dev/null | tail -n +2 | wc -l)
UPGRADABLE="$(apt list --upgradable 2>/dev/null | tail -n +2 || true)"
PACKAGES=0
SECURITY_PACKAGES=0
if [[ -n "$UPGRADABLE" ]]; then
PACKAGES="$(printf '%s\n' "$UPGRADABLE" | sed '/^[[:space:]]*$/d' | wc -l)"
SECURITY_PACKAGES="$(printf '%s\n' "$UPGRADABLE" | grep -Ec '(^|,|-)(security|esm-apps-security|esm-infra-security)(,|/|[[:space:]]|$)' || true)"
fi
if [[ "$PACKAGES" -gt 0 ]]; then
log "Applying upgrades ($PACKAGES pending)"
if apt-get -y upgrade --with-new-pkgs; then
if apt-get "${APT_OPTS[@]}" -y upgrade --with-new-pkgs; then
UPDATED=true
else
MESSAGE="apt-get upgrade failed"
@ -61,6 +68,7 @@ payload=$(cat <<JSON
"success": $( [[ "$MESSAGE" == "" ]] && echo true || echo false ),
"updated": $( $UPDATED && echo true || echo false ),
"packages": $PACKAGES,
"security_packages": $SECURITY_PACKAGES,
"reboot_required": $( $REBOOT && echo true || echo false ),
"auto_reboot_scheduled": $( [[ "$REBOOT" == true && "$AUTO_REBOOT" == true && "$MESSAGE" == "" ]] && echo true || echo false ),
"reboot_delay_minutes": ${REBOOT_DELAY_MINUTES:-0},
@ -76,9 +84,9 @@ if [[ "$REBOOT" == true && "$AUTO_REBOOT" == true && "$MESSAGE" == "" ]]; then
shutdown -c >/dev/null 2>&1 || true
if [[ "$REBOOT_DELAY_MINUTES" -eq 0 ]]; then
log "Reboot required; rebooting immediately."
shutdown -r now "APT weekly maintenance reboot"
shutdown -r now "APT maintenance reboot"
else
log "Reboot required; scheduling reboot in ${REBOOT_DELAY_MINUTES} minute(s)."
shutdown -r +"$REBOOT_DELAY_MINUTES" "APT weekly maintenance reboot"
shutdown -r +"$REBOOT_DELAY_MINUTES" "APT maintenance reboot"
fi
fi

@ -0,0 +1,10 @@
[Unit]
Description=Run APT maintenance twice weekly
[Timer]
OnCalendar=Mon,Thu *-*-* 12:00:00
Persistent=true
Unit=apt_weekly.service
[Install]
WantedBy=timers.target
Loading…
Cancel
Save

Powered by TurnKey Linux.