From 825b26c8006ab55e6557d5fcc35de6a930f6a7cd Mon Sep 17 00:00:00 2001 From: Carlo Costanzo Date: Wed, 7 Oct 2026 10:49:44 -0400 Subject: [PATCH] Replace LLM Vision package checks with local Frigate classification --- README.md | 2 +- config/README.md | 2 +- config/dashboards/README.md | 2 +- config/packages/README.md | 4 +- config/packages/frigate_classification.yaml | 185 ++++++++++++ config/packages/llmvision.yaml | 302 -------------------- docs/FRIGATE_STATE_CLASSIFICATION.md | 96 +++++++ 7 files changed, 286 insertions(+), 307 deletions(-) create mode 100644 config/packages/frigate_classification.yaml delete mode 100644 config/packages/llmvision.yaml create mode 100644 docs/FRIGATE_STATE_CLASSIFICATION.md diff --git a/README.md b/README.md index 6897a8c4..9af83e32 100755 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ Bring Cloudflare certificate, tunnel, and origin alerts into Home Assistant thro - Garage routines and entry lighting: [![YAML source: garadget](https://img.shields.io/static/v1?label=YAML&message=garadget&color=lightgrey&logo=github&logoColor=181717)](config/packages/garadget.yaml), [![YAML source: garage_entry_light](https://img.shields.io/static/v1?label=YAML&message=garage_entry_light&color=lightgrey&logo=github&logoColor=181717)](config/automation/garage_entry_light.yaml), [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/wFO0H5zgnzk) - Tesla tire-pressure alerts with shared briefing snoozes: [![YAML source: tesla_model_y](https://img.shields.io/static/v1?label=YAML&message=tesla_model_y&color=lightgrey&logo=github&logoColor=181717)](config/packages/tesla_model_y.yaml), [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/hDNyZfTfTtE), [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/09/home-assistant-tesla-tire-pressure-alerts.html) - Frigate driveway parked-car alert: [![YAML source: driveway_parked_car](https://img.shields.io/static/v1?label=YAML&message=driveway_parked_car&color=lightgrey&logo=github&logoColor=181717)](config/automation/driveway_parked_car.yaml), [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/X9qrywP3XS0), [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/09/home-assistant-frigate-driveway-car-alert.html) · Icon styling: [![Notification icon tip](https://img.shields.io/badge/Icons-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![Notification icon guide](https://img.shields.io/static/v1?label=Icons&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) -- LLM Vision package reminders: [![YAML source: llmvision](https://img.shields.io/static/v1?label=YAML&message=llmvision&color=lightgrey&logo=github&logoColor=181717)](config/packages/llmvision.yaml), [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/nAhCezFetvI) · Icon styling: [![Notification icon tip](https://img.shields.io/badge/Icons-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![Notification icon guide](https://img.shields.io/static/v1?label=Icons&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) +- Local Frigate package reminders ([setup and validation notes](docs/FRIGATE_STATE_CLASSIFICATION.md)): [![YAML source: frigate_classification](https://img.shields.io/static/v1?label=YAML&message=frigate_classification&color=lightgrey&logo=github&logoColor=181717)](config/packages/frigate_classification.yaml). Original LLM Vision experiment: [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/nAhCezFetvI) · Icon styling: [![Notification icon tip](https://img.shields.io/badge/Icons-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![Notification icon guide](https://img.shields.io/static/v1?label=Icons&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) - Docker update reporting and exception review: [![YAML source: tugtainer_updates](https://img.shields.io/static/v1?label=YAML&message=tugtainer_updates&color=lightgrey&logo=github&logoColor=181717)](config/packages/tugtainer_updates.yaml), [config/dashboards/infrastructure](config/dashboards/infrastructure), [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/4NNOkXzUyYw), [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/07/home-assistant-tugtainer-docker-updates.html) - Entity registry and Docker telemetry cleanup: [![YAML source: docker_infrastructure](https://img.shields.io/static/v1?label=YAML&message=docker_infrastructure&color=lightgrey&logo=github&logoColor=181717)](config/packages/docker_infrastructure.yaml), [![YAML source: infrastructure](https://img.shields.io/static/v1?label=YAML&message=infrastructure&color=lightgrey&logo=github&logoColor=181717)](config/packages/infrastructure.yaml), [recorder](config/recorder.yaml), [logbook](config/logbook.yaml), [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/aceV7ObKJVA), [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/07/using-codex-clean-up-home-assistant-entities.html) - TeslaMate road-trip reports and milestone alerts: [![YAML source: bearclaw](https://img.shields.io/static/v1?label=YAML&message=bearclaw&color=lightgrey&logo=github&logoColor=181717)](config/packages/bearclaw.yaml), [issue #1845](https://github.com/CCOSTAN/Home-AssistantConfig/issues/1845), [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/9-9T6v17NEw), [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/07/teslamate-road-trip-reports-codex.html) diff --git a/config/README.md b/config/README.md index e2cb2b77..d17c25d7 100755 --- a/config/README.md +++ b/config/README.md @@ -48,7 +48,7 @@ Live view of the `config/` directory my production Home Assistant instance loads ### Featured automations by section | Section | Why start here | Featured automation(s) | | --- | --- | --- | -| Packages | Self-contained subsystems that highlight patterns like combined alerts + actions. Cloudflare alerts: [![Watch Cloudflare walkthrough](https://img.shields.io/badge/Watch-Cloudflare%20Webhooks-FF0000?logo=youtube&logoColor=white)](https://youtu.be/-v1JkKwDrI8) [![Cloudflare blog guide](https://img.shields.io/static/v1?label=vCloudInfo&message=Cloudflare%20Guide&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/cloudflare-free-webhooks-home-assistant-website-alerts.html) | [![YAML source: alarm](https://img.shields.io/static/v1?label=YAML&message=alarm&color=lightgrey&logo=github&logoColor=181717)](packages/alarm.yaml), [![YAML source: garadget](https://img.shields.io/static/v1?label=YAML&message=garadget&color=lightgrey&logo=github&logoColor=181717)](packages/garadget.yaml), [![YAML source: infrastructure](https://img.shields.io/static/v1?label=YAML&message=infrastructure&color=lightgrey&logo=github&logoColor=181717)](packages/infrastructure.yaml), [![YAML source: speedtest](https://img.shields.io/static/v1?label=YAML&message=speedtest&color=lightgrey&logo=github&logoColor=181717)](packages/speedtest.yaml), [![YAML source: bgw_gateway](https://img.shields.io/static/v1?label=YAML&message=bgw_gateway&color=lightgrey&logo=github&logoColor=181717)](packages/bgw_gateway.yaml), [![YAML source: llmvision](https://img.shields.io/static/v1?label=YAML&message=llmvision&color=lightgrey&logo=github&logoColor=181717)](packages/llmvision.yaml), [![YAML source: phynplus](https://img.shields.io/static/v1?label=YAML&message=phynplus&color=lightgrey&logo=github&logoColor=181717)](packages/phynplus.yaml), [![YAML source: maintenance_log](https://img.shields.io/static/v1?label=YAML&message=maintenance_log&color=lightgrey&logo=github&logoColor=181717)](packages/maintenance_log.yaml), [![YAML source: tugtainer_updates](https://img.shields.io/static/v1?label=YAML&message=tugtainer_updates&color=lightgrey&logo=github&logoColor=181717)](packages/tugtainer_updates.yaml), [![YAML source: rachio](https://img.shields.io/static/v1?label=YAML&message=rachio&color=lightgrey&logo=github&logoColor=181717)](packages/rachio.yaml), [![YAML source: kiosk_tablet](https://img.shields.io/static/v1?label=YAML&message=kiosk_tablet&color=lightgrey&logo=github&logoColor=181717)](packages/kiosk_tablet.yaml), [![YAML source: vacation_mode](https://img.shields.io/static/v1?label=YAML&message=vacation_mode&color=lightgrey&logo=github&logoColor=181717)](packages/vacation_mode.yaml), [![YAML source: vacuum](https://img.shields.io/static/v1?label=YAML&message=vacuum&color=lightgrey&logo=github&logoColor=181717)](packages/vacuum.yaml), [![YAML source: powerwall](https://img.shields.io/static/v1?label=YAML&message=powerwall&color=lightgrey&logo=github&logoColor=181717)](packages/powerwall.yaml), [![YAML source: tesla_model_y](https://img.shields.io/static/v1?label=YAML&message=tesla_model_y&color=lightgrey&logo=github&logoColor=181717)](packages/tesla_model_y.yaml) | +| Packages | Self-contained subsystems that highlight patterns like combined alerts + actions. Cloudflare alerts: [![Watch Cloudflare walkthrough](https://img.shields.io/badge/Watch-Cloudflare%20Webhooks-FF0000?logo=youtube&logoColor=white)](https://youtu.be/-v1JkKwDrI8) [![Cloudflare blog guide](https://img.shields.io/static/v1?label=vCloudInfo&message=Cloudflare%20Guide&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/cloudflare-free-webhooks-home-assistant-website-alerts.html) | [![YAML source: alarm](https://img.shields.io/static/v1?label=YAML&message=alarm&color=lightgrey&logo=github&logoColor=181717)](packages/alarm.yaml), [![YAML source: garadget](https://img.shields.io/static/v1?label=YAML&message=garadget&color=lightgrey&logo=github&logoColor=181717)](packages/garadget.yaml), [![YAML source: infrastructure](https://img.shields.io/static/v1?label=YAML&message=infrastructure&color=lightgrey&logo=github&logoColor=181717)](packages/infrastructure.yaml), [![YAML source: speedtest](https://img.shields.io/static/v1?label=YAML&message=speedtest&color=lightgrey&logo=github&logoColor=181717)](packages/speedtest.yaml), [![YAML source: bgw_gateway](https://img.shields.io/static/v1?label=YAML&message=bgw_gateway&color=lightgrey&logo=github&logoColor=181717)](packages/bgw_gateway.yaml), [![YAML source: frigate_classification](https://img.shields.io/static/v1?label=YAML&message=frigate_classification&color=lightgrey&logo=github&logoColor=181717)](packages/frigate_classification.yaml), [![YAML source: phynplus](https://img.shields.io/static/v1?label=YAML&message=phynplus&color=lightgrey&logo=github&logoColor=181717)](packages/phynplus.yaml), [![YAML source: maintenance_log](https://img.shields.io/static/v1?label=YAML&message=maintenance_log&color=lightgrey&logo=github&logoColor=181717)](packages/maintenance_log.yaml), [![YAML source: tugtainer_updates](https://img.shields.io/static/v1?label=YAML&message=tugtainer_updates&color=lightgrey&logo=github&logoColor=181717)](packages/tugtainer_updates.yaml), [![YAML source: rachio](https://img.shields.io/static/v1?label=YAML&message=rachio&color=lightgrey&logo=github&logoColor=181717)](packages/rachio.yaml), [![YAML source: kiosk_tablet](https://img.shields.io/static/v1?label=YAML&message=kiosk_tablet&color=lightgrey&logo=github&logoColor=181717)](packages/kiosk_tablet.yaml), [![YAML source: vacation_mode](https://img.shields.io/static/v1?label=YAML&message=vacation_mode&color=lightgrey&logo=github&logoColor=181717)](packages/vacation_mode.yaml), [![YAML source: vacuum](https://img.shields.io/static/v1?label=YAML&message=vacuum&color=lightgrey&logo=github&logoColor=181717)](packages/vacuum.yaml), [![YAML source: powerwall](https://img.shields.io/static/v1?label=YAML&message=powerwall&color=lightgrey&logo=github&logoColor=181717)](packages/powerwall.yaml), [![YAML source: tesla_model_y](https://img.shields.io/static/v1?label=YAML&message=tesla_model_y&color=lightgrey&logo=github&logoColor=181717)](packages/tesla_model_y.yaml) | | Automations | Real-world triggers that tie Zwave, MQTT, REST, and Frigate occupancy events into the rest of the house. | [![YAML source: driveway_parked_car](https://img.shields.io/static/v1?label=YAML&message=driveway_parked_car&color=lightgrey&logo=github&logoColor=181717)](automation/driveway_parked_car.yaml), [![YAML source: garage_entry_light](https://img.shields.io/static/v1?label=YAML&message=garage_entry_light&color=lightgrey&logo=github&logoColor=181717)](automation/garage_entry_light.yaml), [![YAML source: dark_rainy_day](https://img.shields.io/static/v1?label=YAML&message=dark_rainy_day&color=lightgrey&logo=github&logoColor=181717)](automation/dark_rainy_day.yaml), [![YAML source: dash_buttons](https://img.shields.io/static/v1?label=YAML&message=dash_buttons&color=lightgrey&logo=github&logoColor=181717)](automation/dash_buttons.yaml) | | Scripts | Reusable building blocks for lighting, notifications, safety responses, and Joanna/BearClaw remediation dispatch. Colored iOS sender icons: [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) | [![YAML source: joanna_dispatch](https://img.shields.io/static/v1?label=YAML&message=joanna_dispatch&color=lightgrey&logo=github&logoColor=181717)](script/joanna_dispatch.yaml), [![YAML source: notify_engine](https://img.shields.io/static/v1?label=YAML&message=notify_engine&color=lightgrey&logo=github&logoColor=181717)](script/notify_engine.yaml), [![YAML source: speech_engine](https://img.shields.io/static/v1?label=YAML&message=speech_engine&color=lightgrey&logo=github&logoColor=181717)](script/speech_engine.yaml) | | Scenes | Seasonal and ambiance presets that the scripts and automations call into. | [![YAML source: monthly_colors](https://img.shields.io/static/v1?label=YAML&message=monthly_colors&color=lightgrey&logo=github&logoColor=181717)](scene/monthly_colors.yaml), [![YAML source: living_room](https://img.shields.io/static/v1?label=YAML&message=living_room&color=lightgrey&logo=github&logoColor=181717)](scene/living_room.yaml) | diff --git a/config/dashboards/README.md b/config/dashboards/README.md index 5f9b47c7..0caef82c 100644 --- a/config/dashboards/README.md +++ b/config/dashboards/README.md @@ -84,7 +84,7 @@ This folder is referenced from `config/configuration.yaml` via: - `dashboard-kiosk-satellite-staging` is the hidden validation surface for the shared Kiosk Satellite camera view. - Systems' visible Kiosks tab sits immediately before Joanna and combines attention telemetry with the latest Bedroom Tablet and Kitchen Show screenshots; tapping either snapshot opens that display's live dashboard. - Mini App local/public health leads Joanna's top status row; total 24-hour dispatches sit in Latest Activity, with AI Diagnostics at the bottom. Runtime & Memory retains failure details. Sustained failures appear in Systems Active Issues and the shared issue count, with persistent Repairs and independent mobile alerts. - - The Systems Joanna page links to BearClaw Admin from Dispatch Trend and to a hidden AI Diagnostics subview that shows the latest OpenAI instructions and response together; LLM Vision production diagnostics and retired QMD health telemetry are not shown. + - The Systems Joanna page links to BearClaw Admin from Dispatch Trend and to a hidden AI Diagnostics subview that shows the latest OpenAI instructions and response together; package-classification diagnostics and retired QMD health telemetry are not shown. - Overview's second visible view is **Mobile**, a phone-first quick-control surface for garage doors, the front-door lock, thermostats, and compact household alerts. - Overview Home uses ordered production sections under `overview/sections/`; approved pilot edits now apply directly there. - Overview Home's **Windows** ribbon opens a dedicated GPIO entry-point subview with perimeter, MQTT, and individual contact status. diff --git a/config/packages/README.md b/config/packages/README.md index f9616b70..cf5674e7 100755 --- a/config/packages/README.md +++ b/config/packages/README.md @@ -56,7 +56,7 @@ Kitchen voice uses Kiosk Satellite's [native Assist satellite](https://kiosksate | [![YAML source: lightning](https://img.shields.io/static/v1?label=YAML&message=lightning&color=lightgrey&logo=github&logoColor=181717)](lightning.yaml) | Blitzortung lightning monitoring with guarded distance wording, snoozeable push actions, single-owner safety speech, and a 60-minute duplicate-trigger cooldown. | `sensor.blitzortung_lightning_counter`, `sensor.blitzortung_lightning_distance`, `input_boolean.snooze_lightning`, `script.speech_engine` | | [![YAML source: logbook_activity_feed](https://img.shields.io/static/v1?label=YAML&message=logbook_activity_feed&color=lightgrey&logo=github&logoColor=181717)](logbook_activity_feed.yaml) | Dummy `sensor.activity_feed` + helper to write clean Activity entries (Issue #1550). | `sensor.activity_feed`, `script.send_to_logbook` | | [![YAML source: mariadb_monitoring](https://img.shields.io/static/v1?label=YAML&message=mariadb_monitoring&color=lightgrey&logo=github&logoColor=181717)](mariadb_monitoring.yaml) | MariaDB health sensors and Lovelace dashboard snippet for recorder stats. | `sensor.mariadb_status`, `sensor.database_size` | -| [![YAML source: llmvision](https://img.shields.io/static/v1?label=YAML&message=llmvision&color=lightgrey&logo=github&logoColor=181717)](llmvision.yaml) | Vision-backed garage-can and front-door package checks with rate-limited, downscaled OpenAI calls for package detection. [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/nAhCezFetvI) Package sender icons: [![Notification icon tip](https://img.shields.io/badge/Icons-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![Notification icon guide](https://img.shields.io/static/v1?label=Icons&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) | `input_button.llmvision_*`, `binary_sensor.front_door_packages_present`, `llmvision.stream_analyzer` | +| [![YAML source: frigate_classification](https://img.shields.io/static/v1?label=YAML&message=frigate_classification&color=lightgrey&logo=github&logoColor=181717)](frigate_classification.yaml) | Local Frigate state classification of the white porch floor, with shared package reminders and stable helper IDs. Garage classification remains disabled work in progress. [Setup and validation notes](../../docs/FRIGATE_STATE_CLASSIFICATION.md). Original LLM Vision experiment: [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/nAhCezFetvI) Package sender icons: [![Notification icon tip](https://img.shields.io/badge/Icons-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![Notification icon guide](https://img.shields.io/static/v1?label=Icons&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) | `input_button.front_door_package_check`, `binary_sensor.front_door_packages_present`, Frigate classification sensors | | [![YAML source: docker_infrastructure](https://img.shields.io/static/v1?label=YAML&message=docker_infrastructure&color=lightgrey&logo=github&logoColor=181717)](docker_infrastructure.yaml) | Docker host patching telemetry with a staggered daily 06:00 check-then-update window, persisted maintenance deadlines, container/stack Repairs automation, retained Portainer health telemetry, retired Portainer image-update controls, 20-minute Joanna escalation for persistent container outages including stuck `restarting`/`created` states, and weekly scheduled prune actions across docker_10/13/14/17/69. [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/aceV7ObKJVA) [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/07/using-codex-clean-up-home-assistant-entities.html) | `sensor.docker_*_apt_status`, `input_datetime.apt_docker_*_action_due`, `binary_sensor.portainer_status_2`, `sensor.portainer_cpu_usage_total_2`, `sensor.portainer_memory_usage_percentage_2`, `binary_sensor.*_stack_status`, `sensor.docker_stacks_down_count`, `repairs.create`, `repairs.remove`, `script.joanna_dispatch` | | [![YAML source: proxmox](https://img.shields.io/static/v1?label=YAML&message=proxmox&color=lightgrey&logo=github&logoColor=181717)](proxmox.yaml) | Proxmox update detection with amber dashboard maintenance state, 02:15 Joanna patch orchestration, a self-releasing four-hour container-alert fail-safe lease, kernel-refresh handoff hints, sequential migration and capacity gates, plus runtime and disk pressure monitoring. Docker14 recovery startup is pre-authorized only after expected VM placement and memory headroom are verified. Update availability does not create Repairs. | `binary_sensor.node_proxmox*_updates_packages`, `sensor.node_proxmox*_total_updates`, `input_datetime.docker_container_alerts_snooze_until`, `script.joanna_dispatch`, `binary_sensor.proxmox*_runtime_healthy`, `sensor.proxmox*_disk_used_percentage` | | [![YAML source: synology_dsm](https://img.shields.io/static/v1?label=YAML&message=synology_dsm&color=lightgrey&logo=github&logoColor=181717)](synology_dsm.yaml) | Synology DSM integration health normalization for Carlo-NAS01 and Carlo-NVR, with outage-aware Joanna-first handling for lone post-outage volume warnings and Repairs escalation for persistent or non-outage problems. | `binary_sensor.carlo_*_synology_problem`, `sensor.carlo_*_synology_problem_summary`, `binary_sensor.powerwall_grid_status`, `repairs.create`, `script.joanna_dispatch` | @@ -121,7 +121,7 @@ When a package has a dedicated blog post or video, I link it right inside the YA | [![YAML source: garage_entry_light](https://img.shields.io/static/v1?label=YAML&message=garage_entry_light&color=lightgrey&logo=github&logoColor=181717)](../automation/garage_entry_light.yaml) | Simple door-open and door-closed hallway lighting pattern from the garage walkthrough. | [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/wFO0H5zgnzk) · [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/05/home-assistant-garage-automations-arrival-alerts-tesla.html) | | [![YAML source: kiosk_tablet](https://img.shields.io/static/v1?label=YAML&message=kiosk_tablet&color=lightgrey&logo=github&logoColor=181717)](kiosk_tablet.yaml) + [Bedroom dashboard](../dashboards/bedroom) + [Kiosk Satellite preview](../dashboards/kiosk_satellite_staging) | Shared event priority with purpose-built display surfaces: clock-free Bedroom night-watch context and cameras through Fully, plus Kitchen clock/forecast, reminders, native camera overlays, motion-aware dimming, and household-aware screen power through Kiosk Satellite. | [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/ChgEu0IDWzc) · [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2017/08/project-update-visualization-of-home.html) | | [![YAML source: tesla_model_y](https://img.shields.io/static/v1?label=YAML&message=tesla_model_y&color=lightgrey&logo=github&logoColor=181717)](tesla_model_y.yaml) | Tesla tire-pressure alerts and garage-driven plug-in reminders, plus charging-aware protection: fresh home draw above 18 A requests a verified stop; missing reports dispatch investigation at 3 minutes, protect an unverified home session at 5 minutes, and quietly warn about persistent monitoring at 15 minutes. Tagged incident cleanup and a restoring human restart-review latch prevent false recovery and unattended resumption. | [![Watch tire-pressure video](https://img.shields.io/badge/Watch-Tire%20Pressure-FF0000?logo=youtube&logoColor=white)](https://youtu.be/hDNyZfTfTtE) · [![Tire-pressure blog post](https://img.shields.io/static/v1?label=vCloudInfo&message=Tire%20Pressure&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/09/home-assistant-tesla-tire-pressure-alerts.html) · [![Watch garage video](https://img.shields.io/badge/Watch-Garage%20Automations-FF0000?logo=youtube&logoColor=white)](https://youtu.be/wFO0H5zgnzk) · [![Garage automations blog post](https://img.shields.io/static/v1?label=vCloudInfo&message=Garage%20Automations&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/05/home-assistant-garage-automations-arrival-alerts-tesla.html) | -| [![YAML source: llmvision](https://img.shields.io/static/v1?label=YAML&message=llmvision&color=lightgrey&logo=github&logoColor=181717)](llmvision.yaml) | LLM Vision package reminder flow that turns a front-door camera event into a stored package-present state and phone alert. Package sender icons: [![Notification icon tip](https://img.shields.io/badge/Icons-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![Notification icon guide](https://img.shields.io/static/v1?label=Icons&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) | [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/nAhCezFetvI) | +| [![YAML source: frigate_classification](https://img.shields.io/static/v1?label=YAML&message=frigate_classification&color=lightgrey&logo=github&logoColor=181717)](frigate_classification.yaml) | Original LLM Vision package reminder experiment. The current implementation uses [local Frigate state classification](../../docs/FRIGATE_STATE_CLASSIFICATION.md) while preserving the package-present state and shared phone alert. Package sender icons: [![Notification icon tip](https://img.shields.io/badge/Icons-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/TMfdxv-WprM) [![Notification icon guide](https://img.shields.io/static/v1?label=Icons&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html) | [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/nAhCezFetvI) | | [![YAML source: maintenance_log](https://img.shields.io/static/v1?label=YAML&message=maintenance_log&color=lightgrey&logo=github&logoColor=181717)](maintenance_log.yaml) + [![YAML source: 09_salt_trends](https://img.shields.io/static/v1?label=YAML&message=09_salt_trends&color=lightgrey&logo=github&logoColor=181717)](../dashboards/infrastructure/views/09_salt_trends.yaml) + [![YAML source: home_water_sections](https://img.shields.io/static/v1?label=YAML&message=home_water_sections&color=lightgrey&logo=github&logoColor=181717)](../dashboards/infrastructure/partials/home_water_sections.yaml) | Water softener salt maintenance flow with a reusable webhook, duplicate protection, helper-backed history, template sensors, and Salt Trends dashboard views. | [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/_OqTv9i-jAs) · [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/07/home-assistant-water-softener-salt-automation.html) · [Issue #1816](https://github.com/CCOSTAN/Home-AssistantConfig/issues/1816) | | [![YAML source: climate](https://img.shields.io/static/v1?label=YAML&message=climate&color=lightgrey&logo=github&logoColor=181717)](climate.yaml) + [![YAML source: home_water_sections](https://img.shields.io/static/v1?label=YAML&message=home_water_sections&color=lightgrey&logo=github&logoColor=181717)](../dashboards/infrastructure/partials/home_water_sections.yaml) | HVAC blower-failure evidence, runtime-based AC filter reminders, two-unit native NFC tag change capture, per-unit change counts/cadence, reset actions, and 360-day condenser-line cleanout reminders. | [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/nScl2JxYSB8) · [Issue #1817](https://github.com/CCOSTAN/Home-AssistantConfig/issues/1817) · [Issue #273](https://github.com/CCOSTAN/Home-AssistantConfig/issues/273) | | [![YAML source: climate](https://img.shields.io/static/v1?label=YAML&message=climate&color=lightgrey&logo=github&logoColor=181717)](climate.yaml) + [![YAML source: printer](https://img.shields.io/static/v1?label=YAML&message=printer&color=lightgrey&logo=github&logoColor=181717)](printer.yaml) + [![YAML source: vacation_mode](https://img.shields.io/static/v1?label=YAML&message=vacation_mode&color=lightgrey&logo=github&logoColor=181717)](vacation_mode.yaml) | Snoozeable Home Assistant mobile notifications for AC filters, printer ink, and Vacation Mode missed-visit reminders. | [![Watch on YouTube](https://img.shields.io/badge/Watch-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/y47KSflS1aw) · [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2026/06/home-assistant-notification-snooze-buttons.html) · [![vCloudInfo Blog Post](https://img.shields.io/static/v1?label=vCloudInfo&message=Blog%20Post&color=21759B&logo=wordpress&logoColor=white)](https://www.vcloudinfo.com/2020/11/home-assistant-ios-actionable-notifications-example.html) | diff --git a/config/packages/frigate_classification.yaml b/config/packages/frigate_classification.yaml new file mode 100644 index 00000000..ba57aafa --- /dev/null +++ b/config/packages/frigate_classification.yaml @@ -0,0 +1,185 @@ +###################################################################### +# @CCOSTAN - Follow Me on X +# For more info visit https://www.vcloudinfo.com/click-here +# Original Repo : https://github.com/CCOSTAN/Home-AssistantConfig +# ------------------------------------------------------------------- +# Frigate - local front-door package state classification +# Related Issue: 1984 +# Classify the white porch floor locally; keep package reminders shared. +# ------------------------------------------------------------------- +# Notes: Frigate 0.18 classifies clear/package_present with a trained +# custom model, motion checks, a 15-second interval, and 3-state consensus. +# Notes: Either crop can mark a package present; both must report clear +# before clearing it. The second crop covers partial doorstep-edge boxes. +# Notes: Existing package/cans binary-sensor identities stay compatible. +# LLM Vision is removed. Last sync records state updates, not every +# inference; the status button reads the last verified state. +# Notes: Unknown/unavailable classifications never clear a package. +# Notes: Garage is non-production and disabled. The reflective X is gone; +# any future classifier must use the actual bins in the upper-right. +# Docs: https://docs.frigate.video/configuration/custom_classification/state_classification/ +# Original LLM Vision experiment video: https://youtu.be/nAhCezFetvI +# Notification icons video: https://youtu.be/TMfdxv-WprM +# Notification icons blog: https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html +###################################################################### + +input_button: + front_door_package_check: + name: "Frigate: Front door package status" + icon: mdi:package-variant-closed-check + +input_boolean: + garbage_cans_out: + name: "Garbage Cans Out" + icon: mdi:trash-can + front_door_packages_present: + name: "Front Door Packages Present" + icon: mdi:package-variant-closed + +input_text: + front_door_package_classification: + name: "Frigate front door classification" + max: 255 + +input_datetime: + front_door_package_last_sync: + name: "Frigate front door last state sync" + has_date: true + has_time: true + +template: + - binary_sensor: + - name: "Garbage Cans Out" + unique_id: llmvision_garbage_cans_out + device_class: presence + state: "{{ is_state('input_boolean.garbage_cans_out', 'on') }}" + attributes: + classification_status: not_configured + source_camera: camera.garagecam + + - name: "Front Door Packages Present" + unique_id: llmvision_front_door_packages_present + availability: >- + {{ states('sensor.frontdoorbell_front_door_packages_classification') + in ['clear', 'package_present'] + and states('sensor.frontdoorbell_front_door_package_edge_classification') + in ['clear', 'package_present'] }} + device_class: presence + state: "{{ is_state('input_boolean.front_door_packages_present', 'on') }}" + attributes: + last_sync: "{{ states('input_datetime.front_door_package_last_sync') }}" + classification: "{{ states('input_text.front_door_package_classification') }}" + source_camera: camera.frontdoorbell + source_classifier: sensor.frontdoorbell_front_door_packages_classification + source_classifier_edge: sensor.frontdoorbell_front_door_package_edge_classification + classifier_state: "{{ states('sensor.frontdoorbell_front_door_packages_classification') }}" + classifier_edge_state: "{{ states('sensor.frontdoorbell_front_door_package_edge_classification') }}" + +automation: + - alias: Frigate - Front door packages + id: 8647b555-7ba0-41d5-98de-cfd3094e5b4f + description: "Sync Frigate's verified local porch state to the existing package helper." + mode: restart + trigger: + - platform: state + entity_id: + - sensor.frontdoorbell_front_door_packages_classification + - sensor.frontdoorbell_front_door_package_edge_classification + from: null + id: classification + - platform: homeassistant + event: start + id: startup + - platform: state + entity_id: input_button.front_door_package_check + id: manual_check + variables: + labels: >- + {{ [states('sensor.frontdoorbell_front_door_packages_classification'), + states('sensor.frontdoorbell_front_door_package_edge_classification')] }} + classification: "{{ 'package_present' if 'package_present' in labels else 'clear' }}" + condition: + - condition: state + entity_id: + - sensor.frontdoorbell_front_door_packages_classification + - sensor.frontdoorbell_front_door_package_edge_classification + state: + - clear + - package_present + action: + - if: + - condition: state + entity_id: + - sensor.frontdoorbell_front_door_packages_classification + - sensor.frontdoorbell_front_door_package_edge_classification + state: package_present + match: any + then: + - service: input_boolean.turn_on + target: + entity_id: input_boolean.front_door_packages_present + else: + - service: input_boolean.turn_off + target: + entity_id: input_boolean.front_door_packages_present + - service: input_text.set_value + target: + entity_id: input_text.front_door_package_classification + data: + value: "{{ classification }}" + - service: input_datetime.set_datetime + target: + entity_id: input_datetime.front_door_package_last_sync + data: + datetime: "{{ now() }}" + - service: script.send_to_logbook + data: + topic: PACKAGES + message: "Local Frigate package state synced: {{ classification }}." + - if: + - condition: trigger + id: manual_check + then: + - service: script.notify_engine + data: + title: Front door package status + notification_icon: mdi:package-variant-closed-check + color: '#6A1B9A' + value1: "Frigate's last verified porch state is {{ classification | replace('_', ' ') }}." + who: carlo + group: information + level: active + camera_entity: camera.frontdoorbell + + - alias: Frigate - Front door package notify + id: 0ddf0fb5-4600-4eff-9236-95b2af3739a6 + mode: restart + trigger: + - platform: state + entity_id: input_boolean.front_door_packages_present + to: 'on' + for: "00:01:00" + action: + - variables: + package_message: >- + Front door camera thinks a package may be waiting at the door. + notify_target: >- + {% if is_state('group.family', 'home') %} + family + {% else %} + carlo + {% endif %} + - service: script.notify_engine + data: + title: Package check + notification_icon: mdi:package-variant-closed-check + color: '#6A1B9A' + value1: "{{ package_message }}" + who: "{{ notify_target }}" + group: information + level: active + camera_entity: camera.frontdoorbell + - service: script.send_to_logbook + data: + topic: PACKAGES + message: "{{ package_message }}" diff --git a/config/packages/llmvision.yaml b/config/packages/llmvision.yaml deleted file mode 100644 index 993d1bd1..00000000 --- a/config/packages/llmvision.yaml +++ /dev/null @@ -1,302 +0,0 @@ -###################################################################### -# @CCOSTAN - Follow Me on X -# For more info visit https://www.vcloudinfo.com/click-here -# Original Repo : https://github.com/CCOSTAN/Home-AssistantConfig -# ------------------------------------------------------------------- -# LLM Vision - garage cans and front-door package checks -# Related Issue: 1773 -# Trigger with input_button.llmvision_garbage_check, input_button.llmvision_front_door_package_check, or front door person activity to update vision-backed package sensors. -# ------------------------------------------------------------------- -# Notes: LLMVision analyzes camera.garagecam; expects strict "on"/"off" output. -# Notes: Front-door package detection waits 3 minutes after person -# activity, skips routine relocks unless a package is already present, -# and rate-limits/downscales OpenAI vision calls for cost control. -# Notes: The garage analyzer uses gpt-5.4-nano and remains manual; -# the front-door analyzer uses gpt-4.1-mini for reliable package detection. -# Docs: https://llmvision.gitbook.io/getting-started/usage/image-analyzer -# Video: https://youtu.be/nAhCezFetvI -# Notification icons video: https://youtu.be/TMfdxv-WprM -# Notification icons blog: https://www.vcloudinfo.com/2026/10/home-assistant-ios-notification-icons.html -###################################################################### - -input_button: - llmvision_garbage_check: - name: "LLM Vision: Garage cans check" - icon: mdi:delete-variant - llmvision_front_door_package_check: - name: "LLM Vision: Front door package check" - icon: mdi:package-variant-closed-check - -input_boolean: - garbage_cans_out: - name: "Garbage Cans Out" - icon: mdi:trash-can - front_door_packages_present: - name: "Front Door Packages Present" - icon: mdi:package-variant-closed - -input_text: - llmvision_garbage_last_response: - name: "LLM Vision garage response" - max: 255 - llmvision_garbage_last_keyframe: - name: "LLM Vision garage key frame" - max: 255 - llmvision_front_door_last_response: - name: "LLM Vision front door response" - max: 255 - llmvision_front_door_last_keyframe: - name: "LLM Vision front door key frame" - max: 255 - -input_datetime: - llmvision_garbage_last_run: - name: "LLM Vision garage last run" - has_date: true - has_time: true - llmvision_front_door_last_run: - name: "LLM Vision front door last run" - has_date: true - has_time: true - -template: - - binary_sensor: - - name: "Garbage Cans Out" - unique_id: llmvision_garbage_cans_out - device_class: presence - state: "{{ is_state('input_boolean.garbage_cans_out', 'on') }}" - attributes: - last_run: "{{ states('input_datetime.llmvision_garbage_last_run') }}" - last_response: "{{ states('input_text.llmvision_garbage_last_response') }}" - last_key_frame: "{{ states('input_text.llmvision_garbage_last_keyframe') }}" - source_camera: camera.garagecam - - - name: "Front Door Packages Present" - unique_id: llmvision_front_door_packages_present - device_class: presence - state: "{{ is_state('input_boolean.front_door_packages_present', 'on') }}" - attributes: - last_run: "{{ states('input_datetime.llmvision_front_door_last_run') }}" - last_response: "{{ states('input_text.llmvision_front_door_last_response') }}" - last_key_frame: "{{ states('input_text.llmvision_front_door_last_keyframe') }}" - source_camera: camera.frontdoorbell - -automation: - - alias: LLM Vision - Garbage cans out - id: d88a2e6d-78f3-4bb7-9d9f-c4d06e6eb5a9 - mode: restart - trigger: - - platform: state - entity_id: input_button.llmvision_garbage_check - variables: - prompt_text: > - Examine the image. I have taped a reflective X on the right wall. Look for the reflective X on the right wall. If you clearly see the X, respond exactly: on. If you cannot see the X, respond exactly: off. No other words. - action: - - service: llmvision.data_analyzer - response_variable: llmvision_result - data: - provider: !secret llmvision_provider_entry - model: gpt-5.4-nano - message: "{{ prompt_text }}" - sensor_entity: input_boolean.garbage_cans_out - image_entity: - - camera.garagecam - include_filename: false - target_width: 1280 - max_tokens: 16 - expose_images: true - - service: input_text.set_value - target: - entity_id: input_text.llmvision_garbage_last_response - data: - value: "{{ (llmvision_result.response_text | default('unknown'))[:250] }}" - - choose: - - conditions: "{{ llmvision_result is defined and llmvision_result.key_frame is defined }}" - sequence: - - service: input_text.set_value - target: - entity_id: input_text.llmvision_garbage_last_keyframe - data: - value: "{{ llmvision_result.key_frame }}" - default: - - service: input_text.set_value - target: - entity_id: input_text.llmvision_garbage_last_keyframe - data: - value: "" - - service: input_datetime.set_datetime - target: - entity_id: input_datetime.llmvision_garbage_last_run - data: - datetime: "{{ now() }}" - - - alias: LLM Vision - Front door packages - id: 8647b555-7ba0-41d5-98de-cfd3094e5b4f - mode: restart - trigger: - - platform: numeric_state - entity_id: sensor.frontdoorbell_person_active_count_2 - above: 0 - id: person_count - - platform: state - entity_id: lock.front_door - from: unlocked - to: locked - id: front_door_locked - - platform: state - entity_id: input_button.llmvision_front_door_package_check - id: manual_check - variables: - trigger_source: >- - {% if trigger.id == 'manual_check' %} - manual button - {% elif trigger.id == 'front_door_locked' %} - front door relock - {% else %} - front door person activity - {% endif %} - prompt_text: >- - Examine these front door camera frames for delivery packages. Focus on the porch and doorstep area, - especially the lower-right and bottom-right edge near the door. Treat cardboard boxes, padded mailers, - poly bags, shopping bags, and white shipping bags as packages. A package can be partially cut off by - the edge of the image or timestamp overlay and should still count as visible if part of the bag, box, - label, or mailer is visible on the porch. Ignore the street, cars, landscaping, and anything not - resting on the porch or doorstep. Respond exactly: on or off. No other words. - action: - - choose: - - conditions: - - condition: template - value_template: "{{ trigger.id == 'person_count' }}" - sequence: - - delay: "00:03:00" - - conditions: - - condition: template - value_template: "{{ trigger.id == 'front_door_locked' }}" - sequence: - - delay: "00:00:20" - - condition: template - value_template: >- - {% set last = states('input_datetime.llmvision_front_door_last_run') %} - {% set minutes_since_last = 999 if last in ['unknown', 'unavailable', 'none', ''] else ((as_timestamp(now()) - as_timestamp(last)) / 60) | int(999) %} - {{ trigger.id == 'manual_check' - or (trigger.id == 'person_count' and minutes_since_last >= 15) - or (trigger.id == 'front_door_locked' - and is_state('input_boolean.front_door_packages_present', 'on') - and minutes_since_last >= 5) }} - - service: llmvision.stream_analyzer - response_variable: llmvision_result - data: - provider: !secret llmvision_provider_entry - model: gpt-4.1-mini - message: "{{ prompt_text }}" - image_entity: - - camera.frontdoorbell - duration: 4 - max_frames: 3 - include_filename: false - target_width: 1280 - max_tokens: 32 - expose_images: true - - variables: - normalized_response: "{{ llmvision_result.response_text | default('') | trim | lower }}" - - choose: - - conditions: - - condition: template - value_template: >- - {{ normalized_response in ['on', 'yes', 'true'] - or normalized_response.startswith('on') - or normalized_response.startswith('yes') }} - sequence: - - service: input_boolean.turn_on - target: - entity_id: input_boolean.front_door_packages_present - - conditions: - - condition: template - value_template: >- - {{ normalized_response in ['off', 'no', 'false'] - or normalized_response.startswith('off') - or normalized_response.startswith('no') }} - sequence: - - service: input_boolean.turn_off - target: - entity_id: input_boolean.front_door_packages_present - - service: input_text.set_value - target: - entity_id: input_text.llmvision_front_door_last_response - data: - value: "{{ (llmvision_result.response_text | default('unknown'))[:250] }}" - - choose: - - conditions: "{{ llmvision_result is defined and llmvision_result.key_frame is defined }}" - sequence: - - service: input_text.set_value - target: - entity_id: input_text.llmvision_front_door_last_keyframe - data: - value: "{{ llmvision_result.key_frame }}" - default: - - service: input_text.set_value - target: - entity_id: input_text.llmvision_front_door_last_keyframe - data: - value: "" - - service: input_datetime.set_datetime - target: - entity_id: input_datetime.llmvision_front_door_last_run - data: - datetime: "{{ now() }}" - - service: script.send_to_logbook - data: - topic: PACKAGES - message: >- - Front door package vision check ran via {{ trigger_source }} - and returned {{ llmvision_result.response_text | default(states('input_boolean.front_door_packages_present')) | lower }}. - - choose: - - conditions: - - condition: template - value_template: "{{ trigger.id == 'manual_check' }}" - sequence: - - service: script.notify_engine - data: - title: Front door package test - notification_icon: mdi:package-variant-closed-check - color: '#6A1B9A' - value1: >- - LLM Vision returned {{ llmvision_result.response_text | default(states('input_boolean.front_door_packages_present')) | lower }} - for the front door package check. - who: carlo - group: information - level: active - camera_entity: camera.frontdoorbell - - - alias: LLM Vision - Front door package notify - id: 0ddf0fb5-4600-4eff-9236-95b2af3739a6 - mode: restart - trigger: - - platform: state - entity_id: input_boolean.front_door_packages_present - to: 'on' - for: "00:01:00" - action: - - variables: - package_message: >- - Front door camera thinks a package may be waiting at the door. - notify_target: >- - {% if is_state('group.family', 'home') %} - family - {% else %} - carlo - {% endif %} - - service: script.notify_engine - data: - title: Package check - notification_icon: mdi:package-variant-closed-check - color: '#6A1B9A' - value1: "{{ package_message }}" - who: "{{ notify_target }}" - group: information - level: active - camera_entity: camera.frontdoorbell - - service: script.send_to_logbook - data: - topic: PACKAGES - message: "{{ package_message }}" diff --git a/docs/FRIGATE_STATE_CLASSIFICATION.md b/docs/FRIGATE_STATE_CLASSIFICATION.md new file mode 100644 index 00000000..0f54b957 --- /dev/null +++ b/docs/FRIGATE_STATE_CLASSIFICATION.md @@ -0,0 +1,96 @@ +# Local Frigate package state classification + +Tracking: [issue #1984](https://github.com/CCOSTAN/Home-AssistantConfig/issues/1984). + +The front-door package flow now uses Frigate 0.18 custom state classification. Two camera crops look at the white porch floor where deliveries are placed. Inference runs locally; this flow makes no LLM Vision or OpenAI calls. The existing Home Assistant package helpers, dashboard references, snapshot notifications, and shared notification engine remain in use. + +The source is [frigate_classification.yaml](../config/packages/frigate_classification.yaml). The package/cans binary sensors keep their internal registry identities so existing consumers remain compatible. The package status and diagnostic helpers use neutral names; obsolete LLM Vision helpers and the disabled garage automation are removed. + +## Frigate setup + +Train two separate state classifiers through Frigate's Classification UI. Use the same two labels, `clear` and `package_present`, for each model. These are camera-specific models, not a generic pretrained package detector. Adjust the crops to match your own camera and delivery area. + +```yaml +classification: + custom: + front_door_packages: + threshold: 0.9 + save_attempts: 200 + state_config: + motion: true + interval: 15 + cameras: + frontdoorbell: + crop: [0.10, 0.74, 1.0, 1.0] + front_door_package_edge: + threshold: 0.9 + save_attempts: 200 + state_config: + motion: true + interval: 15 + cameras: + frontdoorbell: + crop: [0.70, 0.74, 1.0, 1.0] +``` + +The broad crop covers the white floor. The narrower crop gives partially visible boxes near the bottom-right edge more image detail. The timestamp is outside both crops. + +Label real deliveries, cardboard boxes, shipping bags, partial packages, empty floor, shadows, people walking through, and people carrying packages without placing them down. Include day and night images and train after categorizing examples. Keep household images and generated models private. This implementation seeded each model with 46 examples: 15 package-present and 31 clear. + +Frigate checks on motion and every 15 seconds. Its state classifier requires three matching confident observations before publishing a state change, with follow-up verification checks while a change is pending. Low-confidence predictions do not publish a new state; they preserve the previous verified state. The 0.9 threshold is an acceptance setting, not a measured accuracy percentage. + +See the [official state-classification guide](https://docs.frigate.video/configuration/custom_classification/state_classification/). + +## Home Assistant behavior + +The Frigate integration supplies these entities: + +- `sensor.frontdoorbell_front_door_packages_classification` +- `sensor.frontdoorbell_front_door_package_edge_classification` + +Both must have valid labels before the adapter changes the existing helper. Either crop reporting `package_present` turns `input_boolean.front_door_packages_present` on. Both reporting `clear` turn it off. An unknown or unavailable source leaves the helper unchanged and makes `binary_sensor.front_door_packages_present` unavailable. + +The existing one-minute positive-state hold, family/Carlo routing, package icon, camera attachment, and Activity Feed logging are preserved. Continuous local classification replaces the former person/lock-triggered cloud capture, three-minute wait, and API rate limit. + +`input_button.front_door_package_check` reports the last verified classifier state; it does not force a new camera inference. `input_datetime.front_door_package_last_sync` records the latest successful state synchronization, not every inference. `input_text.front_door_package_classification` records the synchronized label. Obsolete key-frame helpers are removed; notifications still use the camera image. + +Create/reload the Frigate integration after adding classifiers. If a new classification sensor stays `Unknown`, ensure it receives a fresh classifier result before enabling the adapter. Frigate publishes classification changes without MQTT retention; the integration restores previously known states, but a newly discovered sensor can miss the initial result. Disabling and re-enabling that classifier through the supported config API after discovery lets it establish a fresh verified state. Do not publish invented classifier labels to initialize production. + +LLM Vision's provider/settings entries and HACS component are removed, along with its old helpers, update entity, provider reference, and timeline artifacts, including the orphaned old cans-in registry entry. A private backup was retained before cleanup. The garage remains non-production with no classifier or active automation: its reflective X marker is gone. A future garage classifier must train on the actual cans in the upper-right image area and confirm how present/absent maps to cans-out semantics. + +## Timestamp and motion mask + +The camera bridge exposes a timestamp on/off control, with no position control. The original camera timestamp is disabled, and the Frigate/go2rtc source applies a top-left overlay. The Wyze app therefore has no burned-in timestamp; Frigate and its Home Assistant streams include the replacement. + +For a go2rtc FFmpeg video source, append a drawtext filter using a font available in the container. This example uses a placeholder source stream: + +```text +ffmpeg:doorbell_raw#video=h264#drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf:fontsize=38:fontcolor=white:borderw=2:bordercolor=black:x=20:y=20:text=%{localtime} +``` + +This encodes the source video, so verify CPU use, live playback, recording, and any audio requirements for your camera. Keep existing compatible presentation streams downstream of the annotated source. Confirm the container's local time matches the desired timezone. + +The existing timestamp motion mask moves to the top-left area (`0,0,0.345,0,0.345,0.045,0,0.045` for this portrait image). Keep other masks unchanged and adapt this polygon to your own timestamp width and camera resolution. Removing the old lower-right overlay and mask exposes the doorstep edge. + +## Validation and remaining checks + +The Frigate configuration and Home Assistant configuration passed validation. After the cutover restart, both classifier states restored as clear and the startup automation completed successfully. No LLM Vision config entries, services, or installed HACS component remain. Both live classification sensors report `clear` on the empty porch; a completed automation trace confirms helper synchronization. The existing mobile dashboard displays Packages: CLEAR. Notification automation remains enabled and no garage classifier or automation is enabled. + +An isolated replay through Frigate's native classifier confirms clear -> package_present -> clear after three confident observations, and no new publication through low-confidence frames. These replay labels were never injected into production MQTT. Replay evaluation uses real historical images. Difficult tall and partly visible boxes initially failed and were moved into the training/calibration data, so they are not independent test results. The final separate check contains 28 clear frames and one unseen shipping-bag delivery. The broad model's top label matches all 29; the edge model has one wrong label below the acceptance threshold, so it does not publish that prediction. Some clear frames fall below threshold in both models. This small set does not establish production accuracy or removal reliability. Per-crop inference measured roughly 5-6 ms on the current host; this is not end-to-end alert latency. + +Before calling the replacement proven, verify: + +- A new delivery produces a stable present state and one notification. +- A stationary package remains present through lighting changes. +- Pickup/removal produces clear from both crops. +- Night, white mailers, shadows, occlusion, and small edge packages behave correctly. +- Camera/MQTT outages and recovery preserve safe state and resume updates. +- New errors are categorized in Frigate and used for retraining. + +## Rollback and future work + +Retain the prior package YAML, Frigate configuration, and protected Home Assistant backup privately. To return to LLM Vision, reinstall its HACS component and configure the provider again, then restore the prior automation configuration and validate before reloading. The pre-removal backup also retains the removed integration configuration. Timestamp rollback removes the overlay, re-enables the camera timestamp, and restores its original mask. Use supported integration/configuration workflows; do not edit Home Assistant storage or Frigate model-cache files manually. + +OpenAI Decisions remains an optional verification experiment in issue #1984. It is unnecessary for this local implementation and was not enabled or benchmarked. A candidate-only verifier cannot catch a package missed by the local classifier, so any later verification design should account for reconciliation too. + +A future blog/video can show the original LLM Vision experiment, camera crops, labeling/training, the timestamp change, difficult cases, and measured real delivery/removal results. The [original video](https://youtu.be/nAhCezFetvI) documents the earlier cloud-based approach.