From d86623180a39165e6fdb6ad819cb4f88dc545689 Mon Sep 17 00:00:00 2001 From: Simon Date: Sun, 24 May 2026 23:35:56 +0100 Subject: [PATCH] Bug fixes in dial, implement dial on both slots, tidy up voice prompts, improve ebm_lc handling, allow Talker Alias and in-call GPS to transit FreeDMR. --- docs/codex-notes.md | 1020 +++++++++++++++++++++++++++++++++++ docs/test-harness-design.md | 106 ++-- docs/testing.md | 52 +- 3 files changed, 1132 insertions(+), 46 deletions(-) diff --git a/docs/codex-notes.md b/docs/codex-notes.md index 8310503..e078e23 100644 --- a/docs/codex-notes.md +++ b/docs/codex-notes.md @@ -4278,3 +4278,1023 @@ - Changed the proxy SIGTERM handler to use the same graceful shutdown handler as SIGINT. - Added auxiliary test coverage for proxy environment boolean parsing. + +## 2026-05-23 - Voice Embedded LC Carry-Over + +### Findings +- `bridge_master.py` regenerated embedded LC for every voice burst B-E on every + forwarded voice stream, even when the outbound TG matched the inbound packet + destination TG. +- That unconditional regeneration preserves late-entry group voice LC for + dial-a-TG/TG-mapped forwarding, but it also overwrites non-routing embedded LC + payloads that may carry Talker Alias or in-call GPS information. + +### Assumptions +- The only production packet fields that require embedded-LC regeneration in the + current bridge path are routing identity fields, especially a changed target + TG. +- Same-TG forwarding should preserve DMR payload bytes unless another explicit + production rewrite is required. +- Dial-a-TG/reflector forwarding still requires embedded LC to be regenerated + for the reflector TG seen by the destination side. + +### Unresolved Questions +- A fuller embedded-LC assembler/classifier may still be useful later so + FreeDMR can preserve Talker Alias/GPS cycles even across TG-mapped forwarding. + The narrow change here does not buffer or reorder live voice bursts. +- Real RF testing should confirm that preserving same-TG embedded LC does not + expose dashboard or terminal interoperability assumptions from earlier + generated-LC behavior. + +### Protocol-Sensitive Areas +- Voice burst B-E payload bits 116..148 carry embedded LC fragments. Rewriting + them is a protocol mutation, not transport simulation. +- Full LC voice header/terminator rewrite remains unchanged. +- Dial-a-TG and TG-mapped bridge targets still need regenerated embedded LC so + late-entry receivers see the destination TG FreeDMR is actually transmitting. + +### Inferred Invariants +- Preserve packet bytes unless FreeDMR intentionally rewrites them for routing. +- Same-TG forwarding should not destroy embedded Talker Alias/GPS-like payloads. +- TG-mapped forwarding should prefer routing correctness over opaque embedded-LC + preservation. + +### Resolution +- Added a small `target_requires_emb_lc_rewrite()` helper in `bridge_master.py` + and gated the four voice burst B-E embedded-LC rewrite sites on a changed + target TG. +- Added deterministic harness coverage for HBP-to-HBP, HBP-to-FBP, FBP-to-HBP + and FBP-to-FBP same-TG voice embedded-LC payload preservation. +- Added deterministic coverage that TG-mapped forwarding still rewrites the + payload and destination TG. +- Added black-box UDP coverage that same-TG HBP voice forwarding preserves the + DMR payload observed by another registered repeater. +- Updated test and harness architecture documentation. + +## 2026-05-23 - In-Call TA/GPS Log Observability + +### Findings +- `dmr_utils3.decode_emblc()` does not match the MMDVMHost embedded-LC bit + mapping closely enough to use for user-facing Talker Alias/GPS diagnostics. + A simple encode/decode fixture changed the TA LC bytes. +- MMDVMHost's `CDMREmbeddedData` implementation unpacks embedded LC by arranging + the four 32-bit burst fragments down 16-bit columns, then extracting the 72 LC + payload bits from the data rows. + +### Assumptions +- System-log TA/GPS visibility should be passive observability. It must not + change packet routing, duplicate/out-of-order handling, or packet rewrite + behaviour. +- Logging should only happen for accepted voice packets after packet-control + filters, so duplicate or late packets do not produce misleading log entries. +- The first useful diagnostic output is decoded TA text/progress, GPS decimal + latitude/longitude, and raw LC hex for comparison against external tools. + +### Unresolved Questions +- The local decoder currently follows MMDVMHost's column unpacking and payload + extraction but does not yet port its Hamming(16,11,4) correction or five-bit + CRC validation. If noisy embedded LC from real RF produces false positives, + porting those checks is the next step. +- Talker Alias vendor/profile differences may need additional text-format + handling after live RF testing. + +### Protocol-Sensitive Areas +- Embedded LC fragments are taken from accepted voice bursts B-E only. +- Talker Alias FLCOs are treated as `0x04` header and `0x05..0x07` blocks. +- GPS Info FLCO is treated as `0x08`; logged coordinates are decoded from the + embedded LC payload bits and raw LC hex is included for validation. + +### Inferred Invariants +- Observability must not mutate packet bytes. +- Embedded-LC diagnostic decode should be based on MMDVMHost's mapping, not + `dmr_utils3.decode_emblc()`. +- Logs should contain enough raw data to compare against MMDVMHost or RF-side + captures. + +### Resolution +- Added a local MMDVMHost-style `decode_embedded_lc()` helper in + `bridge_master.py` for logging only. +- Added accepted-packet logging for in-call Talker Alias and GPS embedded LC in + both HBP and OBP receive paths. +- Added deterministic and UDP black-box tests that verify TA and GPS log output. +- Updated testing and harness architecture documentation. + +## 2026-05-23 - Standalone MMDVMHost Embedded LC Codec + +### Findings +- A correct embedded LC implementation needs more than the column unpacking used + for passive logging. MMDVMHost also applies Hamming(16,11,4) row correction, + column parity, and the DMR 5-bit checksum. +- Keeping this work in a standalone module lets FreeDMR prove codec behaviour + before changing packet forwarding or LC rewrite paths. + +### Assumptions +- `freedmr_dmr_codec.py` should be FreeDMR-owned code, not a drop-in replacement + for all of `dmr_utils3` yet. +- The first useful API should operate on complete embedded LC cycles and DMR + payload embedded-LC slices, because that is the boundary needed for future + selective TA/GPS preservation. +- Existing packet paths should not be switched to the new codec until the module + has direct tests and at least initial live/fixture validation. + +### Unresolved Questions +- The standalone codec has synthetic and internally checked fixtures, but should + still be compared with real MMDVMHost captures or known-good RF samples before + it becomes the production rewrite encoder. +- Full LC, slot type, and other BPTC helpers may also be worth moving into this + module later, but only after embedded LC is proven. + +### Protocol-Sensitive Areas +- Embedded LC encode/decode uses the MMDVMHost `CDMREmbeddedData` matrix mapping. +- Hamming(16,11,4) correction is applied row-by-row before column parity and + checksum validation. +- The codec returns four 32-bit embedded LC fragments suitable for voice bursts + B-E and helpers to insert/extract those fragments from 33-byte DMR payloads. + +### Inferred Invariants +- Codec tests must run independently of `bridge_master.py` routing state. +- A single bit error in an embedded LC row should be corrected; uncorrectable row + errors should be rejected. +- Future packet-path integration should preserve the existing real-time no-jitter + behaviour unless explicitly revisited. + +### Resolution +- Added `freedmr_dmr_codec.py` with MMDVMHost-style embedded LC encode/decode, + Hamming(16,11,4), column parity, 5-bit checksum, and DMR payload slice helpers. +- Added `tests/test_freedmr_dmr_codec.py` covering TA/GPS round trips, expected + encoded fragment placement, single-bit correction, uncorrectable error + rejection, and payload insertion/extraction helpers. +- Updated test and harness documentation. The new module is not wired into + packet forwarding yet. + +## 2026-05-23 - Standalone Full LC and Slot Type Codec + +### Findings +- The next production-relevant codec operations after embedded LC are full LC + header/terminator BPTC generation and slot type encode/decode. These are the + operations FreeDMR already uses for generated voice, dial-a-TG/TG-mapped LC + rewrite, and frame classification. +- Full LC generation depends on RS(12,9) parity, DMR header/terminator parity + masks, Hamming(15,11,3), Hamming(13,9,3), and the BPTC(196,96) interleaver. +- Slot type generation depends on Golay(20,8,7). The decoder can safely correct + up to three bits in the standalone layer before any production path chooses to + use it. + +### Assumptions +- This remains a proving layer. `bridge_master.py`, `bridge.py`, and + `mk_voice.py` should continue using their existing runtime codec paths until + we explicitly decide to integrate the new module. +- The compatibility baseline for this step is the current FreeDMR/dmr_utils3 + output, because changing generated LC bytes would be a production behaviour + change. +- LC classification should expose routing-relevant fields without deciding + routing policy. + +### Unresolved Questions +- The full LC decoder currently mirrors the existing FreeDMR behaviour by + extracting the LC payload from the BPTC matrix. It does not yet apply full + production-grade correction/validation to noisy received headers. +- Real MMDVMHost or RF capture vectors should still be added before using this + module as the production encoder/decoder. + +### Protocol-Sensitive Areas +- Full LC header and terminator parity masks differ. A wrong mask changes + whether receivers treat the LC as a voice header or terminator. +- Slot type bits carry color code and DMR data type; mutating them can change + packet class interpretation. +- LC byte layout is preserved as options, target/destination ID, then source ID + for the current FreeDMR voice paths. + +### Inferred Invariants +- Standalone codec tests must not require `bridge_master.py` globals or network + harness setup. +- Codec helpers must not rewrite packet bytes by themselves; they only encode or + decode caller-provided fields. +- Future integration should compare output byte-for-byte against existing + production behaviour before replacing `dmr_utils3` calls. + +### Resolution +- Extended `freedmr_dmr_codec.py` with full LC header/terminator encode/decode, + RS(12,9) LC parity, Hamming(15,11,3), Hamming(13,9,3), Golay(20,8,7) slot + type encode/decode and a small LC classifier. +- Extended `tests/test_freedmr_dmr_codec.py` with fixtures matching the current + codec output, parity-mask checks, group/unit LC classification, slot type + fixtures, three-bit correction and uncorrectable slot type rejection. +- Updated test and architecture documentation. Runtime packet forwarding is not + changed by this step. + +## 2026-05-23 - Embedded LC Logging Uses Standalone Codec + +### Findings +- `bridge_master.py` still had a local embedded-LC unpacker for TA/GPS logging + after the standalone codec module was added. +- That local helper did not perform the Hamming, column parity, or checksum + validation now available in `freedmr_dmr_codec.py`. + +### Assumptions +- TA/GPS logging is observability only. Decode failures must not interrupt voice + packet handling or change routing. +- Moving logging decode onto the standalone codec is a safe first production + integration because it does not rewrite packet bytes. +- Existing hand-coded TA/GPS test payloads should be replaced with payloads + generated by the standalone encoder so tests exercise valid embedded-LC + cycles. + +### Unresolved Questions +- Live RF may produce noisy embedded LC that is now rejected instead of being + logged from a best-effort unpack. That is preferable for avoiding false + positives, but should be watched during RF testing. +- Future work may replace `dmr_utils3` LC generation in packet rewrite paths, + but that is not part of this change. + +### Protocol-Sensitive Areas +- Embedded LC validation is stricter than the previous logging-only unpacker. + This affects only whether a TA/GPS log line is emitted. +- Packet forwarding, LC rewrite decisions, and payload bytes are unchanged by + this integration. + +### Inferred Invariants +- Codec decode errors in observability paths are non-fatal. +- TA/GPS logging should only use complete B-E embedded-LC cycles. +- Test fixtures for codec-sensitive logging should be generated through the + codec layer, not handwritten as raw payload fragments. + +### Resolution +- Switched `bridge_master.py` TA/GPS embedded-LC logging to + `freedmr_dmr_codec.decode_embedded_lc()`. +- Added non-fatal debug logging for embedded-LC validation failures. +- Updated deterministic and UDP TA/GPS fixture setup to generate payloads with + `freedmr_dmr_codec.encode_embedded_lc()`. +- Added codec tests that compare full LC header/terminator and routing-style + embedded LC output with the current `dmr_utils3` encoders before any rewrite + path replacement. +- Updated test and harness documentation. + +## 2026-05-23 - UDP Harness Startup Readiness + +### Findings +- The UDP black-box subprocess could be alive but not yet bound to HBP UDP + sockets when the first emulated repeater sent `RPTL`. +- The failure mode was a timeout waiting for the login challenge, before any + DMR packet or TA/GPS fixture was sent. +- Waiting a few seconds before login allowed the same subprocess to answer, + confirming this was a harness readiness race rather than packet-path logic. + +### Assumptions +- Retrying only the initial HBP login challenge is safe for the harness because + it models a client retrying while the server is still starting. +- Authentication/configuration failures after a challenge is received should + still fail immediately. + +### Unresolved Questions +- `bridge_master.py` still has no explicit "UDP sockets bound" readiness log. + A future harness improvement could detect readiness more directly if the + process emits such a message. + +### Protocol-Sensitive Areas +- This affects test transport timing only. It does not change FreeDMR packet + handling, HBP authentication, routing or rewrite logic. + +### Inferred Invariants +- UDP black-box tests should not assume that subprocess creation means UDP + sockets are already bound. +- Login retry belongs in the test client, not in production code. + +### Resolution +- Updated `HbpRepeater.login()` in the UDP harness to retry the initial `RPTL` + challenge for a bounded startup window. +- Verified a baseline UDP route and the TA/GPS UDP log scenarios pass after the + retry change. +- Updated testing and harness architecture documentation. + +## 2026-05-23 - Runtime LC Generation Uses FreeDMR Codec + +### Findings +- Runtime LC generation was still routed through `dmr_utils3.bptc` even after + `freedmr_dmr_codec.py` had byte-compatible full LC and embedded LC encoders. +- The production call sites only need `encode_header_lc()`, + `encode_terminator_lc()` and `encode_emblc()`, so a small compatibility + surface avoids broader packet logic edits. + +### Assumptions +- Replacing encode calls is acceptable only because codec tests compare the new + output byte-for-byte with the current `dmr_utils3` output for the runtime + function names. +- Decode paths can remain on `dmr_utils3` until they are ported and validated + separately. +- `bridge.py` and `mk_voice.py` should move with `bridge_master.py` because + they already use the same supported LC generation operations. + +### Unresolved Questions +- Real RF testing should still watch generated prompts and dial-a-TG/TG-mapped + LC rewrite behavior, because this is production voice-path code even though + the bytes are covered by compatibility tests. +- Full noisy received LC correction/validation is not yet used in production + decode paths. + +### Protocol-Sensitive Areas +- Full LC header/terminator bytes and embedded LC fragments must remain + byte-compatible unless FreeDMR intentionally changes the protocol behavior. +- `encode_emblc()` keeps the historical dict keyed by voice burst sequence + `1..4` so existing rewrite code can index by `_dtype_vseq`. + +### Inferred Invariants +- Runtime LC generation must preserve existing `dmr_utils3` byte output during + this migration. +- The codec module may offer compatibility names, but routing and rewrite + policy stay in `bridge_master.py` / `bridge.py`. + +### Resolution +- Added `encode_header_lc()`, `encode_terminator_lc()` and + `encode_emblc()` compatibility functions to `freedmr_dmr_codec.py`. +- Switched `bridge_master.py`, `bridge.py` and `mk_voice.py` LC generation + imports from `dmr_utils3.bptc` to `freedmr_dmr_codec`. +- Added a runtime-compatibility codec test that checks those function names + against the current `dmr_utils3` encoder output. +- Verified codec tests, deterministic LC rewrite/generated-prompt tests, full + non-UDP discovery, and focused UDP voice/prompt/TA/GPS scenarios. + +## 2026-05-23 - Runtime Voice Decode Helpers Use FreeDMR Codec + +### Findings +- After moving LC generation onto `freedmr_dmr_codec.py`, `bridge_master.py` and + `bridge.py` still used `dmr_utils3.decode.voice_head_term()` for full LC + extraction and `dmr_utils3.decode.voice()` for embedded-LC fragment extraction. +- Those decode helpers are part of the same codec surface as the LC work already + ported. + +### Assumptions +- The replacement must preserve the existing dict return shape: + `voice_head_term()` returns `LC`, `CC`, `DTYPE`, `SYNC`; `voice()` returns + `AMBE`, `CC`, `LCSS`, `EMBED`. +- Full received-LC correction/validation remains out of scope for this step; the + goal is byte-compatible replacement of the current helper behavior. +- `dmr_utils3` can remain for unrelated utilities/constants while these codec + operations move to FreeDMR-owned code. + +### Unresolved Questions +- Whether to port more received full-LC correction from MMDVMHost before using + decoded noisy headers more aggressively. +- Whether to remove compatibility tests against `dmr_utils3` after enough + external vectors or recorded RF fixtures exist. + +### Protocol-Sensitive Areas +- `voice_head_term()` feeds stream LC state used for TG rewrite and reporting. +- `voice()` feeds embedded-LC TA/GPS logging and must preserve the burst + fragment bit slice exactly. + +### Inferred Invariants +- Runtime decode helper return values must remain compatible with the old call + sites until packet logic is deliberately refactored. +- Replacing helpers must not change routing, stream lifecycle, packet mutation + decisions or generated voice bytes. + +### Resolution +- Added `voice_head_term()`, `voice_sync()` and `voice()` helpers to + `freedmr_dmr_codec.py`. +- Switched `bridge_master.py` and `bridge.py` from `dmr_utils3.decode` to + `freedmr_dmr_codec` for those helpers. +- Added codec tests comparing `voice_head_term()` and `voice()` outputs against + the current `dmr_utils3` behavior on known DMR payload fixtures. +- Verified focused deterministic voice/LC paths, bridge backport coverage, full + non-UDP discovery, and focused UDP voice/prompt/TA/GPS scenarios. + +## 2026-05-23 - Active Runtime dmr_utils3 Helper Cleanup + +### Findings +- Active runtime modules still used `dmr_utils3.utils` for `bytes_3()`, + `bytes_4()`, `int_id()` and `get_alias()` after packet codec operations had + moved to `freedmr_dmr_codec.py`. +- `mk_voice.py` also used fixed DMR bit constants from `dmr_utils3.const`. +- The helper implementations are small and deterministic, and the constants + are fixed bit patterns already covered by generated voice prompt tests. + +### Assumptions +- Replacing these helpers is behaviour-preserving because the FreeDMR versions + intentionally match the observed `dmr_utils3` implementations. +- `const.py` remains the source for FreeDMR/HBP constants such as ACL bounds and + the existing late-entry `LC_OPT`; the DMR prompt-generation bit constants live + with codec helpers to avoid changing those older meanings. +- Legacy/lab scripts may continue to import `dmr_utils3` until they are updated + or retired. + +### Unresolved Questions +- Whether to update legacy tools such as playback/app templates now, or leave + them until their current use is confirmed. +- Whether `requirements.txt` should keep `dmr_utils3` while legacy tools still + import it, even though active runtime and harness paths no longer need it. + +### Protocol-Sensitive Areas +- `mk_voice.py` generated prompt frames rely on the exact LC option, sync, + embedded-LC and slot-type bit patterns. +- `int_id()` is used widely for packet fields and logging; it must keep treating + byte strings as big-endian integer IDs. + +### Inferred Invariants +- Codec/helper replacement must not alter routing, stream lifecycle, packet + mutation decisions, prompt packet bytes or UDP harness behaviour. +- Test fixtures should not require `dmr_utils3` once fixed vectors exist. + +### Resolution +- Added FreeDMR-owned `bytes_3()`, `bytes_4()`, `int_id()` and `get_alias()` to + `utils.py`. +- Added the prompt-generation DMR bit constants to `freedmr_dmr_codec.py`. +- Switched active runtime imports in `bridge_master.py`, `bridge.py`, + `hblink.py`, `API.py`, `hotspot_proxy_v2.py`, + `hdstack/hotspot_proxy_v2.py`, `config.py` and `mk_voice.py`. +- Removed `dmr_utils3` from the codec tests and UDP harness runtime dependency + check, replacing comparisons with fixed vectors. +- Added direct utility-helper tests and updated test/harness documentation. + +## 2026-05-23 - Consistency Review After Codec Cleanup + +### Findings +- Active runtime imports are now internally consistent in that `dmr_utils3` is + no longer used by `bridge_master.py`, `bridge.py`, `hblink.py`, `API.py`, + `mk_voice.py`, `config.py` or the proxy modules. +- The compatibility choice of importing `freedmr_dmr_codec` as both `bptc` and + `decode` kept the initial diff small, but it was visually odd because both + names referred to the same module. +- `LC_OPT` existed in two places with different values: `const.py` kept the + existing FreeDMR/HBP late-entry value, while `freedmr_dmr_codec.py` provided + the older DMR prompt-generation value previously imported from + `dmr_utils3.const`. This was behaviour-preserving but a readability hazard. +- The new embedded-LC observation helpers in `bridge_master.py` are logically + grouped, but the block would be easier to scan with a short section comment + and a couple of wrapped long lines. +- The utility helper implementations in `utils.py` intentionally mirror + `dmr_utils3.utils`, but the existing file comment says "modified" helpers, + which is now slightly misleading for the compatibility helpers. + +### Assumptions +- Preserving old names at call sites was preferred for the first cleanup pass + because it reduced production diff size. +- Legacy/lab tools can continue to import `dmr_utils3` until their status is + revisited. + +### Unresolved Questions +- Whether the older `const.py` late-entry `LC_OPT` value should eventually be + renamed too. It was left unchanged in this cleanup because it is a broad + wildcard-imported runtime constant. + +### Protocol-Sensitive Areas +- Any cleanup around `LC_OPT` must preserve `mk_voice.py` prompt bytes and the + existing `const.py` late-entry behaviour. +- Any cleanup around `bptc`/`decode` aliases must be import/name-only and not + change packet routing or codec output. + +### Inferred Invariants +- Human-readable cleanup should stay mechanical unless a real behaviour bug is + identified. +- Compatibility vectors and UDP black-box tests remain the guardrails for codec + and prompt-generation cleanup. + +### Resolution +- Replaced the dual `bptc`/`decode` compatibility imports with a single + `dmr_codec` import in `bridge_master.py`, `bridge.py`, `mk_voice.py` and the + codec compatibility test. +- Added a short section comment for embedded-LC observation in + `bridge_master.py` and wrapped the longest log/decode lines in that helper + block. +- Updated the `utils.py` module comment to say the helper functions mirror the + old `dmr_utils3` surface rather than implying they are all modified copies. +- Renamed the generated-prompt LC option constant to + `GROUP_VOICE_LC_OPT`; the byte value remains `b'\x00\x00\x00'`. +- Left `const.py` `LC_OPT = b'\x00\x00\x20'` unchanged for the existing + late-entry fallback behaviour. +- Verified py_compile, full non-UDP discovery, codec/helper tests and focused + UDP prompt/routing smoke tests. + +## 2026-05-23 - OVCM and Late-Entry LC Option Review + +### Findings +- ETSI defines OVCM as Open Voice Channel Mode: a service option that allows + configured non-addressed users to monitor and participate in an active voice + call. +- MMDVMHost represents OVCM as bit `0x04` in the voice LC options byte. Its + generated/default LC constructor starts with zero options unless OVCM + configuration changes that bit. +- The current FreeDMR prompt generator uses `GROUP_VOICE_LC_OPT = + b'\x00\x00\x00'`, matching MMDVM/MMDVMHost default generated group voice LC + behaviour. +- The older FreeDMR late-entry fallback `LC_OPT = b'\x00\x00\x20'` was inherited + through HBLink-era constants. Local git history does not explain what the + `0x20` bit was intended to signal. + +### Assumptions +- Setting OVCM may be useful for interoperability with commercial DMR + implementations even if many amateur terminals and servers ignore it. +- Jonathan/MMDVM may have used OVCM deliberately for commercial compatibility, + but that needs confirmation from upstream history or mailing-list discussion. + +### Unresolved Questions +- Whether FreeDMR should set OVCM on generated/fallback group voice LC, leave it + clear, or preserve the historical `0x20` value until live RF testing proves a + safer default. +- Whether any commercial hardware connected to FreeDMR-derived paths relies on + the historical `0x20` option bit. + +### Protocol-Sensitive Areas +- Changing LC options affects generated prompts, late-entry fallback LC, + embedded LC rewrites, and potentially how commercial DMR equipment treats + channel busy/monitor behaviour. +- OVCM is not the same as the historical `0x20` value; treating them as + equivalent would be a protocol assumption not supported by the checked code. + +### Inferred Invariants +- If OVCM is introduced, it should be named explicitly and represented by the + known `0x04` service-option bit. +- The late-entry fallback LC option value should have a single source of truth + and tests should pin whichever byte pattern is intentionally selected. + +### Follow-Up Research From User +- Independent research agrees that `LC_OPT = b'\x00\x00\x20'` should not be + blindly preserved as the semantic default. It appears to set one of the + reserved service-option bits, not OVCM. +- For DMR Group Voice Channel User LC, the first three bytes are understood as + FLCO, FID and Service Options. Under the MMDVMHost-compatible interpretation, + service option bits are: emergency `0x80`, privacy `0x40`, reserved `0x30`, + broadcast `0x08`, OVCM `0x04`, priority `0x03`. +- The likely safe policy is: preserve real decoded inbound LC bytes unchanged; + use `0x00` for normal synthetic group voice LC; keep `0x20` only as an + explicit, documented HBLink legacy compatibility option during transition. +- This is a protocol-cleanliness/late-entry correctness issue, not a direct + dial-a-TG TG9/slot2 cause, because the destination TG and slot are carried + separately from the service-options byte. +- User later found that MMDVMHost originally implemented OVCM as `0x20` and + corrected it to `0x04` four days later. This supports treating FreeDMR/HBLink + `0x20` as a likely copied early OVCM bit mistake rather than an intentional + modern service option. +- More detailed research: MMDVMHost commit `6bababe` ("Add OVCM support", + 2019-10-11) initially used `0x20` in `CDMRLC::getOVCM()/setOVCM()`. Commit + `0711a2b` ("Fix issue in DMRLC.cpp", 2019-10-15) changed the same logic to + `0x04`. Later MMDVMHost behaviour changed again after Motorola compatibility + reports: do not remove OVCM unless configured, allow directional generation, + and eventually add force-off mode. This reinforces two FreeDMR rules: decoded + inbound LC should be passed through unchanged, while synthetic LC should use + standards-clean defaults unless a compatibility option is explicitly selected. + +### Resolution +- Added named LC service-option constants and `build_group_voice_lc()` to + `freedmr_dmr_codec.py`. +- Switched only synthetic late-entry/fallback LC construction in + `bridge_master.py` and `bridge.py` to generate normal group voice LC with + service options `0x00`. +- Left decoded inbound voice-header LC preservation unchanged. +- Kept `LC_SERVICE_OPTIONS_HBLINK_LEGACY = 0x20` as an explicit compatibility + value for tests or future interop switches, but removed active runtime use of + the old `LC_OPT` fallback. +- Documented `const.py` `LC_OPT` as a legacy compatibility alias rather than an + active synthetic LC default. +- Added codec and deterministic harness tests for normal synthetic LC fallback + and real inbound LC preservation. + +## 2026-05-23 - Bridge Table Storage and Activation Critique + +### Findings +- `BRIDGES` is currently a global dict keyed by bridge name/TG, where each value + is a list of mutable per-system dict entries. +- The same per-system bridge entries combine static configuration, derived + defaults, dynamic session state, timer state, activation triggers and + reporting-facing state. +- Packet routing and in-band activation scan the full bridge table repeatedly to + find entries matching `(SYSTEM, TS, TGID, ACTIVE)`. +- Dial-a-TG, default reflector, static TG and reset paths often replace list + entries by rebuilding temporary lists, while timer paths mutate entries in + place. +- Bridge names encode semantics: normal TGs use string names such as `"91"`, + reflectors use prefixed names such as `"#4400"`, and some code toggles between + those two forms to route between TG and reflector sides. + +### Assumptions +- The current bridge semantics are validated and should not be changed as part + of storage cleanup. +- Any replacement structure must preserve the existing reporting/API shape until + dashboard and API consumers are migrated. + +### Unresolved Questions +- Whether FreeDMR 2.0 should expose the existing raw `BRIDGES` structure to the + dashboard/API, or publish a compatibility view generated from an internal + bridge store. +- Whether bridge updates will remain single-reactor-thread operations, or + whether future multi-process work will require explicit ownership/snapshot + rules. + +### Protocol-Sensitive Areas +- Dial-a-TG and reflector bridge entries intentionally rewrite the RF-visible TG + and network-visible TG differently; a new structure must keep those two + identities distinct. +- Timer semantics differ for `ON`, `OFF`, `NONE` and `STAT`; these are routing + state transitions, not merely presentation flags. + +### Inferred Invariants +- A bridge route is effectively keyed by bridge identity plus local endpoint + `(system, slot)` and RF-visible `TGID`. +- Runtime packet routing needs fast lookup by `(system, slot, tgid)` rather than + repeated full-table scans. +- Configuration/default entries and per-session dynamic activation state should + be separable even if exported as the historical dict shape. + +### Follow-Up Assumption From User +- FreeDMR 2.0 should move away from treating configured `MASTER` stanzas as the + key for client/repeater state. The desired key is the client DMR ID, because a + single master/listener UDP port should be able to serve an arbitrary number of + client connections directly, replacing the proxy. +- This means a future bridge store should distinguish listener/system identity + from client session identity. Existing `SYSTEM`-keyed behaviour may need a + compatibility layer while routing state moves toward `(client_id, slot, tgid)` + and listener state remains tied to the UDP master. + +## 2026-05-23 - FreeDMR 2.0 Rewrite vs Iterative Upgrade + +### Findings +- `bridge_master.py` is currently about 3.4k lines and `hblink.py` about 1.6k + lines. The runtime identity model is widely coupled to `SYSTEM`, + `CONFIG['SYSTEMS']`, `systems[...]`, `PEERS`, `STATUS` and global `BRIDGES`. +- `hblink.py` owns Homebrew/OpenBridge transport, login/auth/config/options and + DMRD packet decoding. `bridge_master.py` owns routing, bridge activation, + dial-a-TG, reports, generated voice, timers, packet-control and much of the + runtime state. +- The desired FreeDMR 2.0 model changes the central identity from configured + master/system stanza to client DMR ID behind a shared UDP listener. That is an + architectural change, not a local bugfix. +- The current deterministic and UDP harnesses make staged replacement more + practical than it would have been before the test work. + +### Assumptions +- Packet/routing behaviour is validated and should be preserved unless a + FreeDMR 2.0 feature explicitly requires a change. +- Proxy removal means direct multi-client handling must be designed at the + listener/session layer rather than bolted onto the current one-client-ish + `MASTER` abstraction. + +### Unresolved Questions +- Whether FreeDMR 2.0 should keep Twisted for the first architecture migration + or move transport to asyncio/another event loop at the same time. +- Whether dashboard/API consumers can tolerate a compatibility payload generated + from a new internal state model. + +### Protocol-Sensitive Areas +- Homebrew login/auth, peer timeout, options parsing, keepalive handling and + DMRD decoding must remain byte-compatible if `hblink.py` is replaced. +- Dial-a-TG, reflector rewrite, BCSQ/source quench, loop-control and stream + timeout behaviour must remain covered by deterministic and UDP tests during + migration. + +### Inferred Invariants +- Do not do a big-bang rewrite of packet semantics. +- Do build a new listener/session/bridge-store core beside the current code, + migrate behaviours behind tests, and keep compatibility adapters until the + old `SYSTEM`-centric shape can be retired. + +### Follow-Up Assumption From User +- The FreeDMR-specific conceptual model is the protected asset: packet model, + dial-a-TG behaviour, protocol operation, loop control, source-quench/mesh + behaviour, and the practical tolerance learned from real global servers and + RF links. +- HBLink-era structure is not the protected asset where it prevents scale. The + assumptions around configured master/system identity, single-process state and + proxy-mediated client fan-out may be replaced to improve scalability, + performance and new functionality. + +## 2026-05-24 - Bridge Store Data Structure Research + +### Findings +- The packet plane should keep bridge/routing lookups in local memory. CPython + `dict`/`set` remain the right hot-path foundation because average lookup, + insert and delete are O(1) with suitable hash keys. +- `heapq` is a good built-in fit for timer expiry because it provides a min-heap + priority queue over a plain list. Timer updates should use lazy invalidation + rather than arbitrary heap removal. +- `typing.NamedTuple` keys provide readable attribute names while retaining + tuple-like hash/lookup performance. A local microbenchmark over 10k lookups + showed raw tuple and `NamedTuple` keys essentially equal; frozen dataclass keys + were materially slower. +- `dataclass(slots=True)` is a good fit for mutable state records because it is + readable, avoids per-instance `__dict__`, and keeps field ownership explicit. +- External sorted-map modules such as `sortedcontainers` are useful if sorted + range queries become central, but they are not necessary for packet routing + lookup and add a dependency. + +### Assumptions +- Bridge routing decisions are read far more often than they are mutated. +- Packet handlers need direct lookup by client/session identity, slot and + RF-visible TG, not full-table scans. +- External state stores can publish control-plane updates, but packet routing + must not block on them. + +### Unresolved Questions +- Whether route indexes should be per-worker only or whether a coordinator will + assign clients/TGs to workers in FreeDMR 2.0. +- Whether dashboard/API compatibility should continue to expose a generated + legacy `BRIDGES` dict or move to a new schema. + +### Protocol-Sensitive Areas +- Dial-a-TG must keep RF-visible TG9/TS2 and network reflector TG identity + separate. +- Loop-control and source-quench state must remain per stream/TG/source path and + cannot be hidden behind eventually consistent external state. + +### Inferred Invariants +- Use `NamedTuple` or tuple keys for hot indexes. +- Use slotted dataclasses for human-readable mutable records. +- Maintain multiple indexes that point to the same `BridgeEntry` objects rather + than one nested dict that forces scans. +- Use an append-only/lazy-invalidated timer heap for expiry, with the entry + itself carrying the authoritative expiry generation. + +### Follow-Up Assumption From User +- FreeDMR 2.0 does not need to preserve the `"#"` bridge-name convention + internally. Reflectors and conventional TG routes may use separate tables or + explicit route kinds if that is cleaner and faster. Any `"#"` form can be a + compatibility/export detail rather than the authoritative internal identity. + +## 2026-05-24 - Reporting and Dashboard Transport Direction + +### Findings +- The existing live dashboard is per-server and real-time, while the global + lastheard service is central and non-real-time. These should remain separate + reporting paths. +- HTTP/SSE or HTTP/WebSocket can replace the current custom reporting feed for + dashboard data, but the packet worker must not block on browser clients, + dashboard rendering, global collectors or external reporting stores. +- The packet process already uses Twisted, so serving a small local HTTP API is + technically feasible, but frontend/static hosting and long-lived dashboard + fanout are cleaner in a separate reporting service or sidecar. + +### Assumptions +- The live dashboard mainly needs one-way real-time updates plus a state + snapshot; it does not need every DMR packet. +- Dashboard control actions can use ordinary authenticated HTTP API calls unless + a future UI genuinely needs bidirectional streaming. + +### Unresolved Questions +- Whether the first implementation should embed the local reporting HTTP/SSE + endpoint in the FreeDMR process or run it as a sidecar fed by an internal + queue/socket/event bus. +- What authentication model should be used for local dashboard control APIs, + especially if dashboards are exposed beyond localhost/reverse proxy. + +### Protocol-Sensitive Areas +- Reporting must be observational. Packet routing, dial-a-TG, loop control and + mesh behaviour must continue if reporting is down, slow or disconnected. +- Global lastheard export should use call summaries/events, not packet-plane + traffic or local dashboard state. + +### Inferred Invariants +- Packet workers emit structured events and maintain/report local snapshots, but + never wait for dashboard consumers. +- Prefer `GET /api/state` plus SSE `GET /api/events/stream` for the live + dashboard; use HTTP POST/PATCH APIs for commands. +- WebSocket is optional and should be reserved for bidirectional dashboard + interactions that cannot be handled cleanly by HTTP API plus SSE. + +### Follow-Up Decision +- Architectural decisions, requirements and open questions for FreeDMR 2.0 are + now recorded in `docs/freedmr-2-architecture-decisions.md`. +- MQTT is selected as the preferred external live reporting transport, with + retained topics for current state and non-retained topics for transient + events. HTTP/SSE remains a possible gateway/fallback for dashboards, not the + primary reporting protocol decision. +- Pressure-test conclusion: MQTT emission is acceptable only through a + non-blocking bounded local queue and an independent publisher worker. Voice + stability takes precedence over reporting completeness; under pressure, + reporting events should be dropped/coalesced rather than delaying packet + handling. +- A network MQTT broker can also support global dashboard/lastheard collection + by receiving a curated subset of summary events from each server. This must be + separated from local live reporting so central outages do not affect local + packet handling or local dashboard state. +- Preferred refinement: a separate global-exporter process subscribes to the + same local real-time MQTT feed as the dashboard, filters/summarizes events and + publishes to the network broker/collector. Core FreeDMR publishes only the + local feed and is not responsible for global curation. +- Reporting fanout invariant: FreeDMR core emits each event once to the local + MQTT publisher/broker path. Additional dashboards/exporters/automation consume + from MQTT and must not add packet-process load. +- General FreeDMR 2.0 performance principle: expensive processing should be + considered for separate processes because CPython is constrained by the GIL + for CPU-bound Python code. Offload boundaries must be asynchronous and must + not add packet-path waits. +- Runtime migration decision: do not replace Twisted yet. Keep Twisted's + single-threaded reactor as a safety boundary while extracting and testing the + protocol/routing/subscription core. Consider asyncio only once Twisted is a + thin transport shell. +- Data packet policy: FreeDMR must keep forwarding DMR data packets, but core + should not become the GPS/SMS application processor. Data services should + connect via FBP or similar interfaces; `DATA_GATEWAY` is an earlier expression + of a data-oriented FBP link. Narrow exceptions may exist for SMS-based + dial-a-TG control or sysop alerts. +- No regression of data support is permitted. Preserve `SUB_MAP`/last-known + location semantics for routing data addressed to a DMR ID toward the last + known HBP/client location. +- Mesh security requirement: FreeDMR should support PKI-backed FBP peer + admission through Bridge Control (`BCXX`). Signed server identity should bind + server ID/public key/validity to the observed endpoint; if the IP changes, the + peer must re-authenticate before traffic is forwarded. Expensive validation is + control-plane only; per-packet checks use cached authenticated session state. +- Mesh auth renewal should be soft where safe: when renewal is due, trigger + asynchronous re-authentication and allow a bounded grace period so voice is not + interrupted by timing alone. Hard stops remain appropriate for revocation, + explicit auth failure, endpoint mismatch outside policy or grace expiry. +- Mesh PKI operational model: sign a sysop/server membership key when the server + joins the network; revoke that key when they leave or are compromised. Runtime + endpoint/session bindings are renewed separately without re-signing membership + each time. +- Mesh security option: distribute signed server membership keys peer-to-peer + over `BCXX` as bounded/rate-limited key gossip. Each server builds its own + validated key table and applies local policy to traffic by originating source + server, including traffic arriving indirectly through the mesh. This supports + FreeDMR's autonomous peer-network/zero-trust model, but requires replay, + revocation, expiry, serial and flood controls. +- Core security/legal principle: FreeDMR may sign/authenticate traffic and + control messages but should not encrypt amateur-radio or mesh traffic by + default. The security model is authenticity/integrity/membership validation + and local policy, not secrecy, because amateur radio is public and IP backhaul + may itself traverse amateur radio links. +- Project philosophy: FreeDMR should remain open-source, open, readable and + intentionally simple enough to encourage community implementation and + experimentation. HBLink proved an open DMR server could exist outside + commercial gatekeeping; FreeDMR extends that to an autonomous peer network + without central control. +- Mesh philosophy: FreeDMR's mesh thinking is influenced by Bob Bruninga's APRS, + Spanning Tree Protocol and similar distributed-network approaches. The project + is partly technical and partly diplomacy; design choices must respect + autonomy, interoperability and trust between independent sysops. +- Deployment/ethos principle: FreeDMR is successful because it works as a + best-effort amateur-radio/hacker system on cheap VPS/Raspberry Pi-class + hardware. FreeDMR 2.0 should improve clarity, quality and scalability without + losing approachability, low-cost deployment or experimental ham spirit. +- Network freedom principle: FreeDMR exists to lower barriers to global-scale + amateur ROIP experimentation. It changed a landscape where DMR server software + and server-level network membership were often closed or gatekept by personal + or team approval. +- Listing distinction: FreeDMR controls public listing/discovery for Pi-Star and + other HBP clients, not all private experimentation. Private servers can run + under their own DMR ID and be gatewayed by an existing sysop who vouches for + the traffic. Public listing carries additional operational requirements. +- Server identity hierarchy: FreeDMR server IDs are 4-digit DMR IDs and server + sub-IDs are 5-digit IDs, giving a sysop up to 10 IDs for backend/failover + deployments. One signed key verification should cover the base server ID and + authorized sub-IDs, subject to local policy and session/endpoint binding. +- Vouching/accountability model: an individual 7-digit DMR ID can be used for a + private server and traffic may pass if a directly connected/listed sysop + allows it. The vouching sysop is accountable for forwarded traffic; peers may + stop peering if that traffic harms the network. +- Analogue network bridges may connect as HBP clients and are permitted more + liberally than on many networks, but they can be problematic and effectively + listen-only. They should be subject to local policy, listing expectations and + peer accountability because they may consume resources without adding value. +- FreeDMR works with/supports the DVSwitch community for analogue bridging. + YSF/NXDN interworking is generally a better digital match because AMBE-family + audio can avoid transcoding; analogue/unlike-codec transcoding can create poor + audio artifacts. +- Analogue bridges often use audio mixing/conference behaviour, which is a poor + fit for DMR's one-audio-source-at-a-time stream and contention model. +- Analogue repeater heritage explains the mismatch: constant carriers, pips, + CWID and courtesy tones can be mixed into output audio, and source identity is + often weak compared with DMR's explicit DMR ID. +- Specific analogue bridge failure mode: an analogue repeater feed can keep the + DMR stream open between analogue overs, play courtesy tones and carry the next + analogue user in the same held stream. This can hold a TG open and prevent + digital stations from breaking in until the analogue carrier/timer drops. + +## 1.x dial-a-TG slot-local control update + +Findings: +- Current `bridge_master.py` forced all dial-a-TG private-call control packets + to mutate TS2 reflector state by setting `_dial_tg_slot = 2`, even when the + control packet arrived on TS1. +- With dial-a-TG enabled on TS1, cross-slot TS1 -> TS2 control is no longer safe: + TS1 must be able to own its own dial-a-TG state. +- Conventional TG dynamic routing is the group-call counterpart to dial-a-TG + reflector routing: unknown conventional group TGs can be created as user + activated bridges by `make_single_bridge()`. + +Assumptions: +- Private-call dial-a-TG control should now affect the slot on which the + private call is received. +- TS2 private-call control still controls TS2. +- TS1 private-call control no longer retunes, disconnects or reports TS2 + reflector state. +- Voice prompts remain RF-visible as TG9 TS2, matching the existing prompt + policy. +- `DEFAULT_REFLECTOR` remains the existing TS2 default-reflector compatibility + setting. Per-slot defaults are handled by the newer + `DEFAULT_DIAL_TS1`/`DEFAULT_DIAL_TS2` settings. + +Implementation notes: +- Added explicit per-master `DIAL_A_TG` and `DYNAMIC_TG_ROUTING` booleans, + both defaulting to `True` for compatibility. +- Peer OPTIONS aliases `DIALTG=0/1` and `DYNAMIC=0/1` update these booleans for + the current session. +- Generated reflector bridges now include both TS1 and TS2 HBP entries, with + only the controlled slot active. +- `bridgeDebug()` now treats one active dial bridge per system per slot as the + valid state, instead of one active dial bridge per system globally. + +Protocol-sensitive areas: +- TG9 remains the RF-visible dial-a-TG presentation TG for both slots. +- HBP <-> FBP dial-a-TG rewrite remains the only intended TG rewrite. +- Prompt/ident traffic is still generated on TG9 TS2. +- Source quench must continue to use the FBP/reflector TG namespace, not local + RF TG9. + +Inferred invariants: +- At most one active dial-a-TG reflector per system per slot. +- Disabling `DIAL_A_TG` must prevent private-call reflector creation/retune. +- Disabling `DYNAMIC_TG_ROUTING` must prevent automatic creation of unknown + conventional TG bridges, while existing/static bridge behaviour remains + otherwise unchanged. + +Unresolved questions: +- Whether disabling `DYNAMIC_TG_ROUTING` should also deactivate already-created + conventional user-activated bridges, or only prevent new automatic creation. + +## 1.x per-slot default dial-a-TG options + +Findings: +- `DEFAULT_REFLECTOR` was the historical TS2 dial-a-TG default setting. +- Slot-local dial-a-TG control needs slot-local startup/session defaults so TS1 + can have its own default dial reflector state. + +Assumptions: +- `DEFAULT_REFLECTOR` is deprecated, but remains a compatibility alias for TS2. +- `DEFAULT_DIAL_TS1` and `DEFAULT_DIAL_TS2` are the canonical per-slot default + dial-a-TG settings. +- If both `DEFAULT_REFLECTOR` and `DEFAULT_DIAL_TS2` are present in config, + `DEFAULT_DIAL_TS2` is preferred. +- Client OPTIONS aliases `DIAL`, `StartRef` and `DEFAULT_REFLECTOR` continue to + control TS2 only. +- If client OPTIONS include both canonical `DEFAULT_DIAL_TS2` and deprecated TS2 + aliases, `DEFAULT_DIAL_TS2` is preferred. + +Implementation notes: +- `config.py` now reads `DEFAULT_DIAL_TS1` with fallback `0`, and + `DEFAULT_DIAL_TS2` with fallback to deprecated `DEFAULT_REFLECTOR`. +- Runtime `CONFIG['SYSTEMS'][system]['DEFAULT_REFLECTOR']` is kept equal to the + effective TS2 default for compatibility with existing callers. +- `make_default_reflector()` now accepts a slot and creates/activates the + default reflector on that slot. +- `make_default_reflectors()` applies startup defaults independently for TS1 and + TS2, logs invalid positive defaults, and normalizes invalid runtime values to + `0`. +- `options_config()` accepts `DEFAULT_DIAL_TS1` and `DEFAULT_DIAL_TS2`; legacy + `DIAL`, `StartRef` and `DEFAULT_REFLECTOR` map to `DEFAULT_DIAL_TS2` only + when the canonical value is not present. + +Protocol-sensitive areas: +- Voice prompts remain generated on TG9 TS2. +- The compatibility `DEFAULT_REFLECTOR` value must continue to describe TS2, + not TS1. +- Default dial reflector activation must use the same validation policy as live + RF dial-a-TG activation. + +Inferred invariants: +- Empty, `0` or false default dial values mean no default reflector. +- Invalid positive defaults must not create bridge state. +- TS1 and TS2 default dial state are independent. +- `DEFAULT_DIAL_TS2` takes precedence over deprecated `DEFAULT_REFLECTOR`. + +Unresolved questions: +- Whether sample configs should keep showing `DEFAULT_REFLECTOR` for one release + as a visible deprecated alias, or remove it after downstream configs have + migrated. + +## Reporting check for slot-local default dial-a-TG changes + +Findings: +- The report socket opcode model is unchanged: `BRIDGE_SND` still sends pickled + `BRIDGES`, and `BRDG_EVENT` still sends comma-separated bridge event strings. +- The per-slot default-dial changes do not add new report event names or change + the existing event field order. +- `DEFAULT_REFLECTOR` remains in runtime config as the effective TS2 default, + reducing risk for existing config/API/report consumers. +- Voice prompts remain TG9 TS2, preserving the current prompt/dashboard policy. +- TS1 dial-a-TG can now legitimately create active `#reflector` entries with + `TS == 1`; this is the main dashboard-facing shape change. + +Assumptions: +- The dashboard treats bridge entries as dictionaries and reads `SYSTEM`, `TS`, + `TGID` and `ACTIVE`, rather than assuming all `#reflector` entries are TS2. +- The dashboard can tolerate seeing TG9 activity on slot 1 for dial-a-TG because + TS1 dial-a-TG is now intentional behavior. + +Protocol-sensitive areas: +- HBP/RF-side dial-a-TG reports should continue to show RF-visible TG9. +- FBP/OpenBridge-side target reports should continue to show the reflector TG + visible on the peer side. +- Source quench remains keyed to the peer-visible TG namespace, not the + dashboard display TG. + +Inferred invariants: +- Reporting must not steer routing. +- Report event field order must remain stable for FreeDMR 1.x dashboard + compatibility. +- `BRIDGE_SND` consumers must inspect `TS` rather than infer slot from bridge + name. + +Unresolved questions: +- The actual dashboard should be live-tested against active TS1 dial-a-TG + reflector state before release, because dashboards may have undocumented + assumptions about `#reflector` entries being TS2-only. diff --git a/docs/test-harness-design.md b/docs/test-harness-design.md index c0c99f7..6150d87 100644 --- a/docs/test-harness-design.md +++ b/docs/test-harness-design.md @@ -38,7 +38,9 @@ The harness code is split as follows: including optional OpenBridge/FBP peer sections. - `FreeDmrProcess`: starts and stops `bridge_master.py`. - `HbpRepeater`: UDP HBP client emulator with login, ping, packet send, stream - send, and capture support. + send, and capture support. The initial login challenge is retried for a + bounded startup window so subprocess startup work does not race the first + test packet. - `FbpPeer`: UDP FBP v5 peer emulator with signed packet sends, keepalive, version negotiation, STUN and source-quench control helpers. - `UdpBlackBoxScenario`: process plus two-master loopback topology with @@ -125,17 +127,19 @@ for rule timeout checks without sleeping. target. When AllStar is disabled it reports busy; when enabled it enters AllStar mode and schedules reset. It should not create or retune reflector state or announce a dial-a-TG link. -- Dial-a-TG default-reflector configuration bugs: startup and live options reload - should use the same prohibited default-reflector targets. Reserved/control - targets such as `6`, `7`, and AllStar control target `8` should not create an - active default TS2 reflector at startup. The FreeDMR policy cap should match - RF dial-a-TG handling: `999999` is valid and higher default reflectors are - rejected. Invalid default-reflector options should disable any existing - default reflector for the current session rather than preserving stale TS2 TG9 - reflector state. Invalid startup defaults should be logged and should not - create bridge state; the in-memory effective default should normalize to `0` - without writing back to the config file. System-wide defaults are intended for - sparing use; client requested settings are preferential. +- Dial-a-TG default-dial configuration bugs: startup and live options reload + should use the same prohibited default targets. Reserved/control targets such + as `6`, `7`, and AllStar control target `8` should not create an active + default reflector at startup. `DEFAULT_DIAL_TS1` and `DEFAULT_DIAL_TS2` are + the canonical per-slot defaults; deprecated `DEFAULT_REFLECTOR`, `DIAL` and + `StartRef` remain TS2 compatibility aliases. The FreeDMR policy cap should + match RF dial-a-TG handling: `999999` is valid and higher defaults are + rejected. Invalid default options should disable any existing default for the + current session rather than preserving stale TG9 reflector state. Invalid + startup defaults should be logged and should not create bridge state; the + in-memory effective default should normalize to `0` without writing back to + the config file. System-wide defaults are intended for sparing use; client + requested settings are preferential. - Static TG configuration bugs: startup and live options reload should reject prohibited local/control TGs consistently on both TS1 and TS2 after parsing the configured TG strings to integers. Invalid IDs at or above `16777215` should @@ -146,9 +150,9 @@ for rule timeout checks without sleeping. - Client options parsing bugs: malformed independent numeric fields such as `IDENTTG=A`, `VOICE=A`, or `SINGLE=A` should not abort otherwise valid session options in the same string. `VOICE` and `SINGLE` accept only `0` or `1`. - Empty `DIAL` / `DEFAULT_REFLECTOR` is equivalent to `0` and means no default - reflector. Invalid `TIMER` values should be logged and should not block valid - static TG changes, which should use the current effective timer. + Empty `DIAL` / `DEFAULT_REFLECTOR` is equivalent to `0` and means no TS2 + default reflector. Invalid `TIMER` values should be logged and should not + block valid static TG changes, which should use the current effective timer. - Voice ident override bugs: `OVERRIDE_IDENT_TG` should be parsed before packet generation. Valid positive TGs below all-call are used as the destination; empty or false override values use all-call; malformed, out-of-range, or @@ -197,7 +201,12 @@ for rule timeout checks without sleeping. data-only state. - Voice LC rewrite boundary bugs: embedded-LC rewrite should apply only to voice bursts B-E and must not mutate data-sync/control payload bytes while forwarding - over HBP or FBP paths. + over HBP or FBP paths. Same-TG voice forwarding should preserve burst payload + bytes so embedded Talker Alias/GPS-like LC can traverse; target-TG mapped + forwarding should still regenerate embedded LC for the rewritten TG. +- Embedded-LC observability bugs: accepted in-call Talker Alias and GPS LC cycles + should be decoded with the standalone MMDVMHost-style embedded-LC codec and + logged without changing packet routing or mutation behaviour. - HBP group/VCSBK rate-control bugs: same-timestamp packet bursts should not raise during local packet-rate calculation before duplicate/drop handling can run. @@ -334,10 +343,11 @@ Implemented: and alias downloads disabled. The generated config supports scenario-level knobs for global ACL fields, static TG lists and optional FBP peers. - Black-box HBP coverage for static routing, global ACL startup parsing, - data/control payload preservation, sequence wrap, duplicate sequence `0` - suppression, terminator lifecycle suppression, recorded fixture replay, burst - loss and duplicate UDP profiles, and local generated prompt output for - dial-a-TG reserved controls. + data/control payload preservation, same-TG voice embedded-LC payload + preservation, in-call Talker Alias/GPS log observability, sequence wrap, + duplicate sequence `0` suppression, terminator lifecycle suppression, recorded + fixture replay, burst loss and duplicate UDP profiles, and local generated + prompt output for dial-a-TG reserved controls. - Black-box FBP coverage for enhanced keepalive/version setup, static TG routing from HBP to FBP and from FBP to HBP, BCKA gating of enhanced HBP-to-FBP forwarding, BCSQ source-quench suppression, invalid BCSQ rejection, BCST STUN @@ -476,6 +486,22 @@ fixture readers once recorded fixtures are added: destination TG or unit ID, stream ID, sequence, frame type, call type, dtype/voice sequence, payload bytes and optional frame delay. - Synthetic fixtures build canonical `DMRD` payload bytes from `PacketSpec`. +- `freedmr_dmr_codec.py` is a standalone codec proving ground for protocol + helpers before they are linked into packet routing. Its first scope is + MMDVMHost-style embedded LC encode/decode with Hamming(16,11,4), column parity + and 5-bit checksum validation. It now also covers full LC header/terminator + BPTC generation, RS(12,9) LC parity masks, a small LC classifier and + Golay(20,8,7) slot type encode/decode correction. TA/GPS logging fixtures are + generated through this module so the logging path sees valid embedded-LC + cycles rather than hand-coded bit slices. The module also exposes + legacy-compatible LC generation function names used by + `bridge_master.py`, `bridge.py` and `mk_voice.py`, plus compatible + `voice_head_term()` and `voice()` decode helpers used by `bridge_master.py` + and `bridge.py`. Synthetic group voice LC fallback is generated through this + module with normal service options (`0x00`); decoded inbound LC bytes remain + the source of truth when a real voice header is available. Active runtime + byte/alias helpers are FreeDMR-owned in `utils.py`; remaining `dmr_utils3` + imports are limited to legacy/lab tools unless those tools are updated later. - Recorded fixture support is not implemented yet. When added, fixtures should keep raw bytes plus sidecar metadata describing expected decoded fields and allowed rewrite regions. @@ -516,11 +542,13 @@ Assertions should be grouped by intent: test claiming login, HMAC, UDP parsing or real cadence coverage belongs in the UDP layer. - Minimal synthetic voice payloads may not be sufficient for scenarios that - assert full LC encoding. Recorded fixtures or carefully generated payloads - should be used for those cases. + assert full packet audio behaviour. Full LC and slot type codec behaviour is + covered in the standalone codec layer, while recorded fixtures or carefully + generated payloads should still be used for packet-path scenarios. - Embedded LC can carry information such as embedded GPS and talker alias. The - current harness protects against accidental mutation of data/control packets, - but it does not yet verify future source-to-destination embedded-LC carry-over. + current harness protects same-TG carry-over and standalone codec behavior, but + does not yet verify selective embedded-LC rewrite/preservation on TG-mapped + streams. - Generated prompt interruption is covered in both layers for state and UDP-visible routing. The harness still does not prove RF-side audio behavior or how a physical repeater/radio reacts to an abandoned prompt without a @@ -539,8 +567,9 @@ Assertions should be grouped by intent: - Deterministic cross-slot routing tests verify that TS1-to-TS2 routing rewrites only the slot bit while preserving source ID, destination TG, peer ID, stream ID and packet bytes outside the expected header bit. -- Deterministic dial-a-TG tests verify that TS1 private calls create, retune, - disconnect and query TS2 reflector state without emitting network traffic. +- Deterministic dial-a-TG tests verify that private-call control is slot-local: + TS1 controls TS1 reflector state, TS2 controls TS2 reflector state, and TS1 no + longer retunes TS2. - Deterministic generated-prompt tests verify first-packet prompt state, prompt cancellation when real HBP voice wins the same slot, and embedded-LC rewrite for late entry after cancellation. @@ -556,20 +585,22 @@ Assertions should be grouped by intent: - Deterministic reserved control-range tests verify that TS1 private calls to `4001..4999` do not create or retune reflector bridges and report busy rather than linked. -- Deterministic echo-target tests verify that TS1 private call `9990` creates - and activates the TS2 reflector state and announces a link. +- Deterministic echo-target tests verify that TS1 private call `9990` is an + intentional linkable echo/test target and announces a link without creating an + FBP route target. - Deterministic information-service tests verify that `9991..9999` schedules both the requested AMBE file and the generic silence prompt without creating a reflector. - Deterministic policy-range tests verify that `999999` is still linkable while `1000000` does not create or retune reflector state and reports busy rather than linked. -- Deterministic default-reflector tests verify that startup rejects reserved - control targets `6`, `7`, and `8`, while still allowing a linkable default - reflector target to create an active TS2 reflector. They also verify - `999999` remains valid and startup/options reject default reflector targets - above that policy cap, with invalid options disabling any active default - reflector state and invalid startup defaults producing a warning while +- Deterministic default-dial tests verify that startup rejects reserved control + targets `6`, `7`, and `8`, while still allowing linkable + `DEFAULT_DIAL_TS1` and `DEFAULT_DIAL_TS2` targets to create active per-slot + reflectors. Deprecated `DEFAULT_REFLECTOR`, `DIAL` and `StartRef` remain TS2 + aliases. These tests also verify `999999` remains valid and startup/options + reject targets above that policy cap, with invalid options disabling active + default state and invalid startup defaults producing a warning while normalizing runtime state to `0`. - Deterministic static-TG configuration tests verify that startup rejects prohibited TS1 and TS2 static TGs after integer parsing, rejects invalid IDs @@ -578,9 +609,10 @@ Assertions should be grouped by intent: prohibition. They also verify whitespace normalization and token-level skipping of invalid static TG tokens while valid tokens still apply. - Deterministic options parser tests verify malformed independent numeric fields - do not block valid DIAL/static fields, boolean-like options reject values other - than `0` or `1`, empty `DIAL` disables default reflector state, and invalid - `TIMER` values are logged without blocking valid static TG changes. + do not block valid default-dial/static fields, boolean-like options reject + values other than `0` or `1`, empty `DIAL` disables TS2 default reflector + state, and invalid `TIMER` values are logged without blocking valid static TG + changes. - Deterministic voice-ident tests verify override destination selection for valid string TGs, empty/false overrides, malformed values, control TGs, and all-call. diff --git a/docs/testing.md b/docs/testing.md index e779e6f..69e86f5 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -32,8 +32,31 @@ the factory-held database object with parameterized inserts. It also covers the hotspot proxy environment boolean parser so Docker settings such as `FDPROXY_IPV6=0` disable the feature. +Standalone DMR codec coverage verifies the new `freedmr_dmr_codec.py` helpers +before they are wired into packet forwarding. These tests cover MMDVMHost-style +embedded LC encode/decode, Hamming(16,11,4) single-bit correction, +uncorrectable error rejection, checksum-checked round trips, DMR payload +embedded-LC slice helpers, full LC header/terminator BPTC generation, RS(12,9) +LC parity masks, group/unit LC classification, and Golay(20,8,7) slot type +encode/decode correction. Synthetic group voice LC generation defaults to +normal service options (`0x00`) and keeps the HBLink `0x20` value only as an +explicit legacy constant. The full LC, routing-style embedded LC, voice +header/terminator and voice burst fixtures are fixed byte/bit vectors captured +from known-good behaviour, so these tests no longer need `dmr_utils3`. +Runtime LC generation and the voice header/terminator and burst decode helpers +used by `bridge_master.py` and `bridge.py` now use compatibility functions in +`freedmr_dmr_codec.py`. Active runtime helper functions such as `bytes_3()`, +`bytes_4()`, `int_id()` and `get_alias()` are provided by FreeDMR `utils.py`. + The deterministic suite includes dial-a-TG coverage. It verifies that private -calls from TS1 can create, retune, disconnect and query the TS2 reflector state. +calls from TS1 create, retune, disconnect and query TS1 reflector state, while +private calls from TS2 control TS2 reflector state. TS1 no longer controls TS2. +Default dial-a-TG startup and OPTIONS handling is covered through canonical +per-slot `DEFAULT_DIAL_TS1` and `DEFAULT_DIAL_TS2`; deprecated +`DEFAULT_REFLECTOR`, `DIAL` and `StartRef` remain TS2 compatibility aliases. +It also covers late-entry synthetic LC fallback: streams without a decodable +voice header fabricate normal group voice LC bytes, while streams with a real +voice header preserve the decoded LC service-options byte unchanged. It verifies these state changes are scoped to the receiving master system. Status query `5000` reports one active reflector and does not repair stale multi-active state. @@ -46,20 +69,20 @@ are covered as scheduling both the requested AMBE file and the generic silence prompt without creating reflector state. The current FreeDMR dial-a-TG policy cap is covered: `999999` remains linkable and higher targets report busy rather than linked. Private dial-a-TG timeout coverage verifies private unit calls do -not emit unmatched `GROUP VOICE,END,RX` lifecycle events. Startup default-reflector handling is covered so reserved/control +not emit unmatched `GROUP VOICE,END,RX` lifecycle events. Startup default-dial handling is covered so reserved/control targets `6`, `7`, and `8` are rejected, `999999` is accepted, and higher targets -are rejected. Invalid default-reflector options disable the effective TS2 TG9 +are rejected. Invalid default-dial options disable the effective slot TG9 default reflector for the session; invalid startup defaults are logged and do not create bridge state, with the in-memory effective default normalized to `0`. -A linkable target can still create an active TS2 default reflector. Static TG startup and options handling is +A linkable target can still create an active per-slot default reflector. Static TG startup and options handling is covered so prohibited local/control TGs are rejected consistently on both TS1 and TS2, and invalid IDs at or above `16777215` are rejected while linkable static TGs are still created. Static TG option parsing also covers simple whitespace normalization and token-level skipping of invalid tokens so valid tokens in the same TS1 or TS2 list still apply. Client options parsing covers -malformed independent numeric fields so valid DIAL/static fields still apply, +malformed independent numeric fields so valid default-dial/static fields still apply, `VOICE` and `SINGLE` accept only `0`/`1`, and empty `DIAL` disables the default -reflector. Invalid `TIMER` values are logged and static TG changes continue +TS2 reflector. Invalid `TIMER` values are logged and static TG changes continue using the current effective timer. Voice ident override coverage verifies valid override TGs are used, empty/false overrides use all-call, and malformed, control, or all-call override values are logged and fall back to all-call. @@ -108,6 +131,12 @@ handling can run. Data-sync control payload preservation is covered across HBP-to-HBP, HBP-to-FBP, FBP-to-HBP and FBP-to-FBP forwarding so voice embedded-LC rewrite does not mutate VCSBK/control payload bytes. +Voice embedded-LC preservation is covered across HBP-to-HBP, HBP-to-FBP, +FBP-to-HBP and FBP-to-FBP same-TG forwarding. Those tests assert voice bursts +B-E keep their DMR payload bytes unless the target TG is intentionally rewritten. +Embedded-LC observability coverage verifies accepted in-call Talker Alias and +GPS LC cycles are decoded through the standalone validated codec to system log +messages without changing packet routing or mutation behaviour. OBP group voice rate-drop coverage verifies per-stream packet-rate protection is calculated from elapsed stream duration rather than the absolute stream start timestamp. OBP voice lifecycle coverage verifies a voice terminator marks the @@ -163,12 +192,17 @@ behavior, and `rule_timer_loop()` clears disconnected FBP-only route targets. The UDP harness starts `bridge_master.py` as a subprocess with a generated loopback-only test config. It emulates HBP repeaters and FBP/OpenBridge peer servers over UDP, performs HBP login, sends signed packets/control messages, and -captures outbound UDP packets. +captures outbound UDP packets. The HBP login helper retries the initial login +challenge for a short startup window because `bridge_master.py` can spend time +loading aliases, keys and voice assets before Twisted has bound the loopback UDP +sockets. Current UDP scenarios cover HBP registration/config handshake, static TG routing, global `USE_ACL: False` startup parsing observed through packet -admission, data-sync/control payload preservation, modulo-256 voice sequence -wrap, sequence `0` duplicate suppression, voice terminator suppression of late +admission, data-sync/control payload preservation, same-TG voice embedded-LC +payload preservation, in-call Talker Alias/GPS log observability, +modulo-256 voice sequence wrap, sequence `0` duplicate suppression, voice +terminator suppression of late same-stream packets, recorded HBP fixture replay, and a dial-a-TG reserved control private call that emits a local TG9 TS2 prompt without leaking traffic to another master. They now also cover FBP v5 static TG routing in both