#82 logged the RX debug line once per stream_id per bridge, but a bridge
with BOTH_SLOTS/several concurrent talkgroups interleaves packets from
multiple calls, so the "last seen" stream flips almost every packet and
the line logs nearly as often as before the fix — confirmed against
production output (three interleaved streams on OBP-CL).
DMRD/DMRE demux by NETWORK_ID, which is never ambiguous, and
*CALL START*/*CALL END* (application/routing_use_cases.py,
application/routing/obp_forward.py) already log once per real call with
better detail (SUB, TGID, TS, duration). Drop the fan-in RX line for voice
frames entirely instead of trying to approximate "once per call" with no
session state; control frames (BCKA/BCSQ/BCST/BCVE) keep logging every
time, since they are rare and that visibility mattered for the #79 fix.
An active call sends a DMRD/DMRE packet roughly every 90-100ms; with
OBP_PROXY.DEBUG on, that logged one RX line per packet, flooding the
log for the whole call. Log once when a stream_id first appears instead.
Non-voice frames (control) are unaffected, they were never the flood.
Two things a real capture showed that synthetic frames could not. An
unfiltered tcpdump carries both directions, and our own egress was being run
through the ingress rules, where it fails the NETWORK_ID check by definition:
10698 frames on one bridge reported as network-id-mismatch that were simply
ours. Direction is now decided by the peer address (both ends of a link
normally share the port number, so the port alone cannot tell), with the
listening port as a fallback for a peer behind NAT; --replay-both-directions
keeps the old behaviour.
And VALIDATE_SERVER_IDS was firing offline against an empty list, because the
server-id table is loaded at runtime and not from the YAML: 10107 frames on
another bridge blamed on source-server-unknown. Offline the check is skipped
unless the table is actually there.
Both were reported against the live ADN 213 master (5 bridges, 30669 real
datagrams in 120s); with them fixed the report says what the master does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The engine from phase 3 takes plain values and answers with effects, so it does
not need the server to run. ``adn-server --replay capture.pcap`` uses that: the
frames from a tcpdump capture go through the same ingress, against the
operator's own adn-server.yaml, and each one comes back with the bridge it
belongs to and either a delivery or the reason it was refused.
12:04:31 82.65.127.86:62201 OBP-FR DMRD v1 2130001 -> 214 delivered
12:04:31 85.241.222.7:62268 OBP-PT DMRE v5 2680015 -> 9 dropped (tg-filter-server) +BCSQ
12:04:32 203.0.113.9:50000 - DMRD v1 unmatched
Nothing is sent and no port is bound, so it runs beside a live master. Which
bridge a frame belongs to is decided on the evidence the server has — the port
it arrived on, then the configured peer, then whoever can verify it — which is
also the answer to "whose keepalive is this?" on a mesh where every bridge
shares one passphrase.
``--system`` narrows it to one link, ``--replay-limit`` stops early and
``--replay-summary`` prints the tally alone.
``infrastructure/pcap.py`` reads classic pcap with no dependencies: both
endiannesses, microsecond and nanosecond timestamps, Ethernet (VLAN tags
included), Linux cooked v1 and v2 (``tcpdump -i any`` writes SLL2, found while
running this against a real capture), raw IP and loopback, IPv4 and IPv6 UDP.
pcapng says which command converts it.
Docs: the OBP proxy page gains a "why did that call not cross" section in both
languages, and its RELAX_CHECKS note now says what phase 2 made true — what the
wire teaches lives in the session, TARGET_IP stays as written.
Tests: 27 new, 97% of the replay module and 87% of the pcap reader, plus an
end-to-end run of the real command against a real YAML. Full suite 1025 passed,
2 skipped (the 2 failures are this machine's and fail on develop too).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 3, and the end of the hblink shape in this path. Phase 1 lifted the
admission rules out of the adapter, phase 2 took the state they read; what was
left in ``_obp_datagram_received`` was the plumbing between them — two long
branches that decoded, decided, logged, quenched and called into routing, all
inside a Twisted ``DatagramProtocol`` where none of it could be run on its own.
``domain/mesh_engine.py`` now takes a decoded frame, the link's session and a
``BridgePolicy``, and answers with a list of effects: ``Reject``, ``Log``,
``NoteStream``, ``StoreTalkerAlias``, ``Deliver``, ``RequestVersion``. It reads
no configuration, opens no socket and calls no logger. The adapter keeps the
three things that are genuinely I/O — verify the MAC, build the policy, carry
out the effects in order — and the handler goes from ~180 lines of nested
branches to ~45 of dispatch.
Two things this buys beyond the shape. A datagram can be replayed through the
engine anywhere: a test, a laptop, a capture from a sysop, with no reactor in
sight. And every refusal carries a ``reason``, now tallied per bridge in the
session and printed by the keepalive loop at debug level — "my call does not
cross" is answered by ``tg-filter-mcc=12, system-sub-acl=3`` instead of by
grepping the log.
No behaviour change intended, and this time checked two ways. The differential
harness ran 9594 frames through this tree and through upstream ``develop``:
identical delivery, quench, egress address and log lines. And the corpus in
``tests/fixtures/obp_ingress_effects.jsonl`` — 96 recorded cases covering the
talkgroup filters, both ACL scopes, the bits byte, the DMRE envelope (age,
hops, source server) and BCKA/BCSQ/BCST from three addresses — was recorded
from the pre-engine handler, verified frame by frame against develop, and still
passes untouched. It stays as the contract for whatever comes next; regenerate
with CAPTURE=1 and read the diff.
Tests: 24 new unit tests for the engine (100% of the module), the recorded
corpus as a regression net, 998 passed, 2 skipped (the 2 failures are this
machine's and fail on develop too).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 2 of separating the OpenBridge path from its hblink ancestry. Phase 1
lifted the admission rules out of the adapter; this one takes the state they
were reading.
Legacy hblink kept what a bridge learns at runtime inside its own SYSTEMS
block: ``_bcka`` (last keepalive), ``_bcsq`` (the peer's quench table),
``_STUN`` and, worst of the four, ``TARGET_IP``/``TARGET_PORT``/``TARGET_SOCK``,
rewritten in place every time RELAX_CHECKS accepted a datagram from an address
the operator never wrote. Configuration and session state shared one mutable
dict, so a single unexpected packet could move a bridge's target for good, and
no reader could tell what came from the YAML and what came from the wire.
``domain/mesh_session.py`` now holds an ``ObpBridgeSession`` per link:
configured_peer (from the YAML, never moves) beside learned_peer (from the
wire), the last keepalive, the quench table and the BCST stun flag, each behind
the question a caller actually asks — ``peer``, ``keepalive_seen``,
``keepalive_stale(now)``, ``quenches(tg, stream)``. The store lives under a
private top-level config key next to ``_SUB_MAP`` and ``_PEER_IDS``, so every
layer that already receives the config reaches the same instance; like
``_SUB_MAP`` it is shared, not deep-copied, across a SIGHUP, and
``sync(config)`` then refreshes the configured peers and drops the sessions of
links that are gone. Editing TARGET_IP in the YAML and reloading is now a
documented way to undo a bad learned address.
Migrated readers: the keepalive gate in routing (to_target and unit data), the
60s keepalive report loop, the monitor/MQTT dashboard blocks and the quench
check and purge in the routing timers. The SYSTEMS blocks of an OPENBRIDGE
system are no longer written to at runtime.
Also removed: ``_config.pop("_no_target_log_time")``, a key nothing has written
since the port from hblink.
No behaviour change intended. The differential harness from phase 1, extended
to compare where egress actually goes (a keepalive and a voice frame sent after
every case) and to cover BCKA/BCSQ/BCST from three source addresses with
RELAX_CHECKS on and off, ran 9594 frames through this commit and through
develop: identical delivery, quench, egress address and log lines.
Tests: 24 new unit tests for the session, 98% coverage of the new module; the
RELAX_CHECKS sync test now asserts what the refactor is for — the session
follows the peer, the configured address stays put. Full suite 876 passed,
2 skipped (the 2 failures are this machine's, and fail on develop too).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review follow-up on the fan-in demux fixes.
1. apply_obp_proxy_config_reload() rebuilds the bridge registry, and a rebuild
creates fresh ObpIngressReplyTransport objects. Nothing re-pinned them to the
legacy sockets that stay bound, so the first `systemctl reload` put egress
back on the shared fan-in port and reintroduced the bug the pin fixes.
Bound legacy transports are now kept in the service state and re-pinned after
every registry rebuild (pin_legacy_egress()). For the same reason the rebuild
now describes the ports that are actually bound (the running LISTEN_PORT /
BIND_LEGACY_PORTS) instead of the incoming ones, which are only logged as
"restart required".
2. Control-frame demux ranked bridges by the live SYSTEMS.<name>["TARGET_SOCK"],
the very dict RELAX_CHECKS rewrites in place when a bridge accepts traffic
from an unexpected source. One misattributed frame could therefore move a
bridge's target and then keep matching it there. Each entry now carries
peer_hint, the peer as configured, snapshotted when the registry is built
(startup and every reload, both of which read a freshly normalized config).
Order is: configured IP:port, live IP:port, configured IP, live IP, then
registration order — so a peer that legitimately moved is still recognised,
but never at the expense of the bridge that has that address in the YAML.
Tests
- test_reload_keeps_egress_pinned_to_the_bridge_socket: starts the service on a
fake reactor, reloads twice, asserts egress still leaves through each bridge's
own socket and nothing goes out of the fan-in port.
- test_control_frame_prefers_configured_peer_over_relaxed_target: a bridge
dragged onto another bridge's address does not steal that peer's keepalives.
- test_control_frame_unknown_peer_falls_back_to_passphrase now asserts the
deterministic outcome (first registered bridge) instead of "either one".
Both new tests fail without this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both listeners bind independent UDP sockets; if both are configured to
the same port only the first one to start would succeed, and the
server would fail at bind time with an unclear error instead of a
config validation message.
Three issues seen on a production ADN Systems master (2.5.6) with five
OpenBridge links, all rooted in the shared fan-in port.
1. YamlConfigLoader rebuilt the config from a fixed list of top-level keys
that did not include OBP_PROXY, so the whole block in adn-server.yaml was
silently ignored: ENABLED, LISTEN_PORT, LISTEN_IP and BIND_LEGACY_PORTS
always fell back to their defaults (--doctor reported 62032 for a config
that asked for 62031, and ENABLED: false still bound the fan-in).
2. Control frames (BCKA/BCSQ/BCST/BCVE) carry no NETWORK_ID, so the fan-in
could only tell bridges apart by passphrase. Every ADN Systems bridge
shares one passphrase, so all of them resolved to whichever bridge was
registered first: keepalives from the Spanish and Portuguese peers were
attributed to the French bridge and, with RELAX_CHECKS, rewrote its
TARGET_SOCK ("Source IP and Port has changed ... updating", in a loop).
Voice was unaffected because DMRD/DMRE do carry NETWORK_ID.
Control frames now prefer the bridge whose configured peer matches the
datagram source (exact IP:port, then same IP, then the passphrase scan).
3. ObpIngressReplyTransport fell back to the shared fan-in socket, so egress
left from LISTEN_PORT; remote peers learn that source and answer there,
funnelling everything onto the one port where control frames cannot be
told apart. Egress is now pinned to the bridge's own legacy socket when
one is bound.
Tests: five regression tests; all five fail without these changes.
TG/ID 4000 is the dynamic-TG reset control code and can never appear in
_SUB_MAP, so the unknown-destination fallback in _pvt_repeat_targets treated
it as a private call to an unlocated radio and blasted it to every connected
peer. That fallback runs before dmrd_received's own dst_id == 4000 guard, so
the control code has to be filtered at the targeting step too.
The periodic security download ran blocking HTTP on the reactor thread, so a
dead selfcare server stalled RPTPING handling well past PING_TIME * MAX_MISSED
and every peer was timed out and forced to reconnect. Move the security and
alias downloads to the thread pool, bound the DNS and HTTP timeouts below that
budget, and skip a cycle when the previous one is still in flight.
Also drop the redundant interval guard that silently stretched the real retry
period to a multiple of the loop interval, restore the process-wide socket
timeout after a failed DNS lookup, keep existing files and cached passwords
when a download returns an empty or invalid payload, and remove the inherited
50Mb cap on subscriber_ids that production was already close to tripping.
PASS_SECURITY is no longer written to the log.
Adds self-echo (a hotspot with the same TG on both slots, static or dynamic, hears its own TX on the other slot) and fixes four occurrences of the same slot-resolution bug that blocked or truncated it: peer_downlink_voice_slot always preferred an unambiguous static/SINGLE=1-locked slot over the caller's own wire slot, corrupting the busy-check and the parrot/echo anti-loopback guard alike. Also makes the "Downlink dropped" log reason specific instead of generic, and throttles repeated identical drop logs.
* fix: deliver TG on both slots when static on one, dynamic on the other
register_peer_ua_multi_tg silently dropped dynamic tracking on a slot
whenever the same TG was already static on the other slot, and the
REPEAT path never expanded to both slots at all.
* fix: strip NUL-padded CALLSIGN instead of showing raw bytes
Some peers NUL-pad instead of space-pad; reuse the existing
normalize_fixed_width_ascii helper instead of a plain .strip().
Use SUB_MAP's stored peer id for delivery (repeat, unit-data, and
pvt_call_received), add Talker Alias support for private calls, and
report the receiving hotspot to the monitor.
* fix: accept NUL-padded RPTC callsigns in login check
Fixed-width RPTC fields may be padded with NUL (ipsc2hbp) or spaces
(MMDVM). str.rstrip() left trailing NULs so matching DB callsigns were
rejected with MSTNAK after a successful passphrase exchange.
* fix: strip NUL padding from RPTO OPTIONS in logs and storage
ipsc2hbp pads the fixed-width RPTO body with NULs; logs showed ^@ noise
and peer CALLSIGN as raw bytes on the options line. Normalize on ingest
and in redact_pass_in_options.
* fix: strip NUL padding from monitor export and shared HBP fields
Centralize fixed-width NUL/space trimming in domain helpers and use them
for dashboard_state peer fields, OPTIONS parsing, proxy self-service, and
remaining CALLSIGN log lines.
Route announcements/TTS through synthetic PTT on the proxy MASTER (SERVER_ID
peer, normal dmrd_received forwarding). Emit START/END TX report events for
inject so monitor fans out to SYSTEM-N; configurable server voice DMR_ID.
* feat: poll peer_dynamic_tgs.need_reload for dynamic TG purge
Proxy send_opts and fallback loop apply TG-4000-equivalent reset when the
monitor sets need_reload, with migration 006 and restore filter updates.
* chore: fix import order in voice subscription plan test
* fix: audit wave 1 server hygiene refactors
Inject call_later into PlaybackUseCases, move voice config mtime watch to
bootstrap, complete SubscriptionStore port methods, and relocate echo
routing seed to the application layer.
* fix: document dashboard_state in report-v2 schema
Add dashboard_state to report-v2.json with a two-master example fixture,
update public protocol docs for HELLO → STATE_SND connect flow, and drop
the unused TOPOLOGY_JSON HELLO feature token.
* fix: add ReportWire contract tests against report-v2 schema
Assert state_frames and bridge_event_frames output validates against
committed example fixtures and the report-v2 JSON schema.
Fix test imports for echo_seed module relocation.
* fix: align OPTIONS static validity checks across routing and report
Delegate subscription_table validation to peer_options_static_valid so empty
OPTIONS is valid and PASS-mixed strings are rejected consistently.
* fix: OBP DMRE source-server validation without ALLOW_UNREG_ID bypass
Port OPENBRIDGE.validate_id lookup for 6-7 digit source servers so OBP
ingress matches legacy production config (VALIDATE_SERVER_IDS=True).
* fix: audit items 11-13 coverage, infra tests, and warning logs
* fix: add tests/fakes shim for application test decoupling
* fix: per-stream OBP bridge TX legs for concurrent MASTER downlink
When two OBP voice streams share the same MASTER timeslot, stop
flip-flopping the flat TX row so per-peer downlink gates stay stable.
* fix: ruff lint in OBP concurrent streams downlink test
Remove dead code after return and unused start_tx_events variable.
* fix: raise UDP SO_RCVBUF on voice listeners (C-LOCAL)
Apply a 4 MB receive buffer on system and proxy UDP sockets to reduce
kernel RcvbufErrors under OBP load; size is configurable via GLOBAL.UDP_RCVBUF.
* fix: exclude byte-identical duplicates from HBP rate counter (A.2)
Check lastData before incrementing the ingress packet counter so compressed
duplicate bursts do not trigger legitimate RATE DROP on call start.
* fix: duplicate-safe TA embed phase and REPEAT VHEAD DMRA (B)
Ignore byte-identical B-E embed bursts so duplicate uplinks do not desync the
TA phase machine, and re-emit DMRA on every VHEAD on the REPEAT path.
* chore: document echo point-to-point and logged_in reconciliation (EN/ES)
Document multi-hotspot echo/service delivery via RX_PEER, the lst_seen
logged_in reconcile loop, and cross-links between user and dev guides.
Echo (TG 9990-9999) must be point-to-point: only the originating hotspot
receives the playback. Two bugs in the inject-only proxy broke this:
Data plane (udp_hbp.py): _peer_should_receive_dmrd had a fuzzy-match
fallback (peer_id // 100 == rf_src) for special TGs that leaked the
first VHEAD to every hotspot sharing the user's DMR-id prefix. Removed
the fallback so echo delivers only to the exact RX_PEER.
Report plane (monitor_topology.py): _echo_tx_target_peer resolved the
monitor chip peer via fuzzy rf_src matching, which picked the wrong
hotspot when a user has several radios sharing a base id (e.g. rf_src
7140023 matched peer 714002301 instead of the real originator
714000103). Now resolves from STATUS[slot].RX_PEER when available.
After a server restart, Clients.logged_in stayed 1 for peers no longer
connected, letting the monitor authenticate disconnected hotspots via
login-by-ip. Replace the hourly clean_tbl (24h idle sweep) with a
reconcile in the existing 120s lst_seen loop: connected peers get
logged_in=1, the rest 0. The lst_seen loop now starts with now=True so
its first tick at boot clears all stale flags immediately.
* fix: add shared PASS= redaction for OPTIONS log lines
Mask PASS= secrets in RPTO/OPTIONS logging to avoid leaking hotspot
passwords in log files.
* fix: rework self-service RPTO to distinguish PASS, empty, and OPTIONS
Three RPTO cases now handled explicitly:
- PASS=password: store hash, fetch OPTIONS from DB (unchanged)
- Empty payload: DB is the authority, fetch OPTIONS from DB
- OPTIONS with content: hotspot is the authority, pass through to master
When the hotspot does not send PASS, the stored password is cleared
(NULL) so only IP auto-login works — password login is impossible
with a NULL hash, matching the legacy dashboard proxy behaviour.
Also removes the noisy DEBUG log on RPTC ("not in mysql_option_peers")
that fired on every normal login, and adds clear_psswd action to the
MySQL repository for explicit NULL writes.
* fix: elevate proxy attach rejection to WARNING for RPTC/RPTO
Control commands (RPTC/RPTO) rejected by attach should be visible at
INFO level, not hidden behind a debug flag, since they indicate a peer
sending configuration before the login exchange completed.
* fix: remove noisy CALL RX log on OBP DMRE voice header
The log fired on every DMRE VHEAD packet from each OBP peer, producing
duplicate entries for the same stream (multiple OBP peers relay the
same call). The existing CALL RX logs for DMRD and OBP v0/v1 paths
already cover stream start diagnostics.
* fix: fetch DB options when hotspot sends no RPTO after RPTC
Some hotspots complete the RPTC (repeater configuration) step but never
send an RPTO packet, leaving the peer without any OPTIONS applied. The
legacy proxy used a 10s timer after RPTC to fetch OPTIONS from MySQL in
that case; restore that behaviour here.
When RPTC arrives and the peer is not already in _mysql_option_peers,
schedule a 10s fallback timer. If no RPTO arrives before it fires, treat
the peer the same as an empty RPTO (BD is the authority) and fetch
OPTIONS from MySQL. If an RPTO does arrive, the existing cancel paths
abort the timer.
Remove 24 functions/methods with zero production callers, including
helpers only exercised by tests. Delete the standalone pickle_legacy
module. Adapt affected tests to use production equivalents or inline
constructions. 569 tests pass.
Enforce one QSO per RF slot for normal hotspots, clear peer_voice_slots
on disconnect, allow matching VTERM through slot gates, exempt lab
witnesses with many static TGs, and log ingress TG-busy drops once per stream.
Remove inject-only bypass in _peer_should_receive_dmrd so peers with
empty OPTIONS (silent witness) never receive group DMRD. Align
peer_listen_slots for simplex with static TG lists.
Introduce downlink.py as the single authority for hotspot eligibility (OPTIONS,
slot busy, post-TX GROUP_HANGTIME). send_peer and BRDG fan-out share the same
gate; monitor events carry stream id from BRDG so new QSOs after hangtime
display without replaying calls blocked during the hangtime window.
Planned release: 2.2.0
* feat: persist peer dynamic TGs in MariaDB across reconnects
Add DATABASE config, async DynamicTgStore, and restore on RPTC so
SINGLE=0/1 dynamics survive hotspot disconnects and server restarts
without blocking the DMRD voice path.
* fix: ensure peer_dynamic_tgs table on server startup
Apply migration 004 idempotently at boot so the server does not depend
on adn-monitor db_bootstrap when peer_dynamic_tgs is missing.
* fix: import DynamicTgEntry for ruff F821 in subscription_table
* fix: log clear MariaDB startup failures to file and stderr
Validate DATABASE at config load and on connect; map common MySQL
errors to actionable messages so the server does not fail silently.
* fix: TG 4000 clears STATUS and bridge legs after dynamic reset
Clear RX slot state to stop RPTO re-seeding cleared sessions, run
in-band 4000 deactivation on inject-only paths, and mark downlink dirty.
* fix: complete TG 4000 reset for monitor and dynamic TG persistence
Emit INGRESS BRDG_EVENT so SINGLE=0 UA chips clear without stuck TX;
wipe all peer dynamic rows from memory and MariaDB on reset. Never store
TG 4000 as a UA session. Require DATABASE only for full peer-server configs.
Normalize all Python sources to the standard ADN copyright block with
complete GPLv3 notice. Add legacy attribution on routing and dmr_utils
ports; drop SemVer wording from changelog and fix an unused test import.
Narrow send_peers and REPEAT fan-out on inject-only proxies using a
precomputed OPTIONS index, cache connected peer count, and debounce
CONFIG_SND pushes to the monitor.
Replace internal bridge terminology with routing (RoutingUseCases, AclRouter,
routing_table export). SubscriptionStore remains runtime authority with O(1)
indexes for router and downlink filters. Fix STATIC TG parity on OPTIONS/RPTO,
parrot in-band edge cases, and remove per-packet routing_table export from the
hot path that caused high CPU under multi-hotspot OBP load.
Require subscription_store in BridgeUseCases and route timer, OPTIONS,
static TG, and OBP mutations through store ops with export-only BRIDGES shim.
Fix dashboard YAML static TG fallback and sole-hotspot monitor remap for
dynamic UA when a bridge leg is active.
Sync subscription store after timer and in-band mutations, add optional
store authority with BRIDGES export shim, wire MeshCodecRegistry into
OBP udp_hbp paths, and extend harness parity tests.
Push STATE_SND when peers send RPTO so monitor chips update without reload.
Fetch and inject MySQL OPTIONS immediately on PASS= instead of a 10s delay.
Track per-peer dynamic TG sets for SINGLE=0 so keyed hotspots hear each other.
Enforce strict SINGLE=1 RX exclusivity, always filter inject-only downlink by
each peer's own OPTIONS, and push self-service DB options only after PASS=.
Stop merged SYSTEM static TG lists from leaking across hotspots in topology.
Wait for stopListening before listenUDP to avoid EADDRINUSE when GENERATOR
collapses (D-APRS-0 -> D-APRS). Migrate instance-0 protocol on bind unchanged.
Reload completion runs via Deferred callback.
Rodrigo Pérez
ce5rpy
yo
github-actions[bot]