Simon
2 weeks ago
parent
d86623180a
commit
c2c7e654cb
@ -0,0 +1,660 @@
|
||||
# FreeDMR 2.0 Architecture Decisions
|
||||
|
||||
This file records architectural decisions, requirements, assumptions and open
|
||||
questions driven out during design discussion. It is intended as source material
|
||||
for a later formal FreeDMR 2.0 design document.
|
||||
|
||||
## Project Philosophy
|
||||
|
||||
FreeDMR is open-source, open, intentionally understandable and intentionally
|
||||
simple enough to encourage community implementation, experimentation and
|
||||
operation by radio amateurs.
|
||||
|
||||
HBLink proved that a DMR server could be written in an open, readable way
|
||||
without DMR being gatekept by commercial vendors. FreeDMR takes the next step:
|
||||
it proves that a DMR network can be built this way without central control.
|
||||
Before HBLink and FreeDMR, DMR server software and server-level network
|
||||
membership were typically closed, gatekept or dependent on personal/team
|
||||
approval. FreeDMR exists in part to lower that barrier and give radio amateurs
|
||||
choice and freedom to experiment with global-scale ROIP networking.
|
||||
FreeDMR does not need to gatekeep all private experimentation. The project
|
||||
controls public listing: the process by which servers are shared with Pi-Star
|
||||
and other HBP hotspots as legitimate public access servers. A sysop can run a
|
||||
private server under their own DMR ID and arrange gatewaying with an existing
|
||||
sysop, who effectively vouches for that traffic. Public listing has additional
|
||||
requirements such as connectivity quality, sysop contactability and basic
|
||||
operational expectations.
|
||||
|
||||
The FreeDMR mesh design is influenced by the late Bob Bruninga's APRS ideas,
|
||||
Spanning Tree Protocol and related distributed-network approaches. The project
|
||||
also has a social purpose: bringing together communities and people connected
|
||||
to earlier amateur-radio networking work. FreeDMR is therefore both a technical
|
||||
system and a diplomacy project; design choices must respect operational
|
||||
autonomy, interoperability and trust between independent sysops.
|
||||
|
||||
FreeDMR is successful because it works in the amateur-radio sense: it is best
|
||||
effort, experimental, approachable and deployable on ordinary low-cost systems
|
||||
such as cheap VPS instances and Raspberry Pi-class hardware. It is not intended
|
||||
to be a safety-assured commercial system. FreeDMR 2.0 should improve quality,
|
||||
clarity and scalability without losing the ham-spirit/hacker-philosophy traits
|
||||
that made the network useful and welcoming.
|
||||
|
||||
Design implications:
|
||||
|
||||
- Prefer clear, inspectable protocols over opaque mechanisms.
|
||||
- Keep the implementation understandable by competent sysops and contributors.
|
||||
- Keep the barrier to compatible implementations low where possible.
|
||||
- Preserve low-cost deployment and modest hardware requirements.
|
||||
- Avoid architectural choices that make FreeDMR dependent on heavyweight
|
||||
infrastructure for ordinary single-server operation.
|
||||
- Treat reliability as best-effort resilience appropriate to amateur radio, not
|
||||
as commercial safety assurance.
|
||||
- Preserve server autonomy and local policy.
|
||||
- Avoid unnecessary central control.
|
||||
- Distinguish private operation, vouched/gatewayed traffic and public listing.
|
||||
- Security should protect authenticity and network integrity without hiding
|
||||
amateur-radio traffic.
|
||||
|
||||
## Protected Model
|
||||
|
||||
The protected asset is the FreeDMR operating model, not the old HBLink-derived
|
||||
object structure.
|
||||
|
||||
Preserve:
|
||||
|
||||
- packet model and protocol behaviour
|
||||
- dial-a-TG semantics
|
||||
- TG/DMR-ID centric routing
|
||||
- loop control
|
||||
- source quench
|
||||
- mesh behaviour
|
||||
- practical RF/network tolerance learned from live servers and real RF links
|
||||
- "everything everywhere" principle, subject to documented exceptions
|
||||
|
||||
Replace or redesign where useful:
|
||||
|
||||
- configured `MASTER` stanza as primary runtime identity
|
||||
- proxy-mediated client fan-out
|
||||
- global mutable `BRIDGES` structure as authoritative state
|
||||
- custom dashboard/reporting socket protocol
|
||||
- packet-path coupling to dashboard/API/report consumers
|
||||
|
||||
## Layer Model
|
||||
|
||||
FreeDMR 2.0 should be described as layered:
|
||||
|
||||
- **Access layer**: client/server access protocols such as HBP today and
|
||||
possible future non-trunk client protocols. Owns login/auth/options/keepalive,
|
||||
client sessions, slot state and RF-facing TG presentation.
|
||||
- **Subscription layer**: talkgroup conference membership. Owns direct TG
|
||||
subscriptions, dial-a-TG subscriptions, static/default/user-activated
|
||||
subscriptions, expiry and RF-visible TG to conference TG mapping.
|
||||
- **Mesh layer**: inter-server FBP/OBP/trunk-style behaviour. Owns loop control,
|
||||
source quench, hop/version handling and inter-server conference traffic.
|
||||
- **Reporting layer**: local dashboard, API observers, logs, global lastheard
|
||||
export and state snapshots. Reporting is observational and must not steer
|
||||
packet handling.
|
||||
|
||||
## Reactor and Runtime Migration
|
||||
|
||||
Do not replace Twisted as part of the first FreeDMR 2.0 architecture work.
|
||||
|
||||
Decision:
|
||||
|
||||
- Keep Twisted's single-threaded reactor as a safety boundary initially.
|
||||
- Extract and test the protocol/routing/subscription core behind deterministic
|
||||
interfaces.
|
||||
- Introduce explicit process/message boundaries only after the state model is
|
||||
clear.
|
||||
- Consider asyncio or another event loop only once Twisted has become a thin
|
||||
transport shell around tested core logic.
|
||||
|
||||
Rationale:
|
||||
|
||||
- The current packet behaviour is subtle and validated through real RF/network
|
||||
deployment.
|
||||
- Replacing the event loop while also replacing the state model would mix too
|
||||
many sources of behavioural change.
|
||||
- Twisted's single-threaded reactor helps preserve current ordering assumptions
|
||||
while bridge/subscription and reporting boundaries are made explicit.
|
||||
- The first migration target is architectural clarity and scalability, not event
|
||||
loop novelty.
|
||||
|
||||
## Identity Model
|
||||
|
||||
The configured master/listener is not the client identity.
|
||||
|
||||
FreeDMR 2.0 should move toward:
|
||||
|
||||
- listener identity: UDP socket/service instance
|
||||
- client identity: DMR peer/client ID
|
||||
- subscription identity: client ID + slot + RF-visible TG + conference TG
|
||||
- mesh identity: server/peer/network ID
|
||||
|
||||
Server identity hierarchy:
|
||||
|
||||
- FreeDMR server IDs are 4-digit DMR IDs.
|
||||
- Server sub-IDs are 5-digit IDs derived from the server ID space.
|
||||
- Each sysop/server identity may therefore cover up to 10 server sub-IDs for
|
||||
backend components, larger deployments, failover or fault-tolerant layouts.
|
||||
- Identity verification should cover the base server ID and its authorized
|
||||
sub-IDs rather than requiring unrelated credentials for each sub-ID.
|
||||
|
||||
A single master/listener UDP port should serve an arbitrary number of clients
|
||||
directly, replacing the proxy where possible.
|
||||
|
||||
## Talkgroup Subscription Model
|
||||
|
||||
Conceptually, each TG is a conference bridge. Clients subscribe to conference
|
||||
TGs. FreeDMR does not primarily decide where to send user traffic; users choose
|
||||
the traffic they want to hear by subscription.
|
||||
|
||||
Subscriptions can be:
|
||||
|
||||
- direct TG: RF-visible TG equals conference TG
|
||||
- dial-a-TG: RF-visible TG is currently TG9, conference TG is the selected TG
|
||||
- alias/rewrite: RF-visible TG may be any configured TG, conference TG is the
|
||||
FreeDMR network identity
|
||||
|
||||
Example:
|
||||
|
||||
```python
|
||||
TalkgroupSubscription(
|
||||
client_id=2345001,
|
||||
slot=2,
|
||||
conference_tg=4400,
|
||||
rf_tg=9,
|
||||
mode="dial",
|
||||
active=True,
|
||||
)
|
||||
```
|
||||
|
||||
The invariant is:
|
||||
|
||||
```text
|
||||
conference_tg = FreeDMR network/conference identity
|
||||
rf_tg = client-facing RF presentation identity
|
||||
```
|
||||
|
||||
This makes arbitrary TG rewrites possible without making TG9 structurally
|
||||
special.
|
||||
|
||||
## Bridge Table Replacement
|
||||
|
||||
The legacy `BRIDGES` dict should be replaced internally by subscription-oriented
|
||||
state and indexes. The `"#"` reflector naming convention does not need to be
|
||||
preserved internally; it can be a compatibility/export detail.
|
||||
|
||||
Recommended hot-path structures:
|
||||
|
||||
- `dict` / `set` for O(1)-style local lookups
|
||||
- `typing.NamedTuple` keys for readable hash keys
|
||||
- `dataclass(slots=True)` records for mutable subscription/session state
|
||||
- `heapq` for expiry timers using lazy invalidation
|
||||
|
||||
Recommended indexes:
|
||||
|
||||
```python
|
||||
subscriptions_by_conference_tg[conference_tg] -> set[SubscriptionKey]
|
||||
subscription_by_rf[(client_id, slot, rf_tg)] -> SubscriptionKey
|
||||
subscriptions_by_client_slot[(client_id, slot)] -> set[SubscriptionKey]
|
||||
expiry_heap -> (expires_at, generation, SubscriptionKey)
|
||||
```
|
||||
|
||||
Packet handlers should not scan all subscriptions/bridges to find routing
|
||||
targets.
|
||||
|
||||
## Packet Plane vs Control Plane
|
||||
|
||||
The packet plane is delay-sensitive.
|
||||
|
||||
Packet-plane rules:
|
||||
|
||||
- local in-memory hot state only
|
||||
- no external database round trips
|
||||
- no blocking API/dashboard/report calls
|
||||
- no cross-process lock waits
|
||||
- no dependency on reporting consumers being connected
|
||||
|
||||
External stores may be used for:
|
||||
|
||||
- config distribution
|
||||
- API/dashboard state
|
||||
- control-plane coordination
|
||||
- snapshots
|
||||
- global lastheard export
|
||||
- optional clustering/multi-process coordination
|
||||
|
||||
General performance principle:
|
||||
|
||||
- Expensive processing should be considered for offload to separate processes
|
||||
because CPython execution is constrained by the GIL for CPU-bound Python code.
|
||||
- Offload is appropriate for reporting fanout, global export, dashboard
|
||||
aggregation, historical database writes, heavy analytics, expensive
|
||||
transcoding/codec experiments and non-critical maintenance jobs.
|
||||
- Offload boundaries must be asynchronous from the packet path. If an offload
|
||||
worker is slow or unavailable, packet handling must continue with local state.
|
||||
- Do not offload hot-path routing decisions if doing so would add inter-process,
|
||||
network or lock waits to every packet.
|
||||
|
||||
## DMR Data Packet Policy
|
||||
|
||||
FreeDMR must maintain DMR data packet forwarding support.
|
||||
|
||||
Decision:
|
||||
|
||||
- FreeDMR should forward supported DMR data packets according to the same
|
||||
conference/subscription and mesh principles as other traffic.
|
||||
- There must be no regression in existing data packet forwarding support.
|
||||
- FreeDMR core should not become an application-level DMR data processor.
|
||||
- GPS, SMS and similar application processing should be implemented by systems
|
||||
connected via FBP or another mesh/access-adjacent interface.
|
||||
- `DATA_GATEWAY` is understood as an earlier expression of this model: an FBP
|
||||
link that carries data-oriented traffic rather than ordinary voice traffic.
|
||||
- Existing `SUB_MAP` behaviour is intentional: data addressed to a DMR ID can be
|
||||
routed toward the last known HBP/client location for that DMR ID.
|
||||
|
||||
Core FreeDMR may inspect/classify data packets only as needed for:
|
||||
|
||||
- packet admission and protocol validation
|
||||
- routing/subscription decisions
|
||||
- loop control and source quench
|
||||
- reporting/logging
|
||||
- preserving packet bytes and metadata across FBP/HBP boundaries
|
||||
- maintaining the subscriber location map needed for data-client routing
|
||||
|
||||
Possible narrow exceptions:
|
||||
|
||||
- dial-a-TG control via DMR SMS
|
||||
- DMR SMS alerts from a server to a sysop
|
||||
|
||||
Any such exceptions must be explicit control-plane features and must not turn
|
||||
FreeDMR core into a general GPS/SMS application processor.
|
||||
|
||||
## Mesh Peer Authentication
|
||||
|
||||
FreeDMR should only accept mesh/FBP traffic from servers that can be validated
|
||||
as legitimate members of the network.
|
||||
|
||||
Core principle:
|
||||
|
||||
- FreeDMR may sign/authenticate traffic and control messages, but should not
|
||||
encrypt amateur-radio traffic or mesh traffic by default.
|
||||
- Amateur radio is public in most jurisdictions and encryption is often not
|
||||
permitted. FreeDMR users may also carry IP backhaul over amateur radio links.
|
||||
- FreeDMR's security model is authenticity, integrity, membership validation and
|
||||
local policy enforcement, not secrecy.
|
||||
- This follows the existing FreeDMR principle, agreed historically by project
|
||||
maintainers, that the network has nothing to hide and should remain cleartext.
|
||||
|
||||
Identity/listing distinction:
|
||||
|
||||
- Signed mesh identity should prove a server/sysop identity or a vouching
|
||||
relationship. It should not automatically imply public listing.
|
||||
- Public listing is a directory/discovery decision for clients and HBP hotspots.
|
||||
- A public access server may need stronger operational requirements than a
|
||||
private or gatewayed server.
|
||||
- Local sysops may still choose whether to carry/vouch for traffic from private
|
||||
servers, even when those servers are not publicly listed.
|
||||
- If an individual 7-digit DMR ID is used as a server identity, traffic may pass
|
||||
when a directly connected/listed sysop chooses to allow and gateway it.
|
||||
- The vouching sysop is accountable to their peers for traffic they forward. If
|
||||
that traffic harms the network, peers may choose to stop peering with the
|
||||
vouching server. This preserves a self-policing social mechanism without
|
||||
requiring central control for all private experimentation.
|
||||
|
||||
Analogue network bridges:
|
||||
|
||||
- Analogue ROIP/network bridges commonly connect as if they are DMR clients via
|
||||
HBP.
|
||||
- FreeDMR permits this and is generally more permissive than many other DMR
|
||||
networks.
|
||||
- FreeDMR works with/supports the DVSwitch community on this. DVSwitch provides
|
||||
a common mechanism by which analogue networks can be bridged into DMR-style
|
||||
access.
|
||||
- These bridges are operationally sensitive: technical limitations can make
|
||||
them effectively listen-only, consuming CPU and bandwidth while adding little
|
||||
value if they do not contribute actual two-way user activity.
|
||||
- Analogue bridges are often implemented using audio mixing/conference style
|
||||
behaviour. This is a poor fit for DMR and similar digital modes, which enforce
|
||||
one audio source at a time and rely on stream, hang-time and contention
|
||||
behaviour rather than mixed audio.
|
||||
- This mismatch comes partly from analogue repeater heritage: analogue systems
|
||||
may maintain a continuous transmit carrier and mix notification sounds such as
|
||||
pips, CWID and courtesy tones into the output audio. Analogue systems also
|
||||
often have little or no strong source identity, whereas DMR traffic carries a
|
||||
DMR ID.
|
||||
- A common failure mode is that a feed from an analogue repeater keeps the DMR
|
||||
stream open between analogue overs, plays courtesy/notification tones and then
|
||||
carries the next analogue user in the same held stream. This can hold the TG
|
||||
open and prevent a digital station from breaking in until the analogue
|
||||
repeater times out and its carrier drops.
|
||||
- Analogue bridges should therefore be subject to local sysop policy, public
|
||||
listing expectations and peer accountability. Permitted does not mean
|
||||
automatically valuable or immune from peering/listing consequences.
|
||||
|
||||
Other digital network bridges:
|
||||
|
||||
- Digital voice networks such as YSF and NXDN are generally a better technical
|
||||
match for DMR than analogue networks because they also use AMBE-family vocoder
|
||||
audio.
|
||||
- AMBE-to-AMBE interworking can be lossless at the codec level and avoids
|
||||
transcoding artifacts.
|
||||
- Transcoding from analogue or unlike codecs can degrade audio quality
|
||||
significantly and should be treated carefully.
|
||||
|
||||
Desired direction:
|
||||
|
||||
- Add PKI-backed mesh peer admission to the Bridge Control (`BCXX`) mechanism.
|
||||
- A peer server presents public identity material signed by a FreeDMR network
|
||||
master key or trusted network CA.
|
||||
- The authenticated identity must bind at least:
|
||||
- server ID
|
||||
- authorized server sub-IDs
|
||||
- public key
|
||||
- validity period
|
||||
- permitted protocol/features where useful
|
||||
- Runtime admission should bind the authenticated server identity to the
|
||||
observed transport endpoint, including IP address.
|
||||
- If the observed IP address changes, the FBP peer must perform a new key
|
||||
exchange/authentication step before its traffic is forwarded.
|
||||
- Network membership should be represented by a signed sysop/server key that is
|
||||
issued when the sysop/server joins the network and revoked when they leave or
|
||||
are compromised. Runtime endpoint/session bindings are renewed separately and
|
||||
do not require re-signing the long-lived membership key.
|
||||
- One successful verification of the signed identity should authorize the
|
||||
covered server ID and declared/authorized sub-IDs for that sysop, subject to
|
||||
local policy and endpoint/session binding.
|
||||
|
||||
Packet-plane rule:
|
||||
|
||||
- Expensive signature/certificate validation happens during control-plane
|
||||
admission or re-admission, not for every DMR packet.
|
||||
- Per-packet mesh traffic should use a cached authenticated peer/session state
|
||||
check keyed by server ID and endpoint.
|
||||
|
||||
Initial conceptual flow:
|
||||
|
||||
```text
|
||||
FBP peer connects/sends keepalive
|
||||
-> BC auth exchange presents signed server identity/public key
|
||||
-> FreeDMR validates signature against trusted network key
|
||||
-> FreeDMR binds server_id + endpoint + protocol features to peer session
|
||||
-> DMR traffic is accepted only while that authenticated binding is valid
|
||||
```
|
||||
|
||||
Security requirements:
|
||||
|
||||
- Reject unauthenticated FBP traffic by default once this mode is enabled.
|
||||
- Reject traffic where server ID, key identity and source endpoint do not match
|
||||
the authenticated binding.
|
||||
- Expire authenticated bindings and require renewal.
|
||||
- Support soft renewal: when an authenticated binding reaches its renewal
|
||||
timestamp, schedule asynchronous re-authentication while allowing a bounded
|
||||
grace period so in-flight voice is not interrupted purely by renewal timing.
|
||||
- Hard-stop forwarding only for explicit authentication failure, revoked
|
||||
identity/key, endpoint mismatch outside policy, expired grace period, or
|
||||
policy requiring immediate re-authentication.
|
||||
- Log authentication failure reasons clearly without leaking private material.
|
||||
- Provide a controlled transition mode for existing networks while PKI is rolled
|
||||
out.
|
||||
|
||||
Open questions:
|
||||
|
||||
- Whether to use X.509 certificates, raw Ed25519 public keys with signed
|
||||
metadata, or another compact identity format.
|
||||
- How network master keys/CAs are generated, rotated and revoked.
|
||||
- Whether peer authorization policy should live in config, MQTT/control-plane
|
||||
state, or a signed network membership list.
|
||||
- How to handle legitimate dynamic-IP servers without weakening endpoint
|
||||
binding.
|
||||
- What renewal and grace-period defaults best preserve voice continuity without
|
||||
weakening mesh admission.
|
||||
|
||||
### Distributed Key Gossip Option
|
||||
|
||||
FreeDMR may also use a peer-to-peer signed-key dissemination mechanism over the
|
||||
Bridge Control (`BCXX`) out-of-band channel.
|
||||
|
||||
Concept:
|
||||
|
||||
- Each server periodically advertises the signed server public keys/membership
|
||||
documents it knows to its direct FBP peers.
|
||||
- Peers validate the signatures and build a local table of legitimate server
|
||||
identities as knowledge propagates through the mesh.
|
||||
- Each server uses its local signed-key table and local policy to decide whether
|
||||
to route or reject packets that originated from a given source server, even
|
||||
when that source server is not directly connected.
|
||||
|
||||
Rationale:
|
||||
|
||||
- FreeDMR is a peer network, not hub-and-spoke or master/slave.
|
||||
- Servers are autonomous and independently operated.
|
||||
- Direct FBP peers should not be blindly trusted to make correct routing
|
||||
decisions on behalf of the local server.
|
||||
- Open-source, human-readable code deliberately lowers the barrier to
|
||||
modification, so each server must be able to protect itself from incorrect or
|
||||
malicious upstream forwarding decisions.
|
||||
|
||||
Security requirements for key gossip:
|
||||
|
||||
- Only signed membership documents are accepted; peers cannot create trust by
|
||||
merely repeating a key.
|
||||
- Membership documents need issuer, subject server ID, public key fingerprint,
|
||||
authorized sub-IDs, validity period, serial/version and signature.
|
||||
- Revocation data must propagate by the same or a stronger mechanism.
|
||||
- Each server must enforce local policy after validation. A valid signed key
|
||||
proves membership, not mandatory carriage.
|
||||
- Key gossip must be rate-limited and bounded so it cannot become a BCXX flood
|
||||
or memory-growth vector.
|
||||
- Received membership data must be replay-resistant enough to handle expiry,
|
||||
superseded serials and revoked keys.
|
||||
- The packet path must use cached key/policy state; signature validation and
|
||||
gossip processing are control-plane work.
|
||||
|
||||
This complements direct-peer endpoint authentication. Direct-peer auth proves
|
||||
the connected FBP peer is legitimate for this session; distributed signed-key
|
||||
knowledge lets the local server make autonomous decisions about traffic whose
|
||||
source server is elsewhere in the mesh.
|
||||
|
||||
## Reporting Protocol Decision
|
||||
|
||||
FreeDMR 2.0 should define a structured reporting event protocol and use MQTT as
|
||||
the preferred external live reporting transport.
|
||||
|
||||
Rationale:
|
||||
|
||||
- MQTT is already familiar in DMR network dashboard/reporting contexts.
|
||||
- BrandMeister uses MQTT, providing a useful precedent for dashboard consumers.
|
||||
- MQTT topics map naturally to server/client/subscription/call state.
|
||||
- Retained messages are useful for current state snapshots.
|
||||
- Last Will and Testament can represent server/reporting disconnects.
|
||||
- MQTT-over-WebSocket allows browser dashboards to subscribe directly when the
|
||||
broker supports it.
|
||||
|
||||
Constraints:
|
||||
|
||||
- MQTT publishing must be asynchronous from the packet worker.
|
||||
- Packet routing must continue if the MQTT broker/dashboard is down.
|
||||
- Event generation must be state-change/summary oriented, not per DMR frame.
|
||||
- The event schema is the compatibility contract; internal Python objects are
|
||||
not.
|
||||
- Local live dashboard and central global lastheard remain separate paths.
|
||||
- Voice stability takes precedence over reporting completeness. If the system
|
||||
must choose between dropping/reporting-losing events and delaying packet
|
||||
handling, it must drop or coalesce reporting events.
|
||||
|
||||
Implementation requirement:
|
||||
|
||||
```text
|
||||
packet path -> non-blocking local event queue -> MQTT publisher worker
|
||||
```
|
||||
|
||||
The packet path must not call an MQTT broker synchronously. The local event
|
||||
queue should be bounded. On overflow, the publisher layer should drop or
|
||||
coalesce low-priority events and emit a later reporting-health event rather than
|
||||
blocking packet handling.
|
||||
|
||||
Suggested event priority:
|
||||
|
||||
- retain/coalesce latest state: server/client/slot/subscription state
|
||||
- keep best effort: call start/end summaries
|
||||
- drop first under pressure: high-volume debug/warning/statistical updates
|
||||
|
||||
MQTT publishing should support reconnect with exponential backoff and should
|
||||
refresh retained state after reconnect so a dashboard can recover even if
|
||||
transient events were missed.
|
||||
|
||||
Suggested MQTT namespace:
|
||||
|
||||
```text
|
||||
freedmr/v2/{server_id}/state
|
||||
freedmr/v2/{server_id}/client/{client_id}/state
|
||||
freedmr/v2/{server_id}/client/{client_id}/slot/{slot}/activity
|
||||
freedmr/v2/{server_id}/subscription/{subscription_id}/state
|
||||
freedmr/v2/{server_id}/call/{stream_id}/start
|
||||
freedmr/v2/{server_id}/call/{stream_id}/end
|
||||
freedmr/v2/{server_id}/mesh/{peer_id}/state
|
||||
freedmr/v2/{server_id}/event
|
||||
```
|
||||
|
||||
Use retained messages for current state:
|
||||
|
||||
```text
|
||||
server state
|
||||
client state
|
||||
slot activity
|
||||
subscription state
|
||||
mesh peer state
|
||||
```
|
||||
|
||||
Use non-retained messages for transient events:
|
||||
|
||||
```text
|
||||
call start/end
|
||||
loop-control event
|
||||
source-quench event
|
||||
packet-rate/loss summary
|
||||
warnings
|
||||
```
|
||||
|
||||
Example event:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 2,
|
||||
"event_id": 1849281,
|
||||
"type": "call.started",
|
||||
"timestamp": 1710000000.123,
|
||||
"server_id": 234099,
|
||||
"client_id": 2345001,
|
||||
"slot": 2,
|
||||
"conference_tg": 4400,
|
||||
"rf_tg": 9,
|
||||
"source_id": 2351234,
|
||||
"stream_id": 16909060,
|
||||
"access": "hbp"
|
||||
}
|
||||
```
|
||||
|
||||
Dashboard delivery options:
|
||||
|
||||
- preferred: dashboard subscribes to MQTT over WebSocket
|
||||
- alternative: local reporting sidecar translates MQTT to SSE/HTTP
|
||||
- control actions should use authenticated HTTP APIs unless a future UI needs
|
||||
bidirectional streaming
|
||||
|
||||
## Local Dashboard vs Global Lastheard
|
||||
|
||||
Each FreeDMR server has its own local live dashboard. The global lastheard
|
||||
service is centrally hosted and non-real-time.
|
||||
|
||||
Local dashboard:
|
||||
|
||||
- consumes local MQTT live state/events
|
||||
- displays current client/repeater traffic
|
||||
- must tolerate reconnects and missed transient events by reloading retained
|
||||
state topics
|
||||
|
||||
Global lastheard:
|
||||
|
||||
- consumes call summaries or batched exports
|
||||
- should not depend on packet-plane or dashboard delivery
|
||||
- should tolerate central outage via spool/retry
|
||||
|
||||
Possible MQTT global feed:
|
||||
|
||||
- Each server publishes local live dashboard topics to a local broker or local
|
||||
reporting service.
|
||||
- Prefer a separate exporter process for the curated global feed. The exporter
|
||||
subscribes to the same local real-time MQTT feed as the dashboard, filters and
|
||||
summarizes what is needed, then publishes to the network MQTT broker or writes
|
||||
to the global collector.
|
||||
- The exporter publishes only summary topics needed for the 30-day database,
|
||||
such as call end summaries, client/server presence, selected mesh health and
|
||||
selected subscription changes.
|
||||
- Raw packet events and high-volume live slot updates should not be exported to
|
||||
the global broker by default.
|
||||
- Central broker, global dashboard or exporter failure must not back up into
|
||||
local packet processing or local dashboard state.
|
||||
|
||||
Preferred flow:
|
||||
|
||||
```text
|
||||
FreeDMR core -> local MQTT feed -> local dashboard
|
||||
-> global-exporter process -> network MQTT/collector
|
||||
```
|
||||
|
||||
Core publishing invariant:
|
||||
|
||||
- FreeDMR core emits each reporting event once to its configured local MQTT
|
||||
broker/publisher queue.
|
||||
- Fanout to dashboards, exporters, automation and global collectors is handled
|
||||
by the MQTT broker and separate subscriber processes.
|
||||
- Adding more reporting consumers must not increase FreeDMR packet-process work
|
||||
beyond the single local event emission.
|
||||
|
||||
Suggested global MQTT subjects:
|
||||
|
||||
```text
|
||||
freedmr/v2/global/{server_id}/call/end
|
||||
freedmr/v2/global/{server_id}/client/state
|
||||
freedmr/v2/global/{server_id}/server/state
|
||||
freedmr/v2/global/{server_id}/mesh/state
|
||||
```
|
||||
|
||||
## Reporting Event Types
|
||||
|
||||
Initial event families:
|
||||
|
||||
```text
|
||||
server.started
|
||||
server.stopping
|
||||
client.connected
|
||||
client.disconnected
|
||||
client.options_changed
|
||||
subscription.activated
|
||||
subscription.deactivated
|
||||
subscription.expired
|
||||
call.started
|
||||
call.ended
|
||||
call.lost
|
||||
mesh.peer_up
|
||||
mesh.peer_down
|
||||
mesh.source_quench
|
||||
loop.detected
|
||||
packet.rate_limited
|
||||
```
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which MQTT broker should be packaged by default: Mosquitto, EMQX, NATS MQTT
|
||||
compatibility, or another option?
|
||||
- Should MQTT be mandatory for FreeDMR 2.0 dashboards, or optional with an
|
||||
embedded/local fallback?
|
||||
- What authentication/authorization model should protect MQTT topics and
|
||||
dashboard control APIs?
|
||||
- What retained-topic expiry policy should be used to prevent stale state?
|
||||
- Should global lastheard consume MQTT directly or use a separate HTTP/queue
|
||||
exporter fed from reporting events?
|
||||
- Should FreeDMR expose a legacy `BRIDGES` compatibility view during migration?
|
||||
@ -0,0 +1,65 @@
|
||||
# FreeDMR 2 Glossary
|
||||
|
||||
This glossary defines FreeDMR 2 terms. Legacy terms such as `SYSTEM`, `MASTER`, `BRIDGES`, and `#` reflector names describe current implementation details or compatibility views, not the primary FreeDMR 2 model.
|
||||
|
||||
**Access layer**: The part of FreeDMR that accepts client/repeater protocols such as HBP. It owns login, authentication, options, keepalive, access sessions, RF-facing slots, and RF-visible TG presentation.
|
||||
|
||||
**Access session**: A live connection-like relationship between FreeDMR and a client/repeater. It is identified by client DMR ID and transport/session metadata, not by a configured listener stanza.
|
||||
|
||||
**Listener**: A UDP socket/service endpoint that accepts one or more access sessions. A configured `MASTER` is a legacy listener-like concept, not the client identity.
|
||||
|
||||
**Client/repeater**: An HBP hotspot, repeater, gateway, bridge, or future access-side system connected to FreeDMR.
|
||||
|
||||
**Client DMR ID**: The DMR ID used by a client/repeater access session. Future routing should key primarily on client DMR ID, slot, and RF-visible TG.
|
||||
|
||||
**RF-visible TG**: The talkgroup number seen by the RF terminal or access-side device. In dial-a-TG this may be TG9 while the network conference TG is different.
|
||||
|
||||
**Conference TG**: The FreeDMR network talkgroup identity. Conceptually this is the conference bridge to which clients subscribe.
|
||||
|
||||
**Subscription**: Membership of a client slot/RF TG presentation in a conference TG.
|
||||
|
||||
**Static subscription**: A subscription created by configuration or policy and normally present for the session.
|
||||
|
||||
**Dial-a-TG subscription**: A subscription created by dial-a-TG control. It maps an RF-visible TG, traditionally TG9, to a selected conference TG.
|
||||
|
||||
**Default reflector**: A configured or client-requested default dial-a-TG style subscription for a session. Empty string, integer 0, or boolean false means no default reflector.
|
||||
|
||||
**Stream**: A voice call flow identified by stream metadata. AMBE voice is a stream; DMR data packets are packet-oriented and should not be treated as AMBE streams.
|
||||
|
||||
**Source server**: The server that originated or advertised the packet into the mesh, according to the FBP/OBP protocol version in use.
|
||||
|
||||
**Source repeater**: The access-side repeater/client identity carried when the protocol version supports it.
|
||||
|
||||
**Mesh peer**: Another FreeDMR/OpenBridge/FBP peer server connected through the mesh layer.
|
||||
|
||||
**Server ID**: A FreeDMR server identity. Server IDs are treated separately from client DMR IDs.
|
||||
|
||||
**Server sub-ID**: A subordinate server identity authorized under a server/sysop identity, for example backend or fault-tolerant deployments.
|
||||
|
||||
**Bridge-control message**: An out-of-band FBP/OBP control message, such as source quench or STUN.
|
||||
|
||||
**Packet plane**: The delay-sensitive path that receives, parses, routes, mutates where necessary, and sends DMR packets.
|
||||
|
||||
**Control plane**: Authenticated configuration, API, bridge-control, admission, and policy operations.
|
||||
|
||||
**Reporting plane**: Observational events and state export for dashboards, logs, lastheard, monitoring, and operators.
|
||||
|
||||
**Compatibility/export state**: A derived view that presents old FreeDMR/HBLink/dashboard shapes to consumers. It is not authoritative state.
|
||||
|
||||
**HBP**: Homebrew Protocol, used by many hotspots/repeaters on the access side.
|
||||
|
||||
**OBP**: OpenBridge Protocol version 1. It remains important as an open interop path where intentionally configured.
|
||||
|
||||
**FBP**: FreeDMR Bridge Protocol. Any OpenBridge-derived peer protocol version higher than 1 is termed FBP for FreeDMR clarity.
|
||||
|
||||
**DATA-GATEWAY**: A historical/early expression of a data-oriented FBP link. FreeDMR 2 should preserve data forwarding without making the core a GPS/SMS application processor.
|
||||
|
||||
**Source quench / BCSQ**: A bridge-control hint asking a peer to suppress a stream/TG toward us. It is optional and per stream/TG.
|
||||
|
||||
**STUN / BCST**: A broader bridge-control gate intended to stop all FBP traffic from a peer under the current conceptual model.
|
||||
|
||||
**OVCM**: Open Voice Channel Mode. ETSI service option bit 0x04 when explicitly used. HBLink legacy 0x20 is compatibility history, not standards-clean OVCM.
|
||||
|
||||
**Synthetic LC**: A generated Link Control value used when FreeDMR has to create fallback LC information.
|
||||
|
||||
**Real inbound LC**: LC decoded from received traffic. It should be preserved unchanged unless FreeDMR deliberately rewrites it.
|
||||
@ -0,0 +1,41 @@
|
||||
# FreeDMR 2 System Model
|
||||
|
||||
FreeDMR 2 is a layered system. The layers are design boundaries, not necessarily separate processes at first.
|
||||
|
||||
## Access Layer
|
||||
|
||||
The access layer owns HBP and future client/repeater protocols. It handles login, authentication, options, keepalive, access sessions, RF-facing slot state, and RF-visible TG presentation.
|
||||
|
||||
A configured listener is not the client identity. A single listener should eventually support multiple clients directly, replacing proxy-mediated fan-out where possible.
|
||||
|
||||
## Subscription Layer
|
||||
|
||||
The subscription layer owns talkgroup conference membership. It handles direct TG subscriptions, dial-a-TG subscriptions, static subscriptions, default reflectors, user/API/SMS activated subscriptions, expiry, and RF-visible TG to conference TG mapping.
|
||||
|
||||
Packet routing should consume subscription state. It should not need to know whether a subscription came from static config, dial-a-TG, API, SMS, or a future UI.
|
||||
|
||||
## Mesh Layer
|
||||
|
||||
The mesh layer owns FBP/OBP/trunk-style inter-server traffic. It handles loop control, source quench, hop/version handling, bridge control, source server/repeater metadata, and conference traffic between servers.
|
||||
|
||||
FreeDMR remains a peer network, not hub-and-spoke. Local sysops retain local routing and policy autonomy.
|
||||
|
||||
## Packet/Stream Layer
|
||||
|
||||
The packet/stream layer owns packet parsing, stream lifecycle, sequence handling, terminators, LC/embedded LC handling, data-vs-voice classification, and packet mutation boundaries.
|
||||
|
||||
Raw packet bytes are immutable input until an explicit named rewrite operation occurs.
|
||||
|
||||
## Reporting Layer
|
||||
|
||||
The reporting layer is observational only. It emits state and events to local dashboards, global lastheard exporters, logs, and monitoring consumers.
|
||||
|
||||
Reporting must not steer packet routing.
|
||||
|
||||
## Control/API Layer
|
||||
|
||||
The control/API layer provides explicit authenticated operations for sysop and control-plane actions. It should operate on access sessions, subscriptions, mesh peers, and reporting state without blocking the packet path.
|
||||
|
||||
## Critical Invariant
|
||||
|
||||
Reporting, dashboards, APIs, databases, exporters, and monitoring consumers must not block or steer packet handling.
|
||||
@ -0,0 +1,251 @@
|
||||
# FreeDMR 2 State Model
|
||||
|
||||
The FreeDMR 2 state model separates listener identity, client identity, subscription state, stream state, mesh peer state, and reporting/export views. The legacy `BRIDGES` dict is not the authoritative FreeDMR 2 model.
|
||||
|
||||
Recommended hot-path structures:
|
||||
|
||||
- `NamedTuple` or tuple keys for hot dict/set indexes.
|
||||
- `dataclass(slots=True)` for mutable state records.
|
||||
- `heapq` expiry queue with lazy invalidation.
|
||||
- Local in-memory packet-plane state.
|
||||
- No packet-path dependency on external databases or reporting consumers.
|
||||
|
||||
Suggested indexes:
|
||||
|
||||
```python
|
||||
subscriptions_by_conference_tg[conference_tg] -> set[SubscriptionKey]
|
||||
subscription_by_rf[(client_id, slot, rf_tg)] -> SubscriptionKey
|
||||
subscriptions_by_client_slot[(client_id, slot)] -> set[SubscriptionKey]
|
||||
expiry_heap -> (expires_at, generation, SubscriptionKey)
|
||||
```
|
||||
|
||||
## AccessSession
|
||||
|
||||
Purpose: Represents a live client/repeater session.
|
||||
|
||||
Owner: Access layer.
|
||||
|
||||
Key: Client DMR ID plus listener/session endpoint data.
|
||||
|
||||
Mutable fields: Authentication state, options, keepalive time, endpoint, active slots, supported protocol features.
|
||||
|
||||
Expiry/timer behaviour: Keepalive/session timeout expires the session and resets session-scoped options to system defaults.
|
||||
|
||||
Packet-plane rules: Read for packet admission, option interpretation, slot state, and source identity. Writes only minimal hot state such as last packet/keepalive.
|
||||
|
||||
Control-plane rules: API may read and update bounded session options.
|
||||
|
||||
Reporting/export view: Client connected/disconnected/options/state events.
|
||||
|
||||
Compatibility mapping: Current configured `MASTER`/`SYSTEM` session fields and peer status entries.
|
||||
|
||||
## Listener
|
||||
|
||||
Purpose: Owns a UDP socket/service endpoint.
|
||||
|
||||
Owner: Access layer or transport shell.
|
||||
|
||||
Key: Listener name or bind address/port.
|
||||
|
||||
Mutable fields: Socket state, configured admission policy, active access sessions.
|
||||
|
||||
Expiry/timer behaviour: None beyond transport lifecycle.
|
||||
|
||||
Packet-plane rules: Receives and sends packets; should not be treated as client identity.
|
||||
|
||||
Control-plane rules: Configuration and lifecycle only.
|
||||
|
||||
Reporting/export view: Listener up/down and session counts.
|
||||
|
||||
Compatibility mapping: Current `MASTER` stanza.
|
||||
|
||||
## ClientSlotState
|
||||
|
||||
Purpose: Tracks per-client per-slot RF-facing state.
|
||||
|
||||
Owner: Access and subscription layers.
|
||||
|
||||
Key: `(client_id, slot)`.
|
||||
|
||||
Mutable fields: RF-visible TG activity, current stream, hang/timeout state, default reflector, static and dial subscriptions.
|
||||
|
||||
Expiry/timer behaviour: Stream timers, dial/default reflector expiry, session reset on disconnect.
|
||||
|
||||
Packet-plane rules: Read for RF-visible TG mapping and stream admission. Writes current stream and observed activity.
|
||||
|
||||
Control-plane rules: API may activate/deactivate subscriptions and defaults.
|
||||
|
||||
Reporting/export view: Slot activity and active subscription state.
|
||||
|
||||
Compatibility mapping: Existing slot options, dial-a-TG state, timeout fields.
|
||||
|
||||
## TalkgroupSubscription
|
||||
|
||||
Purpose: Represents membership of a client slot/RF TG in a conference TG.
|
||||
|
||||
Owner: Subscription layer.
|
||||
|
||||
Key: Stable `SubscriptionKey`, likely `(client_id, slot, rf_tg, conference_tg, mode)`.
|
||||
|
||||
Mutable fields: Active flag, source, expiry, generation, priority/policy metadata.
|
||||
|
||||
Expiry/timer behaviour: Static subscriptions normally session-bound; dial/default/user subscriptions may expire.
|
||||
|
||||
Packet-plane rules: Read heavily for routing. Writes should be explicit activation/deactivation/expiry only.
|
||||
|
||||
Control-plane rules: API and bridge control may create/remove/update subscriptions subject to policy.
|
||||
|
||||
Reporting/export view: Subscription activated/deactivated/expired events.
|
||||
|
||||
Compatibility mapping: Current `BRIDGES` entries and `#` reflector export names.
|
||||
|
||||
## StreamState
|
||||
|
||||
Purpose: Tracks voice stream lifecycle and packet ordering.
|
||||
|
||||
Owner: Packet/stream layer.
|
||||
|
||||
Key: Stream ID plus source identity and direction namespace where needed.
|
||||
|
||||
Mutable fields: Source ID, destination/conference TG, RF-visible TG when relevant, slot, last sequence, last packet time, LC state, source server/repeater metadata, loop-control state.
|
||||
|
||||
Expiry/timer behaviour: Explicit terminator is strong end; timeout is softer, especially on HBP.
|
||||
|
||||
Packet-plane rules: Read/write in packet path. Must be local and fast.
|
||||
|
||||
Control-plane rules: Normally read-only, except explicit reset/debug operations.
|
||||
|
||||
Reporting/export view: Call started/ended/lost events.
|
||||
|
||||
Compatibility mapping: Current stream tracking dicts and report socket call state.
|
||||
|
||||
## MeshPeerState
|
||||
|
||||
Purpose: Tracks a peer server/link.
|
||||
|
||||
Owner: Mesh layer.
|
||||
|
||||
Key: Peer/server ID and authenticated endpoint/session where available.
|
||||
|
||||
Mutable fields: Protocol version, endpoint, auth state, last seen, stun/quench state, supported metadata layout, send/receive counters.
|
||||
|
||||
Expiry/timer behaviour: Peer keepalive/control timeout; auth renewal timers.
|
||||
|
||||
Packet-plane rules: Read for admission, protocol layout, and source metadata. Writes last-seen counters and cached safety state only.
|
||||
|
||||
Control-plane rules: API/BCXX may stun, clear stun, authenticate, or update policy.
|
||||
|
||||
Reporting/export view: Peer up/down/stun/source-quench events.
|
||||
|
||||
Compatibility mapping: Current OBP/FBP peer entries.
|
||||
|
||||
## BridgeControlState
|
||||
|
||||
Purpose: Holds bridge-control effects such as source quench, STUN, and authentication state.
|
||||
|
||||
Owner: Mesh/control layer.
|
||||
|
||||
Key: Peer ID plus control scope, for example `(peer_id, stream_id, conference_tg)` for BCSQ.
|
||||
|
||||
Mutable fields: Active flag, reason, expiry, generation, authenticated issuer.
|
||||
|
||||
Expiry/timer behaviour: Source quench and soft controls should expire; hard policy blocks may persist until cleared.
|
||||
|
||||
Packet-plane rules: Read for admission/suppression. Writes only when handling bridge-control packets.
|
||||
|
||||
Control-plane rules: API/BCXX may set or clear controls.
|
||||
|
||||
Reporting/export view: Mesh control events.
|
||||
|
||||
Compatibility mapping: Current BCSQ/BCST handling.
|
||||
|
||||
## ReportingState
|
||||
|
||||
Purpose: Tracks reporting pipeline health and retained current state.
|
||||
|
||||
Owner: Reporting layer.
|
||||
|
||||
Key: Event family or retained state identity.
|
||||
|
||||
Mutable fields: Queue depth, dropped counts, publisher connected state, last emitted state.
|
||||
|
||||
Expiry/timer behaviour: Retained state refresh and reconnect backoff.
|
||||
|
||||
Packet-plane rules: Packet path may enqueue non-blocking events only.
|
||||
|
||||
Control-plane rules: API may read reporting health.
|
||||
|
||||
Reporting/export view: Native v2 reporting state.
|
||||
|
||||
Compatibility mapping: Current dashboard socket state, preferably through a sidecar adapter.
|
||||
|
||||
## CompatibilityExportState
|
||||
|
||||
Purpose: Derived legacy-shaped view for old dashboard/API/HBLink-compatible consumers.
|
||||
|
||||
Owner: Compatibility adapter.
|
||||
|
||||
Key: Consumer-specific.
|
||||
|
||||
Mutable fields: Cached translated state.
|
||||
|
||||
Expiry/timer behaviour: Follows source state; may drop stale export entries.
|
||||
|
||||
Packet-plane rules: Must not be read by packet routing.
|
||||
|
||||
Control-plane rules: May expose old-compatible admin views if required.
|
||||
|
||||
Reporting/export view: Legacy compatibility only.
|
||||
|
||||
Compatibility mapping: `BRIDGES`, `SYSTEM`, `MASTER`, and `#` reflector names.
|
||||
|
||||
## Worker Ownership Considerations
|
||||
|
||||
Authoritative packet-plane state must have one owner. Reporting/export state is derived and must not drive routing. External stores may distribute snapshots or control-plane updates, but they are not per-packet routing dependencies.
|
||||
|
||||
Process boundaries must preserve the same state ownership rules as in-process modules.
|
||||
|
||||
AccessSession:
|
||||
|
||||
- Classification: Access/session state with packet-plane admission impact.
|
||||
- Likely owner: Transport/listener process initially.
|
||||
- Future ownership: May be assigned to a routing worker once admitted.
|
||||
|
||||
ClientSlotState:
|
||||
|
||||
- Classification: Packet-plane authoritative state.
|
||||
- Likely owner: Single routing owner for that client/slot.
|
||||
|
||||
TalkgroupSubscription:
|
||||
|
||||
- Classification: Packet-plane authoritative state.
|
||||
- Likely owner: Single routing/subscription owner.
|
||||
|
||||
StreamState:
|
||||
|
||||
- Classification: Packet-plane authoritative state.
|
||||
- Likely owner: Single stream owner; all packets for a given stream should be handled by one owner.
|
||||
|
||||
MeshPeerState:
|
||||
|
||||
- Classification: Split transport/session state and routing policy state.
|
||||
- Likely owner: Transport/session owner for socket/auth/session facts; routing policy owner for cached packet decisions.
|
||||
- Rule: Authenticated peer/session state must be cached locally for packet decisions.
|
||||
|
||||
BridgeControlState:
|
||||
|
||||
- Classification: Control-plane input with packet-plane effect.
|
||||
- Likely owner: Relevant packet/routing owner for active BCSQ/STUN effects.
|
||||
- Rule: BCSQ/STUN state used by the packet path must be local to the relevant packet owner.
|
||||
|
||||
ReportingState:
|
||||
|
||||
- Classification: Reporting/export snapshot and event state.
|
||||
- Likely owner: Reporting worker.
|
||||
- Rule: Not authoritative for packet routing.
|
||||
|
||||
CompatibilityExportState:
|
||||
|
||||
- Classification: Derived compatibility state.
|
||||
- Likely owner: Compatibility adapter/export worker.
|
||||
- Rule: Never authoritative.
|
||||
@ -0,0 +1,66 @@
|
||||
# FreeDMR 2 Subscription Model
|
||||
|
||||
The subscription model is the centrepiece of FreeDMR 2.
|
||||
|
||||
Conceptually, each TG is a conference bridge. Client systems subscribe to conference TGs. FreeDMR routes traffic according to active subscriptions, not according to the legacy shape of the `BRIDGES` dict.
|
||||
|
||||
Definitions:
|
||||
|
||||
- `conference_tg`: FreeDMR network/conference identity.
|
||||
- `rf_tg`: Client-facing RF presentation identity.
|
||||
|
||||
Examples:
|
||||
|
||||
Direct TG:
|
||||
|
||||
```text
|
||||
rf_tg == conference_tg
|
||||
```
|
||||
|
||||
Dial-a-TG:
|
||||
|
||||
```text
|
||||
rf_tg == 9
|
||||
conference_tg == selected reflector/TG
|
||||
```
|
||||
|
||||
Alias/rewrite:
|
||||
|
||||
```text
|
||||
rf_tg may differ from conference_tg by policy/configuration
|
||||
```
|
||||
|
||||
Example subscription:
|
||||
|
||||
```python
|
||||
TalkgroupSubscription(
|
||||
client_id=2345001,
|
||||
slot=2,
|
||||
rf_tg=9,
|
||||
conference_tg=4400,
|
||||
mode="dial",
|
||||
active=True,
|
||||
)
|
||||
```
|
||||
|
||||
## Routing Invariant
|
||||
|
||||
Packet routing should not need to know whether a subscription came from static config, default reflector, dial-a-TG, API, SMS control, or a future UI action. Those are subscription sources, not routing modes.
|
||||
|
||||
## Dial-a-TG Rationale
|
||||
|
||||
Dial-a-TG exists so terminal users can access arbitrary FreeDMR TGs without programming every TG into the terminal/codeplug. It is an amateur-radio usability feature and should be evaluated against that goal, not only against commercial DMR fleet assumptions.
|
||||
|
||||
Control of dial-a-TG from TS1 as well as TS2 is intentional. If TS2 is blocked by unwanted traffic, a user can transmit private-call control on TS1 to disconnect or change the TS2 reflector/TG state.
|
||||
|
||||
Voice prompts should remain RF-visible as TG9 slot 2 unless that policy is deliberately changed.
|
||||
|
||||
## FreeDMR Routing Model
|
||||
|
||||
- TGs are conference groups.
|
||||
- DMR IDs are like phone numbers.
|
||||
- Timeslots are access/capacity paths, more like phone lines.
|
||||
- FreeDMR is intended to be relatively timeslot agnostic.
|
||||
- TS1 control affecting TS2 reflector state is consistent with the FreeDMR PBX/line model.
|
||||
|
||||
This model also allows future arbitrary RF TG aliases, not only the traditional TG9 dial-a-TG rewrite.
|
||||
@ -0,0 +1,61 @@
|
||||
# Packet and Stream Model
|
||||
|
||||
## Packet Mutation Boundaries
|
||||
|
||||
Raw DMR packet bytes should be treated as immutable input until an explicit rewrite operation. Transport simulation and protocol mutation must remain separate.
|
||||
|
||||
Packet mutation must be named, explicit, and testable. FreeDMR should preserve packet bytes unless it intentionally rewrites them.
|
||||
|
||||
Protocol-sensitive rewrite areas include:
|
||||
|
||||
- Slot bit rewrite.
|
||||
- TG rewrite.
|
||||
- Stream ID preservation.
|
||||
- Source ID preservation.
|
||||
- Voice header LC rewrite.
|
||||
- Terminator LC rewrite.
|
||||
- Embedded LC rewrite.
|
||||
|
||||
Voice header/terminator LC and embedded LC must be handled carefully. Embedded LC rewrite should apply only to voice bursts B-E, not data/control packets.
|
||||
|
||||
Same-TG voice forwarding should preserve embedded LC payloads where possible. TG-mapped forwarding may regenerate embedded LC for routing correctness.
|
||||
|
||||
Data/control packets are packet-oriented and not AMBE voice streams. Group-addressed data is valid and can be routed as data, not reported as voice. Data/control classification must remain separate from group-vs-unit addressing.
|
||||
|
||||
Unit/private calls are control-plane only in FreeDMR. Do not introduce general private voice routing unless project policy changes.
|
||||
|
||||
## Sequence and Lifecycle Principles
|
||||
|
||||
DMRD sequence numbers are one byte and modulo-256.
|
||||
|
||||
- Delta `0`: duplicate.
|
||||
- Delta `1`: normal progress.
|
||||
- Delta `2..127`: forward progress with loss.
|
||||
- Delta `128..255`: stale or out-of-order.
|
||||
|
||||
Explicit voice terminator is a strong end-of-stream signal. Timeout without terminator is softer and may remain recoverable on HBP to preserve audio continuity.
|
||||
|
||||
HBP should be more tolerant because it is RF-facing and real deployments include imperfect terminals, repeaters, RF paths, cellular links, and RF IP links. FBP/OBP can be stricter because it is server-to-server, but should still preserve audio where possible on unreliable links.
|
||||
|
||||
Loop-control safety must not be overridden by tolerance for delayed or out-of-order packets.
|
||||
|
||||
## LC and OVCM
|
||||
|
||||
For DMR Group Voice Channel User LC, the first bytes are:
|
||||
|
||||
- FLCO
|
||||
- FID
|
||||
- Service Options
|
||||
|
||||
Normal synthetic group voice LC should use service options `0x00`.
|
||||
|
||||
OVCM is `0x04` if explicitly required.
|
||||
|
||||
HBLink legacy `0x20` should be documented as legacy/compatibility only. It is not standards-clean OVCM and should not be used as a new synthetic/system-generated traffic marker.
|
||||
|
||||
Decoded real inbound LC must be preserved unchanged unless there is a deliberate reason to rewrite. Synthetic/fallback LC generation must be explicit and tested. FreeDMR routing metadata should be used for routing state, not magic bits in synthetic LC.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact live RF behaviour after long HBP gaps still needs validation with real repeaters and terminals.
|
||||
- Some prompt/late-entry behaviour may need live testing because terminal interpretation of DMR standards can be loose or incomplete.
|
||||
@ -0,0 +1,15 @@
|
||||
# Data Packet Policy
|
||||
|
||||
No regression of DMR data support is permitted.
|
||||
|
||||
FreeDMR should forward supported DMR data packets according to conference/subscription and mesh rules. The FreeDMR core should not become a general GPS/SMS application processor.
|
||||
|
||||
GPS, SMS, and similar application processing should be implemented by systems connected via FBP or another mesh/access-adjacent interface. `DATA-GATEWAY` is understood as an earlier expression of this model.
|
||||
|
||||
Existing `SUB_MAP` / last-known-location behaviour is intentional: data addressed to a DMR ID can be routed toward the last known HBP/client location.
|
||||
|
||||
Narrow exceptions may exist for SMS-based dial-a-TG control or DMR SMS alerts to the sysop.
|
||||
|
||||
Data/control classification must be separate from group-vs-unit addressing. A group-addressed data packet is not automatically a voice stream.
|
||||
|
||||
The packet layer may inspect data packets for admission, routing, loop/source-quench safety, reporting, and metadata preservation. Application-level GPS/SMS semantics should live outside the core unless a specific control-plane feature requires it.
|
||||
@ -0,0 +1,40 @@
|
||||
# Mesh Model
|
||||
|
||||
FreeDMR is a peer network, not hub-and-spoke. Local sysops retain policy autonomy.
|
||||
|
||||
The guiding principle remains "everything everywhere", subject to source quench, STUN, ACLs, local policy, authentication, loop-control, and documented exceptions.
|
||||
|
||||
## Loop Control and Bridge Control
|
||||
|
||||
Loop control, source selection, duplicate suppression, source quench, and STUN are packet-plane safety mechanisms.
|
||||
|
||||
Source quench is a control hint to suppress a stream/TG toward a peer. It is optional and scoped per stream/TG.
|
||||
|
||||
STUN is a broader FBP/OpenBridge traffic gate. Under the current conceptual model, `BCST`/STUN applies to all FBP traffic from that peer until cleared or expired by policy.
|
||||
|
||||
`BCSQ` is per stream/TG.
|
||||
|
||||
`BCST`/STUN is all FBP traffic.
|
||||
|
||||
## Protocol Versions and Metadata
|
||||
|
||||
Source server and source repeater metadata must be preserved according to the protocol version actually in use for that session.
|
||||
|
||||
OBP/FBP protocol version controls metadata layout and option order.
|
||||
|
||||
Protocol v1 OBP remains an important open interop path where intentionally configured. FBP v5 is the current richer peer-server protocol target. FBP v4 is historical/deprecation context unless explicitly retained.
|
||||
|
||||
## TG Namespace Rule
|
||||
|
||||
HBP/RF-visible TG and FBP/OBP-visible conference TG can intentionally differ, especially with dial-a-TG.
|
||||
|
||||
Source quench must use the TG namespace visible to the peer sending or receiving the quench.
|
||||
|
||||
For HBP-to-FBP dial-a-TG, `BCSQ` should use the FBP/reflector TG, not local RF TG9.
|
||||
|
||||
For OBP-source traffic, `BCSQ` should use the inbound OBP TG because that is the source-server namespace.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Final FBP v5 identity/auth fields still need a concrete wire-format decision.
|
||||
- STUN recovery policy needs an operator workflow, likely API-driven.
|
||||
@ -0,0 +1,203 @@
|
||||
# Reporting Model
|
||||
|
||||
Decision: FreeDMR 2 replaces the legacy dashboard/report socket model with a new structured reporting event model.
|
||||
|
||||
The existing dashboard is not a compatibility constraint for the FreeDMR 2 core. It must be updated separately or supported by an optional out-of-process adapter.
|
||||
|
||||
Reporting is observational only. Packet routing must not depend on dashboard/report consumers.
|
||||
|
||||
The v2 event schema is the compatibility contract. Raw `BRIDGES`/`SYSTEM` state should not be exposed as the primary v2 API. Old dashboard/report socket event names should not shape the FreeDMR 2 core.
|
||||
|
||||
Any old dashboard compatibility must live in a sidecar/adapter, not inside packet routing. FreeDMR 1.x/current code remains live until FreeDMR 2 is ready, so FreeDMR 2 can make a clean reporting break.
|
||||
|
||||
## Preferred Transport
|
||||
|
||||
MQTT is the preferred external live reporting transport.
|
||||
|
||||
Architecture:
|
||||
|
||||
```text
|
||||
packet path -> non-blocking bounded local event queue -> MQTT publisher worker -> local broker/feed
|
||||
```
|
||||
|
||||
Constraints:
|
||||
|
||||
- MQTT publishing must be asynchronous from the packet worker.
|
||||
- Use a bounded queue.
|
||||
- The bounded local event queue is the only coupling from packet path to reporting worker.
|
||||
- Drop or coalesce low-priority events under pressure.
|
||||
- Emit a later reporting-health event rather than blocking packet handling.
|
||||
- Voice stability takes precedence over reporting completeness.
|
||||
- Reconnect with exponential backoff.
|
||||
- Refresh retained state after reconnect.
|
||||
- Reporting backpressure must be visible through reporting-health events but must not delay DMR packets.
|
||||
|
||||
Reporting is the first major candidate for out-of-process execution. The MQTT publisher should be an independent worker or sidecar where practical. The global lastheard exporter should be a separate process. Dashboard aggregation should not run in the packet hot path.
|
||||
|
||||
Reporting worker crash must not affect packet routing. Reporting worker restart should refresh retained state after reconnect.
|
||||
|
||||
## Local Dashboard and Global Lastheard
|
||||
|
||||
Local dashboard:
|
||||
|
||||
- Consumes local MQTT live state/events.
|
||||
- Displays live client/repeater/server traffic.
|
||||
- Recovers from retained state after reconnect.
|
||||
|
||||
Global lastheard:
|
||||
|
||||
- Central/non-real-time.
|
||||
- Consumes summaries, not packet-plane traffic.
|
||||
- Should preferably be fed by a separate exporter process.
|
||||
- Central outage must not affect local packet handling or local dashboard.
|
||||
|
||||
Preferred flow:
|
||||
|
||||
```text
|
||||
FreeDMR core -> local MQTT feed -> local dashboard
|
||||
-> global-exporter process -> network MQTT/collector
|
||||
```
|
||||
|
||||
## Initial Event Families
|
||||
|
||||
- `server.started`
|
||||
- `server.stopping`
|
||||
- `client.connected`
|
||||
- `client.disconnected`
|
||||
- `client.options_changed`
|
||||
- `subscription.activated`
|
||||
- `subscription.deactivated`
|
||||
- `subscription.expired`
|
||||
- `call.started`
|
||||
- `call.ended`
|
||||
- `call.lost`
|
||||
- `mesh.peer_up`
|
||||
- `mesh.peer_down`
|
||||
- `mesh.source_quench`
|
||||
- `mesh.stun`
|
||||
- `loop.detected`
|
||||
- `packet.rate_limited`
|
||||
- `reporting.queue_overflow`
|
||||
- `reporting.publisher_disconnected`
|
||||
- `reporting.publisher_reconnected`
|
||||
- `reporting.events_dropped`
|
||||
|
||||
## Suggested MQTT Topics
|
||||
|
||||
```text
|
||||
freedmr/v2/{server_id}/state
|
||||
freedmr/v2/{server_id}/client/{client_id}/state
|
||||
freedmr/v2/{server_id}/client/{client_id}/slot/{slot}/activity
|
||||
freedmr/v2/{server_id}/subscription/{subscription_id}/state
|
||||
freedmr/v2/{server_id}/call/{stream_id}/start
|
||||
freedmr/v2/{server_id}/call/{stream_id}/end
|
||||
freedmr/v2/{server_id}/mesh/{peer_id}/state
|
||||
freedmr/v2/{server_id}/event
|
||||
```
|
||||
|
||||
Use retained messages for current state and non-retained messages for transient events.
|
||||
|
||||
## Example Events
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "server.started",
|
||||
"server_id": "2345",
|
||||
"version": "2.0-dev",
|
||||
"time": "2026-05-24T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "client.connected",
|
||||
"server_id": "2345",
|
||||
"client_id": 2345001,
|
||||
"listener": "hbp-public",
|
||||
"endpoint": "198.51.100.10:62031",
|
||||
"time": "2026-05-24T12:00:01Z"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "subscription.activated",
|
||||
"server_id": "2345",
|
||||
"subscription_id": "2345001-2-9-4400",
|
||||
"client_id": 2345001,
|
||||
"slot": 2,
|
||||
"rf_tg": 9,
|
||||
"conference_tg": 4400,
|
||||
"mode": "dial",
|
||||
"source": "dial-a-tg",
|
||||
"time": "2026-05-24T12:00:02Z"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "call.started",
|
||||
"server_id": "2345",
|
||||
"stream_id": 12345678,
|
||||
"client_id": 2345001,
|
||||
"slot": 2,
|
||||
"source_id": 2345678,
|
||||
"rf_tg": 9,
|
||||
"conference_tg": 4400,
|
||||
"source": "hbp",
|
||||
"time": "2026-05-24T12:00:03Z"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "call.ended",
|
||||
"server_id": "2345",
|
||||
"stream_id": 12345678,
|
||||
"reason": "terminator",
|
||||
"duration_ms": 18420,
|
||||
"packets": 614,
|
||||
"time": "2026-05-24T12:00:21Z"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "call.lost",
|
||||
"server_id": "2345",
|
||||
"stream_id": 12345678,
|
||||
"reason": "timeout",
|
||||
"last_seen_ms_ago": 7000,
|
||||
"time": "2026-05-24T12:00:28Z"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "mesh.source_quench",
|
||||
"server_id": "2345",
|
||||
"peer_id": "2350",
|
||||
"stream_id": 12345678,
|
||||
"conference_tg": 4400,
|
||||
"reason": "duplicate-source",
|
||||
"time": "2026-05-24T12:00:04Z"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "reporting.queue_overflow",
|
||||
"server_id": "2345",
|
||||
"dropped_events": 42,
|
||||
"queue_limit": 2048,
|
||||
"policy": "drop-low-priority",
|
||||
"time": "2026-05-24T12:00:05Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Broker packaging and defaults: Mosquitto, embedded broker, external broker, or optional dependency.
|
||||
- Exact retained-state expiry policy.
|
||||
- Authentication model for MQTT clients.
|
||||
- Whether legacy dashboard compatibility is a supplied sidecar or a separate dashboard migration task.
|
||||
@ -0,0 +1,52 @@
|
||||
# API and Control Model
|
||||
|
||||
The current HTTP/JSON API is experimental and should be treated as a starting point, not a fixed FreeDMR 2 contract.
|
||||
|
||||
## Current API Principles
|
||||
|
||||
- Local administration and automation.
|
||||
- Not for public internet exposure.
|
||||
- Small in-memory operations only.
|
||||
- Bounded request bodies.
|
||||
- No expensive live serialization of internal state.
|
||||
- No blocking packet path.
|
||||
- User-level authentication by connected peer/client session key.
|
||||
- System-level authentication by system API key.
|
||||
|
||||
The API should bind to localhost by default unless explicitly configured otherwise.
|
||||
|
||||
## FreeDMR 2 Direction
|
||||
|
||||
API operations should be bounded control-plane operations over access sessions, subscriptions, mesh peers, and reporting state.
|
||||
|
||||
Destructive system actions such as kill, resetall, and STUN clear should be separately enableable. Audit logs are required. Keys and secrets must never be logged.
|
||||
|
||||
Dashboard controls should use authenticated HTTP API operations unless a future UI genuinely needs bidirectional streaming.
|
||||
|
||||
Compatibility with the old API should be an adapter concern if needed.
|
||||
|
||||
The API may eventually run as a separate control-plane worker or sidecar. API requests should become `ControlCommand` messages sent to the owner of the relevant state.
|
||||
|
||||
The API must not directly mutate packet-plane state it does not own. Destructive operations must go through explicit owner-handled commands. API worker failure should not stop existing packet routing, although new control actions may fail until the worker recovers.
|
||||
|
||||
## Suggested v2 Operations
|
||||
|
||||
```text
|
||||
GET /api/v2/health
|
||||
GET /api/v2/version
|
||||
GET /api/v2/state
|
||||
GET /api/v2/client/{client_id}
|
||||
GET /api/v2/client/{client_id}/slot/{slot}
|
||||
POST /api/v2/client/{client_id}/slot/{slot}/subscriptions
|
||||
DELETE /api/v2/client/{client_id}/slot/{slot}/subscriptions/{subscription_id}
|
||||
POST /api/v2/mesh/{peer_id}/stun
|
||||
DELETE /api/v2/mesh/{peer_id}/stun
|
||||
POST /api/v2/system/reset
|
||||
POST /api/v2/system/stop
|
||||
```
|
||||
|
||||
## Packet-Path Rule
|
||||
|
||||
API handlers must not perform work that delays packet routing. Expensive state export, dashboard compatibility, global reporting, and administrative analysis should use snapshots, bounded queues, or separate processes.
|
||||
|
||||
No API path should force expensive live serialization of packet-plane state.
|
||||
@ -0,0 +1,56 @@
|
||||
# Security Model
|
||||
|
||||
Core principle: FreeDMR may sign/authenticate traffic and control messages. FreeDMR should not encrypt amateur-radio or mesh traffic by default.
|
||||
|
||||
The security model is authenticity, integrity, membership validation, and local policy, not secrecy. Amateur radio is public, and users may provide IP backhaul over amateur-radio links where encryption rules matter.
|
||||
|
||||
## Mesh Authentication
|
||||
|
||||
Preferred direction:
|
||||
|
||||
- PKI-backed FBP peer admission through Bridge Control / BCXX.
|
||||
- Signed server/sysop identity.
|
||||
- Bind server ID, authorized sub-IDs, public key, validity, and features where useful.
|
||||
- Bind authenticated identity to observed endpoint/IP.
|
||||
- If endpoint changes, peer must re-authenticate.
|
||||
- Expensive signature/cert validation is control-plane work.
|
||||
- Packet-plane uses cached authenticated session state.
|
||||
- Soft renewal should avoid interrupting in-flight voice when safe.
|
||||
- Hard stop on revocation, explicit failure, endpoint mismatch outside policy, grace expiry, or local policy.
|
||||
|
||||
## Identity and Listing
|
||||
|
||||
Signed identity proves membership/identity, not mandatory carriage. Public listing is separate from mesh identity.
|
||||
|
||||
Local sysops may choose whether to carry or vouch for traffic. A valid signed key does not override local policy.
|
||||
|
||||
Vouching sysop accountability is part of FreeDMR's social trust model. A sysop allowing problematic traffic onto the mesh may see other peers stop peering with them.
|
||||
|
||||
One verification of a key may cover the server ID and authorized sub-IDs for that sysop/server deployment.
|
||||
|
||||
## Distributed Key Gossip Option
|
||||
|
||||
Signed membership documents may be gossiped over bounded/rate-limited BCXX.
|
||||
|
||||
Peers validate signatures and build local key tables. Revocation, expiry, serials, and replay protection are required.
|
||||
|
||||
Key gossip cannot create trust by mere repetition. The packet path must use cached key/policy state.
|
||||
|
||||
This supports autonomous routing decisions for packets that originated from a server even when that source server is not directly connected.
|
||||
|
||||
## Analogue and Digital Bridge Policy
|
||||
|
||||
Analogue ROIP bridges may connect as HBP clients. Permitted does not mean automatically valuable.
|
||||
|
||||
Analogue bridges can be operationally sensitive because mixed or continuous analogue audio is a poor fit for DMR one-source-at-a-time stream behaviour. They may hold a TG open, play tones, or prevent digital users from breaking in until a carrier/timer drops.
|
||||
|
||||
Analogue bridges should be subject to local policy, listing expectations, and peer accountability.
|
||||
|
||||
YSF/NXDN and other AMBE-family networks are often a better technical match than analogue or unlike-codec transcoding, because they can avoid lossy audio translation.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- X.509 certificates versus simpler Ed25519 signed membership documents.
|
||||
- Exact revocation and renewal distribution process.
|
||||
- Default grace period for soft re-authentication.
|
||||
- How much key gossip should be enabled by default.
|
||||
@ -0,0 +1,69 @@
|
||||
# Runtime and Concurrency
|
||||
|
||||
Decision: Do not replace Twisted as the first FreeDMR 2 architecture move.
|
||||
|
||||
## Rationale
|
||||
|
||||
Current packet behaviour is subtle. Twisted's single-threaded reactor is currently a safety boundary. Replacing the event loop and the state model at the same time mixes too many changes.
|
||||
|
||||
The first goal is architectural clarity and testability, not event-loop novelty.
|
||||
|
||||
## Immediate Runtime Strategy
|
||||
|
||||
- Keep Twisted initially as the transport shell.
|
||||
- Use Twisted's single-threaded reactor as a safety boundary while the core is extracted.
|
||||
- Do not replace the event loop and state model at the same time.
|
||||
- Extract the protocol/routing/subscription core behind deterministic interfaces.
|
||||
- Keep packet-plane state local and deterministic.
|
||||
- No blocking work in reactor callbacks.
|
||||
- No dashboard/API/database/MQTT waits in the packet path.
|
||||
- Single-owner state is preferred.
|
||||
- Explicit messages/events are preferred over shared mutable dictionaries across threads/processes.
|
||||
|
||||
## Eventual Capacity Strategy
|
||||
|
||||
FreeDMR 2 should support worker-process scaling once state ownership and message boundaries are explicit and tested.
|
||||
|
||||
The purpose of worker processes is not merely performance. It is also:
|
||||
|
||||
- Clearer state ownership.
|
||||
- Failure isolation.
|
||||
- Safer concurrency.
|
||||
- Testable boundaries.
|
||||
- Future capacity scaling.
|
||||
|
||||
Prefer process/actor ownership over shared-memory no-GIL threading for authoritative routing state. No-GIL Python does not remove the need for clear ownership of mutable packet-plane state.
|
||||
|
||||
Offload non-packet-path work first, including reporting, MQTT publishing, global export, SQL writes, dashboard aggregation, alias refresh, analytics, and lab/codec work.
|
||||
|
||||
Routing-core workers are a later stage. Multi-worker sharding should only be considered after single-worker message-boundary behaviour is proven.
|
||||
|
||||
Twisted can remain the transport shell while reporting/export/control workers move out-of-process.
|
||||
|
||||
## Ownership Split
|
||||
|
||||
Twisted parent/transport process may own:
|
||||
|
||||
- UDP sockets.
|
||||
- HBP/FBP packet receive/send.
|
||||
- Timers.
|
||||
- Process supervision.
|
||||
|
||||
Routing core should eventually own:
|
||||
|
||||
- Stream state.
|
||||
- Subscription state.
|
||||
- Dial-a-TG state.
|
||||
- Loop-control state.
|
||||
- Duplicate suppression.
|
||||
- Routing decisions.
|
||||
|
||||
Migration must be staged and covered by tests. FreeDMR should remain deployable on ordinary low-cost systems such as cheap VPS instances and Raspberry Pi-class hardware.
|
||||
|
||||
See `13-worker-process-scaling.md` for the eventual worker-process capacity model.
|
||||
|
||||
## External Databases
|
||||
|
||||
External stores can be useful for configuration, reporting snapshots, global lastheard, operator UI, and coordination. They should not sit in the packet hot path.
|
||||
|
||||
Packet-plane state should stay local and in memory unless a future design proves a bounded, non-blocking alternative.
|
||||
@ -0,0 +1,100 @@
|
||||
# Testing and Release Gates
|
||||
|
||||
FreeDMR 2 must preserve behaviour through tests before changing architecture. The existing deterministic harness, UDP black-box harness, codec tests, support tests, and future live RF validation form the release gate structure.
|
||||
|
||||
## Test Commands
|
||||
|
||||
General test run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover -v
|
||||
```
|
||||
|
||||
Focused support/codec tests:
|
||||
|
||||
```bash
|
||||
python -m unittest tests.test_freedmr_dmr_codec tests.test_utils -v
|
||||
```
|
||||
|
||||
UDP black-box tests:
|
||||
|
||||
```bash
|
||||
FREEDMR_RUN_UDP_TESTS=1 python -m unittest tests.test_udp_blackbox_harness -v
|
||||
```
|
||||
|
||||
UDP black-box tests with venv bootstrap:
|
||||
|
||||
```bash
|
||||
FREEDMR_RUN_UDP_TESTS=1 \
|
||||
FREEDMR_UDP_BOOTSTRAP_VENV=1 \
|
||||
FREEDMR_UDP_VENV_DIR=/tmp/freedmr-blackbox-venv3 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
python -m unittest tests.test_udp_blackbox_harness -v
|
||||
```
|
||||
|
||||
## Release Gates
|
||||
|
||||
Level 0: Codec/unit/config/support tests. Must pass for every commit.
|
||||
|
||||
Level 1: Deterministic packet/state harness. Must pass before merge to main.
|
||||
|
||||
Level 2: Black-box UDP harness. Must pass before release candidate.
|
||||
|
||||
Level 3: Live RF / real repeater / real peer validation. Required before changing protocol-visible behaviour.
|
||||
|
||||
## What Each Layer Proves
|
||||
|
||||
Deterministic harness:
|
||||
|
||||
- Good for packet parsing seams, routing state, dial-a-TG state, fake-clock expiry, rewrite boundaries, LC/embedded LC tests, data-vs-voice classification, and reporting event generation.
|
||||
- Bypasses real UDP, socket binding, subprocess startup, and Twisted timing.
|
||||
|
||||
UDP black-box harness:
|
||||
|
||||
- Good for subprocess startup, HBP login, UDP parsing, FBP signing, bridge-control, malformed/hostile packets, cadence, packet ordering, and link impairment.
|
||||
- Cannot prove RF-side modem/radio behaviour.
|
||||
|
||||
Live RF validation:
|
||||
|
||||
- Required for protocol-visible changes, prompt/ident behaviour, late entry, OVCM/LC options, repeater/radio compatibility, and real terminal quirks.
|
||||
|
||||
## Required Assertions
|
||||
|
||||
Tests should assert:
|
||||
|
||||
- Route recipients and non-recipients.
|
||||
- Packet byte preservation outside allowed rewrite regions.
|
||||
- Explicit rewrite ranges.
|
||||
- Stream lifecycle.
|
||||
- Subscription state.
|
||||
- Reporting events.
|
||||
- Source quench/STUN behaviour.
|
||||
- Absence of unintended traffic.
|
||||
|
||||
## Worker/process-boundary Release Gates
|
||||
|
||||
Before moving packet-plane behaviour across a process boundary:
|
||||
|
||||
- Deterministic in-process behaviour must already be covered.
|
||||
- The same scenario must be covered through message-boundary tests.
|
||||
- The same scenario must be covered through UDP black-box tests where observable.
|
||||
- Packet bytes must be compared before and after crossing the process boundary.
|
||||
- Route recipient/non-recipient sets must match.
|
||||
- Allowed rewrite regions must match.
|
||||
- Source quench/STUN/loop-control behaviour must match.
|
||||
- Failure injection must prove worker crash/restart does not replay stale packets.
|
||||
- Reporting/control worker backpressure must not block packet routing.
|
||||
- Live RF validation is required for protocol-visible behaviour.
|
||||
|
||||
Suggested future test categories:
|
||||
|
||||
- Reporting worker crash during active call.
|
||||
- Global exporter outage.
|
||||
- API worker unavailable during normal traffic.
|
||||
- Routing worker restart while stream active.
|
||||
- Routing worker backpressure.
|
||||
- Queue overflow from packet process to reporting worker.
|
||||
- Stale `PacketReceived` replay prevention.
|
||||
- Duplicate packet prevention after worker restart.
|
||||
- Stream ownership handoff/drain test.
|
||||
- Coordinator restart test, if a coordinator is introduced.
|
||||
@ -0,0 +1,56 @@
|
||||
# Migration Plan
|
||||
|
||||
Constraints:
|
||||
|
||||
- No big-bang rewrite of packet semantics.
|
||||
- Current FreeDMR remains live until FreeDMR 2 is tested.
|
||||
- Build FreeDMR 2 core beside current code where practical.
|
||||
- Preserve behaviours behind tests.
|
||||
- Compatibility adapters are allowed, but old internal shape should not define the new core.
|
||||
- Worker-process scaling is a design direction, not a reason for a big-bang rewrite.
|
||||
|
||||
## Stages
|
||||
|
||||
Stage 0: Stabilise current code, harness, docs, codec, and known behaviour.
|
||||
|
||||
Stage 1: Distil architecture, glossary, state model, and reporting event contract.
|
||||
|
||||
Stage 2: Extract packet/codec helpers and deterministic routing/subscription seams.
|
||||
|
||||
Stage 3: Introduce explicit internal message/event objects in-process.
|
||||
|
||||
Stage 4: Implement new subscription store in parallel with compatibility export if needed.
|
||||
|
||||
Stage 5: Move reporting/MQTT publisher to an independent worker/sidecar.
|
||||
|
||||
Stage 6: Move global lastheard exporter, SQL writes, dashboard aggregation, and non-critical analytics to workers.
|
||||
|
||||
Stage 7: Define v2 API/control-plane operations over sessions, subscriptions, mesh state, and reporting health, and express them as owner-handled `ControlCommand` messages.
|
||||
|
||||
Stage 8: Introduce listener/client session model supporting multiple clients per listener.
|
||||
|
||||
Stage 9: Introduce mesh auth/BCXX identity admission into the control plane.
|
||||
|
||||
Stage 10: Experiment with a single routing-core worker behind the already-tested message interface.
|
||||
|
||||
Stage 11: Evaluate multi-routing-worker sharding only after single-worker routing is stable and covered.
|
||||
|
||||
Stage 12: Cut over only after deterministic, UDP, and live RF validation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not add general user-to-user private voice routing.
|
||||
- Do not make FreeDMR core a GPS/SMS application processor.
|
||||
- Do not make reporting/dashboard consumers part of packet routing.
|
||||
- Do not encrypt amateur-radio traffic by default.
|
||||
- Do not replace Twisted before extracting and test-covering the core.
|
||||
- Do not preserve the old dashboard protocol inside the FreeDMR 2 packet core.
|
||||
- Do not make worker-process scaling an immediate rewrite requirement.
|
||||
- Do not use Redis, Postgres, MQTT, dashboards, or APIs for live per-packet routing decisions.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact cut-over mechanism from current `BRIDGES` state to subscription state.
|
||||
- Whether old dashboard compatibility is shipped as part of FreeDMR 2 or maintained with the dashboard.
|
||||
- Final mesh authentication wire format and key distribution policy.
|
||||
- Live RF validation matrix for repeaters, hotspots, terminals, and analogue/digital bridges.
|
||||
@ -0,0 +1,289 @@
|
||||
# Worker Process Scaling
|
||||
|
||||
## Decision
|
||||
|
||||
FreeDMR 2 should be designed so that, after the protocol/routing/subscription core has been extracted and tested, selected parts of the system can be moved into separate worker processes to improve capacity, isolate failures, and avoid the practical single-thread/GIL limits of one Python process.
|
||||
|
||||
This is not a first-stage rewrite requirement.
|
||||
|
||||
The first stage remains:
|
||||
|
||||
- Keep Twisted initially.
|
||||
- Extract the deterministic core.
|
||||
- Make state ownership explicit.
|
||||
- Preserve packet behaviour through tests.
|
||||
|
||||
But the FreeDMR 2 state model must not block a later move to worker processes.
|
||||
|
||||
## Why Worker Processes
|
||||
|
||||
CPython single-process execution has practical limits for CPU-bound Python code.
|
||||
|
||||
Twisted's single reactor is useful as an initial safety boundary, but one reactor process should not be assumed to be the final capacity architecture.
|
||||
|
||||
Worker processes provide stronger ownership and failure boundaries than shared-memory threads. Process/message boundaries are safer for FreeDMR routing state than no-GIL shared mutable dictionaries.
|
||||
|
||||
Worker processes fit FreeDMR's need for explicit ownership of routing, stream, subscription, loop-control, and reporting state.
|
||||
|
||||
Worker-process scaling must not make ordinary small FreeDMR deployments heavyweight or hard to run. A single-server deployment on a cheap VPS or Raspberry Pi-class system must remain supported.
|
||||
|
||||
## What Should Be Offloaded First
|
||||
|
||||
Low-risk early offload candidates:
|
||||
|
||||
- MQTT/reporting publisher.
|
||||
- Global lastheard exporter.
|
||||
- Dashboard aggregation.
|
||||
- SQL/database writing.
|
||||
- Historical analytics.
|
||||
- Alias download/refresh.
|
||||
- Expensive codec experiments.
|
||||
- Packet capture/replay analysis.
|
||||
- Non-critical maintenance jobs.
|
||||
- Future transcoding/bridge adjuncts.
|
||||
- Future network-analysis or observability tools.
|
||||
|
||||
These workers must be asynchronous from the packet path.
|
||||
|
||||
If they are slow, blocked, crashed, overloaded, or absent, packet routing must continue.
|
||||
|
||||
Reporting completeness is secondary to voice stability.
|
||||
|
||||
## What Should Not Be Offloaded Early
|
||||
|
||||
Do not initially offload hot-path routing decisions if doing so would add IPC, network, database, lock, queue, or back-pressure waits to every DMR packet.
|
||||
|
||||
Specifically keep local and deterministic until the model is proven:
|
||||
|
||||
- Stream admission.
|
||||
- Duplicate suppression.
|
||||
- Loop control.
|
||||
- Source quench checks.
|
||||
- Dial-a-TG state mutation.
|
||||
- Subscription lookup.
|
||||
- Slot/TG rewrite decisions.
|
||||
- Voice/data classification.
|
||||
- Packet mutation/rewrite.
|
||||
- HBP RF-facing tolerance logic.
|
||||
- Protocol-version-sensitive FBP/OBP metadata handling.
|
||||
|
||||
The packet plane must continue to use local in-memory state and must not depend on external databases, MQTT, dashboards, APIs, or reporting consumers.
|
||||
|
||||
## Possible Long-Term Worker Architecture
|
||||
|
||||
### Transport/listener Process
|
||||
|
||||
Owns:
|
||||
|
||||
- UDP sockets.
|
||||
- HBP receive/send.
|
||||
- FBP/OBP receive/send.
|
||||
- Raw packet admission.
|
||||
- Socket identity.
|
||||
- Keepalive.
|
||||
- Low-level protocol parsing.
|
||||
- Forwarding packet events to the owning routing component.
|
||||
|
||||
### Routing Core Worker
|
||||
|
||||
Owns:
|
||||
|
||||
- Subscription state.
|
||||
- Stream state.
|
||||
- Dial-a-TG state.
|
||||
- Loop-control state.
|
||||
- Source-quench state.
|
||||
- Duplicate suppression.
|
||||
- Packet routing decisions.
|
||||
- Explicit packet rewrite decisions.
|
||||
- Authoritative packet-plane state for its assigned clients/streams/TGs.
|
||||
|
||||
### Reporting Worker
|
||||
|
||||
Owns:
|
||||
|
||||
- MQTT publishing.
|
||||
- Retained state refresh.
|
||||
- Reporting event queue.
|
||||
- Dashboard event fanout.
|
||||
- Reporting health events.
|
||||
- Drop/coalesce policy under pressure.
|
||||
|
||||
### Global Exporter Worker
|
||||
|
||||
Owns:
|
||||
|
||||
- Subscribing to local reporting feed.
|
||||
- Filtering and summarising local events.
|
||||
- Publishing curated summaries to global lastheard/network collector.
|
||||
- Retry/spool policy for central outage.
|
||||
|
||||
### Control/API Worker or Control-Plane Adapter
|
||||
|
||||
Owns:
|
||||
|
||||
- Sysop/API requests.
|
||||
- Validating control-plane credentials.
|
||||
- Converting API requests into explicit state-change commands.
|
||||
- Receiving `ControlResult` messages.
|
||||
- Never directly mutating packet-plane state it does not own.
|
||||
|
||||
### Optional Future Codec/Transcode/Analysis Workers
|
||||
|
||||
Own:
|
||||
|
||||
- Expensive or experimental codec work.
|
||||
- Transcoding adjuncts.
|
||||
- Packet replay analysis.
|
||||
- Offline diagnostics.
|
||||
- Future lab features.
|
||||
|
||||
These must remain outside the live packet hot path unless explicitly proven safe.
|
||||
|
||||
## State Ownership Rules
|
||||
|
||||
- Every mutable authoritative state object must have exactly one owner.
|
||||
- Other processes may hold snapshots or caches, but only the owner mutates authoritative state.
|
||||
- Do not use `multiprocessing.Manager().dict()` or shared mutable proxy objects as the main architecture.
|
||||
- Do not recreate a cross-process global `BRIDGES`-style mutable structure.
|
||||
- Use explicit messages/events instead of pretending cross-process state is a normal Python dict.
|
||||
- Packet bytes crossing process boundaries should be immutable.
|
||||
- Packet mutation must remain explicit, named, and testable.
|
||||
- A process boundary must not hide unclear ownership.
|
||||
- State ownership must be visible in tests and documentation.
|
||||
|
||||
## Message Boundary
|
||||
|
||||
Likely internal message families:
|
||||
|
||||
| Message | Plane |
|
||||
| --- | --- |
|
||||
| `PacketReceived` | packet-plane |
|
||||
| `PacketAccepted` | packet-plane |
|
||||
| `PacketDropped` | packet-plane |
|
||||
| `RouteDecision` | packet-plane |
|
||||
| `PacketToSend` | packet-plane |
|
||||
| `PacketMutated` | packet-plane |
|
||||
| `StreamStarted` | packet-plane |
|
||||
| `StreamEnded` | packet-plane |
|
||||
| `StreamLost` | packet-plane |
|
||||
| `SubscriptionActivated` | control-plane |
|
||||
| `SubscriptionDeactivated` | control-plane |
|
||||
| `SubscriptionExpired` | packet-plane |
|
||||
| `SourceQuenchReceived` | packet-plane |
|
||||
| `SourceQuenchSendRequested` | packet-plane |
|
||||
| `StunActivated` | control-plane |
|
||||
| `StunCleared` | control-plane |
|
||||
| `ReportingEvent` | reporting-plane |
|
||||
| `ControlCommand` | control-plane |
|
||||
| `ControlResult` | control-plane |
|
||||
| `WorkerStarted` | worker/supervision-plane |
|
||||
| `WorkerStopping` | worker/supervision-plane |
|
||||
| `WorkerHealth` | worker/supervision-plane |
|
||||
| `WorkerBackpressure` | worker/supervision-plane |
|
||||
| `WorkerCrashed` | worker/supervision-plane |
|
||||
| `WorkerRestarted` | worker/supervision-plane |
|
||||
|
||||
Packet-plane messages must be compact, bounded, and safe for high frequency use.
|
||||
|
||||
## Partitioning / Sharding Options
|
||||
|
||||
Possible future sharding models, without choosing one prematurely:
|
||||
|
||||
- By client/repeater DMR ID.
|
||||
- By listener/access socket.
|
||||
- By conference TG.
|
||||
- By source server / mesh peer.
|
||||
- By stream ID.
|
||||
- Hybrid model.
|
||||
|
||||
Constraints:
|
||||
|
||||
- All packets for a given live stream must be processed in order by the same stream owner.
|
||||
- Dial-a-TG state for one client/slot must have one owner.
|
||||
- Subscription state for one client/slot must have one owner.
|
||||
- Loop-control/source-quench state must be consistent for a given TG/stream/source path.
|
||||
- Cross-worker routing must not reintroduce duplicate packets or loops.
|
||||
- Worker assignment must be observable and testable.
|
||||
- Worker assignment must not depend on dashboard/reporting state.
|
||||
- Sharding must preserve the FreeDMR "everything everywhere" mesh principle, subject to existing source quench, STUN, ACL, policy, and authentication rules.
|
||||
|
||||
## Coordinator Model
|
||||
|
||||
FreeDMR 2 may eventually need a lightweight coordinator.
|
||||
|
||||
The coordinator may:
|
||||
|
||||
- Assign clients/sessions/TGs to workers.
|
||||
- Distribute subscription snapshots.
|
||||
- Manage worker health.
|
||||
- Restart workers.
|
||||
- Publish control-plane updates.
|
||||
- Provide routing-worker discovery.
|
||||
- Coordinate graceful drain/restart.
|
||||
|
||||
The coordinator must not:
|
||||
|
||||
- Synchronously participate in every packet routing decision.
|
||||
- Become a single blocking dependency for live voice.
|
||||
- Hide packet-plane state in an external database.
|
||||
- Make ordinary small deployments require clustered infrastructure.
|
||||
|
||||
Single-process/small-server deployment must remain supported. A coordinator should be optional or internal for simple deployments.
|
||||
|
||||
## Failure Behaviour
|
||||
|
||||
- Reporting worker failure: packet routing continues.
|
||||
- Global exporter failure: local service continues.
|
||||
- Dashboard aggregation failure: packet routing continues.
|
||||
- API/control worker failure: existing packet routing continues, but new control actions may fail.
|
||||
- Alias refresh worker failure: current aliases remain in use.
|
||||
- Analytics worker failure: packet routing continues.
|
||||
- Routing worker failure: affected sessions/streams are dropped or restarted according to explicit policy.
|
||||
- Transport/listener failure: affected sockets/sessions are lost until restart.
|
||||
- Worker restart must not replay stale DMR packets.
|
||||
- Retained/reporting state may be refreshed after recovery.
|
||||
- In-flight voice may be lost during worker crash, but failure must not poison the mesh or produce loops.
|
||||
- Backpressure from non-packet workers must not propagate into the packet path.
|
||||
|
||||
## Tests Required Before Worker Split
|
||||
|
||||
Before moving any packet-plane component into a separate process, require:
|
||||
|
||||
- Deterministic harness coverage of the state machine.
|
||||
- UDP black-box coverage of the same behaviour.
|
||||
- Message-boundary tests proving packet bytes and route decisions are preserved.
|
||||
- Failure-injection tests for worker timeout/crash/restart.
|
||||
- Queue-backpressure tests proving reporting/control workers cannot block packets.
|
||||
- Tests proving no stale packet replay after worker restart.
|
||||
- Tests proving source quench, STUN, loop-control, and duplicate suppression are preserved across the boundary.
|
||||
- Live RF validation for protocol-visible behaviour.
|
||||
|
||||
Worker split must not be considered complete until deterministic, UDP, and live RF validation agree for protocol-visible paths.
|
||||
|
||||
## Migration Path
|
||||
|
||||
Stage A: Extract pure/deterministic routing/subscription state behind explicit interfaces.
|
||||
|
||||
Stage B: Introduce internal message/event objects in-process.
|
||||
|
||||
Stage C: Move reporting/MQTT/global export to separate processes.
|
||||
|
||||
Stage D: Move slow maintenance, alias refresh, SQL/global lastheard, and analytics work to workers.
|
||||
|
||||
Stage E: Experiment with routing core as a child process behind the same message interface.
|
||||
|
||||
Stage F: Evaluate multi-routing-worker sharding only after the single routing-worker process model is stable and fully tested.
|
||||
|
||||
Stage G: Only after the above, consider whether transport/listener processes should be split by listener, client set, protocol, or deployment role.
|
||||
|
||||
## Explicit Non-Goals
|
||||
|
||||
- Do not introduce shared mutable cross-process `BRIDGES`-like state.
|
||||
- Do not depend on Redis/Postgres/MQTT for per-packet routing decisions.
|
||||
- Do not require heavyweight infrastructure for ordinary single-server deployments.
|
||||
- Do not use worker processes to hide unclear state ownership.
|
||||
- Do not move protocol-sensitive packet mutation across process boundaries until byte-preservation and rewrite tests prove equivalence.
|
||||
- Do not assume no-GIL Python solves FreeDMR's state ownership problem.
|
||||
- Do not replace Twisted and introduce worker sharding in the same step.
|
||||
- Do not make reporting, dashboard, SQL, global lastheard, or API availability part of the packet routing path.
|
||||
@ -0,0 +1,28 @@
|
||||
# FreeDMR 2 Architecture Notes
|
||||
|
||||
This directory contains the distilled FreeDMR 2 design notes. The older notes in `docs/codex-notes.md`, `docs/freedmr-2-architecture-decisions.md`, and the test/API docs remain source material and engineering history.
|
||||
|
||||
FreeDMR 2 is not a blind rewrite of packet behaviour. The protected asset is the FreeDMR operating model: DMR packet semantics, dial-a-TG, TG/DMR-ID-centric routing, loop control, source quench, mesh behaviour, RF-side tolerance, data forwarding, and practical amateur-radio interoperability.
|
||||
|
||||
Recommended reading order:
|
||||
|
||||
1. `00-glossary.md`
|
||||
2. `01-system-model.md`
|
||||
3. `02-state-model.md`
|
||||
4. `03-subscription-model.md`
|
||||
5. `04-packet-and-stream-model.md`
|
||||
6. `05-data-packet-policy.md`
|
||||
7. `06-mesh-model.md`
|
||||
8. `07-reporting-model.md`
|
||||
9. `08-api-control-model.md`
|
||||
10. `09-security-model.md`
|
||||
11. `10-runtime-and-concurrency.md`
|
||||
12. `13-worker-process-scaling.md`
|
||||
13. `11-testing-and-release-gates.md`
|
||||
14. `12-migration-plan.md`
|
||||
|
||||
Architecture Decision Records live in `adr/`. They record proposed FreeDMR 2 decisions separately from implementation work.
|
||||
|
||||
Current FreeDMR 1.x remains live until FreeDMR 2 is tested. Compatibility adapters are allowed where useful, but old HBLink object layout, dashboard socket assumptions, and legacy `BRIDGES` structure should not define the FreeDMR 2 core.
|
||||
|
||||
FreeDMR 2 keeps Twisted initially but is designed for eventual worker-process scaling. Non-packet-path workers such as reporting/global export move first; packet-plane routing workers are a later stage after state ownership and message-boundary tests are proven.
|
||||
@ -0,0 +1,24 @@
|
||||
# ADR 0001: Protected Model, Not HBLink Structure
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
FreeDMR current code is HBLink-derived and centred around `bridge_master.py`, `hblink.py`, configured `MASTER`/`SYSTEM` stanzas, global `BRIDGES`, Twisted, and the current dashboard/report model.
|
||||
|
||||
The valuable part is the operating model learned from real deployments: packet semantics, dial-a-TG, TG/DMR-ID-centric routing, loop control, source quench, mesh behaviour, RF-side tolerance, data forwarding, and practical amateur-radio interoperability.
|
||||
|
||||
## Decision
|
||||
The protected asset is FreeDMR behaviour/model, not the HBLink-derived object layout.
|
||||
|
||||
## Rationale
|
||||
HBLink-era structure blocks clarity, scaling, multi-client listeners, and testability. FreeDMR 2 should preserve validated packet behaviour while allowing cleaner internal models.
|
||||
|
||||
## Consequences
|
||||
FreeDMR 2 may replace legacy internal structures. Behaviour changes still require tests and live validation where protocol-visible.
|
||||
|
||||
## Compatibility
|
||||
Compatibility views may expose old shapes such as `BRIDGES`, `MASTER`, `SYSTEM`, or `#` reflector names, but they are adapters, not authoritative core state.
|
||||
|
||||
## Testing Requirements
|
||||
Regression tests must cover routing, dial-a-TG, loop control, source quench, data forwarding, and packet rewrite behaviour before internal models are replaced.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0002: Keep Twisted Initially
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
Twisted currently provides UDP transport, timers, and a single-threaded reactor boundary. Packet behaviour is subtle and production-proven.
|
||||
|
||||
## Decision
|
||||
Keep Twisted initially as a transport safety boundary; consider replacement only after it is a thin shell.
|
||||
|
||||
## Rationale
|
||||
Replacing the event loop and state model at the same time creates avoidable risk. Extracting the routing/subscription core first gives deterministic test coverage and clearer future migration options.
|
||||
|
||||
## Consequences
|
||||
FreeDMR 2 starts with evolutionary architecture work, not an event-loop rewrite. Twisted callbacks must remain non-blocking.
|
||||
|
||||
## Compatibility
|
||||
Current deployment and transport behaviour can remain familiar while the core model is extracted.
|
||||
|
||||
## Testing Requirements
|
||||
Deterministic core tests and UDP black-box tests must cover behaviour before any later Twisted replacement is considered.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0003: Subscription Model Replaces BRIDGES
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
The legacy `BRIDGES` dict mixes configuration, runtime state, reflector naming, routing, and export concerns. FreeDMR 2 models each TG as a conference bridge to which clients subscribe.
|
||||
|
||||
## Decision
|
||||
Use subscription-oriented internal state instead of legacy `BRIDGES` as the authoritative FreeDMR 2 model.
|
||||
|
||||
## Rationale
|
||||
Subscription state directly represents the FreeDMR user model: clients subscribe to TGs they want to hear. It supports direct TGs, dial-a-TG, default reflectors, API control, and future aliases without hard-coding routing modes.
|
||||
|
||||
## Consequences
|
||||
The packet path can use indexed subscription lookups. Existing dashboard/config expectations need compatibility export or migration.
|
||||
|
||||
## Compatibility
|
||||
`BRIDGES` and `#` reflector names may be generated as compatibility/export state where required, but routing should not depend on them.
|
||||
|
||||
## Testing Requirements
|
||||
Tests must assert subscription activation, expiry, direct TG routing, dial-a-TG mapping, default reflector behaviour, and absence of unintended recipients.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0004: Reporting v2 Replaces Legacy Dashboard Protocol
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
The old dashboard/report socket has shaped parts of the current implementation and has caused operational friction. FreeDMR 1.x remains live while FreeDMR 2 is developed.
|
||||
|
||||
## Decision
|
||||
FreeDMR 2 reporting is a new structured event contract. The old dashboard/report socket is not a compatibility constraint inside the core.
|
||||
|
||||
## Rationale
|
||||
Reporting must be observational only. A clean event schema avoids leaking legacy `BRIDGES`/`SYSTEM` state into the new packet core.
|
||||
|
||||
## Consequences
|
||||
The dashboard must be updated or served by an adapter. The core can emit stable v2 events without preserving legacy report names.
|
||||
|
||||
## Compatibility
|
||||
Old dashboard support belongs in a sidecar or adapter, not in packet routing.
|
||||
|
||||
## Testing Requirements
|
||||
Tests must assert expected v2 events for server, client, subscription, call, mesh, loop, and reporting-health changes.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0005: MQTT Reporting Transport
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
FreeDMR needs live local dashboard state and non-real-time global lastheard feeds without blocking packet handling.
|
||||
|
||||
## Decision
|
||||
MQTT is the preferred external live reporting transport, fed through a non-blocking bounded queue and independent publisher.
|
||||
|
||||
## Rationale
|
||||
MQTT is lightweight, familiar in radio/network operations, supports topics, retained state, last-will messages, and network fanout. A local broker lets extra consumers attach without adding work to the packet process.
|
||||
|
||||
## Consequences
|
||||
FreeDMR gains a broker dependency or optional integration. Reporting completeness is best-effort under pressure.
|
||||
|
||||
## Compatibility
|
||||
Legacy dashboard consumers need an adapter or dashboard update. Packet routing must continue if MQTT or consumers are unavailable.
|
||||
|
||||
## Testing Requirements
|
||||
Tests must cover enqueue, overflow/drop policy, publisher disconnect/reconnect events, retained state refresh, and packet-path non-blocking behaviour.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0006: Local Dashboard and Global Lastheard Are Separate
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
Each server has its own live dashboard. The global lastheard service is centrally hosted and non-real-time.
|
||||
|
||||
## Decision
|
||||
Local dashboard consumes local live feed; global lastheard consumes curated summaries via exporter/collector.
|
||||
|
||||
## Rationale
|
||||
Local live visibility must survive central outages. Global aggregation should not add packet-process load or require the core to do database/export work.
|
||||
|
||||
## Consequences
|
||||
A separate exporter process may be needed for global feeds. The broker handles fanout.
|
||||
|
||||
## Compatibility
|
||||
Existing global lastheard behaviour should be migrated to consume summaries rather than packet-plane events.
|
||||
|
||||
## Testing Requirements
|
||||
Tests should confirm local reporting works without global exporter connectivity and that exporter failure does not affect packet handling.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0007: Synthetic LC Service Options
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
Legacy HBLink used `0x20` in synthetic LC service options. Later MMDVMHost evidence indicates `0x20` was an early OVCM bit-position mistake; standards-clean OVCM is `0x04`.
|
||||
|
||||
## Decision
|
||||
Synthetic group voice LC uses normal service options `0x00` by default. OVCM is `0x04` if explicitly selected. HBLink `0x20` is legacy compatibility only. Real inbound LC is preserved.
|
||||
|
||||
## Rationale
|
||||
FreeDMR should not set reserved service-option bits in newly generated LC. Real inbound LC should not be rewritten without a deliberate reason.
|
||||
|
||||
## Consequences
|
||||
Generated fallback LC becomes cleaner. Some legacy interop assumptions may need live RF testing.
|
||||
|
||||
## Compatibility
|
||||
`0x20` may remain as a named legacy compatibility option, not as the default or as a traffic marker.
|
||||
|
||||
## Testing Requirements
|
||||
Codec/unit tests must assert synthetic LC defaults to `0x00`, OVCM uses `0x04`, real inbound LC is preserved, and `0x20` is only used when explicitly configured for compatibility.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0008: Data Packet Forwarding Policy
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
FreeDMR currently supports DMR data forwarding, including data gateway concepts and last-known-location routing. GPS/SMS application processing is better handled outside the core.
|
||||
|
||||
## Decision
|
||||
FreeDMR continues forwarding DMR data packets but does not become a general GPS/SMS application processor.
|
||||
|
||||
## Rationale
|
||||
Data support is part of network interoperability. Application processing would add complexity and CPU cost to the packet process and is better done by FBP-connected systems or sidecars.
|
||||
|
||||
## Consequences
|
||||
The core must preserve data packet routing semantics while keeping application parsing out of the hot path.
|
||||
|
||||
## Compatibility
|
||||
Existing `SUB_MAP` / last-known-location behaviour remains intentional. `DATA-GATEWAY` remains a supported concept where useful.
|
||||
|
||||
## Testing Requirements
|
||||
Tests must cover group-addressed data, unit-addressed data, last-known-location routing, data-vs-voice reporting, and preservation of data forwarding over FBP.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0009: Mesh Authentication Without Default Encryption
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
FreeDMR is an amateur-radio network. In many jurisdictions amateur-radio traffic must not be encrypted, and IP backhaul may itself use amateur-radio links.
|
||||
|
||||
## Decision
|
||||
Use authenticity, integrity, membership validation, and local policy; do not encrypt amateur-radio mesh traffic by default.
|
||||
|
||||
## Rationale
|
||||
Signing and authentication protect the mesh from impersonation and unauthorized traffic while preserving FreeDMR's open, inspectable, amateur-radio character.
|
||||
|
||||
## Consequences
|
||||
Traffic remains visible. Security focuses on who is allowed to inject or carry traffic, not secrecy.
|
||||
|
||||
## Compatibility
|
||||
Existing cleartext FBP/OBP interop remains possible. New authenticated admission can be introduced through bridge-control mechanisms and cached session state.
|
||||
|
||||
## Testing Requirements
|
||||
Tests must cover valid identity, invalid signature, revocation, endpoint change requiring re-authentication, grace expiry, and local policy overriding signed membership.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0010: API Is Control Plane Only
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
The API is useful for local administration, automation, and dashboard control. FreeDMR is a live voice stream program, so API work must not delay packet processing.
|
||||
|
||||
## Decision
|
||||
API operations are bounded control-plane actions and must not perform heavy serialization or block packet routing.
|
||||
|
||||
## Rationale
|
||||
Control operations should act on sessions, subscriptions, mesh peers, and reporting state. Heavy views should come from snapshots or reporting feeds.
|
||||
|
||||
## Consequences
|
||||
The API remains small and predictable. Complex dashboard state should not be assembled synchronously from hot packet state.
|
||||
|
||||
## Compatibility
|
||||
Old API endpoints may be adapted if needed, but the v2 API should not expose raw legacy internals as its primary contract.
|
||||
|
||||
## Testing Requirements
|
||||
Tests must cover auth, bounded request bodies, destructive-action gating, audit logging, and packet-path independence under API load.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0011: Process/Actor Model Over No-GIL Threading
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
FreeDMR may need more concurrency for reporting, export, analysis, or future scaling. Shared mutable routing state is risky.
|
||||
|
||||
## Decision
|
||||
If FreeDMR 2 needs concurrency beyond the reactor, prefer explicit parent/child or actor-style ownership boundaries over shared-memory no-GIL threading for routing state.
|
||||
|
||||
## Rationale
|
||||
Single-owner state and explicit messages are easier for sysops and contributors to reason about. They reduce race risks in delay-sensitive packet handling.
|
||||
|
||||
## Consequences
|
||||
Some features may require serialization and message protocols between processes. This is clearer than shared locks around routing dictionaries.
|
||||
|
||||
## Compatibility
|
||||
Twisted can remain the initial transport shell while workers handle reporting/export or expensive tasks.
|
||||
|
||||
## Testing Requirements
|
||||
Tests must cover worker failure, queue overflow, restart behaviour, message ordering where required, and packet handling continuing when a non-critical worker fails.
|
||||
@ -0,0 +1,22 @@
|
||||
# ADR 0012: Testing Gates for Protocol-Visible Change
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
FreeDMR behaviour was validated through real global servers, RF links, and community use. Protocol-visible changes can affect repeaters, terminals, dashboards, and peer servers.
|
||||
|
||||
## Decision
|
||||
Protocol-visible changes require deterministic, UDP, and live RF validation according to release gates.
|
||||
|
||||
## Rationale
|
||||
Deterministic tests catch state and rewrite errors. UDP black-box tests catch transport/subprocess/protocol integration issues. Live RF catches terminal/repeater quirks that harnesses cannot prove.
|
||||
|
||||
## Consequences
|
||||
Some changes take longer to release. The risk of breaking real deployments is reduced.
|
||||
|
||||
## Compatibility
|
||||
FreeDMR 1.x remains live while FreeDMR 2 behaviour is validated. Changes can be staged behind compatibility adapters.
|
||||
|
||||
## Testing Requirements
|
||||
Level 0 unit/support tests, Level 1 deterministic harness, Level 2 UDP black-box harness, and Level 3 live RF validation are required according to the risk and protocol visibility of the change.
|
||||
@ -0,0 +1,87 @@
|
||||
# ADR 0013: Worker Process Capacity Scaling
|
||||
|
||||
## Status
|
||||
Proposed
|
||||
|
||||
## Context
|
||||
|
||||
FreeDMR currently relies heavily on a single Python process, Twisted reactor callbacks, and in-memory mutable state inherited from the HBLink-era architecture.
|
||||
|
||||
This is useful as an initial safety boundary because it avoids many shared-memory races, but it also creates practical capacity limits and makes some kinds of expensive work unsafe in the packet path.
|
||||
|
||||
CPython's GIL and single-reactor execution mean that CPU-bound work, reporting fanout, SQL/global export, analytics, and future codec/transcoding work should not be assumed to scale inside one process.
|
||||
|
||||
At the same time, FreeDMR's packet/routing state is subtle and protocol-sensitive. Moving it prematurely across process boundaries could introduce latency, ordering bugs, stale packet replay, duplicate packets, routing loops, broken source quench, or incorrect packet mutation.
|
||||
|
||||
## Decision
|
||||
|
||||
FreeDMR 2 will be designed for eventual worker-process scaling, but the first migration stage will keep Twisted as the transport shell and extract/test the routing/subscription core in-process.
|
||||
|
||||
The first worker-process targets are non-packet-path or low-risk side effects:
|
||||
|
||||
- Reporting/MQTT publisher.
|
||||
- Global lastheard exporter.
|
||||
- SQL/database writes.
|
||||
- Dashboard aggregation.
|
||||
- Alias refresh.
|
||||
- Analytics.
|
||||
- Packet replay/diagnostics.
|
||||
- Future codec/transcoding adjuncts.
|
||||
|
||||
Packet-plane routing may move behind a process/message boundary later, but only after state ownership, subscription lookup, stream lifecycle, loop control, source quench, and packet mutation semantics are covered by deterministic, UDP, and live RF tests.
|
||||
|
||||
FreeDMR 2 prefers explicit process/actor ownership boundaries over shared-memory threading or no-GIL Python for authoritative routing state.
|
||||
|
||||
## Rationale
|
||||
|
||||
- Worker processes provide clearer ownership and failure boundaries.
|
||||
- Explicit messages are easier to test than shared mutable dictionaries.
|
||||
- Reporting/export failures must not affect packet routing.
|
||||
- FreeDMR should be able to scale beyond one reactor process without making ordinary small deployments complex.
|
||||
- No-GIL Python does not remove the need for state ownership discipline.
|
||||
- A process model better matches FreeDMR's distributed-system nature: packet events, routing decisions, control messages, and reporting events are already conceptually separate.
|
||||
|
||||
## Consequences
|
||||
|
||||
Positive:
|
||||
|
||||
- Clearer state ownership.
|
||||
- Improved future capacity.
|
||||
- Safer isolation of reporting/export/database work.
|
||||
- Better failure containment.
|
||||
- Better testability of message boundaries.
|
||||
- Easier future sharding by client, TG, stream, listener, or mesh peer.
|
||||
|
||||
Negative:
|
||||
|
||||
- More implementation complexity.
|
||||
- Message schemas must be designed and versioned.
|
||||
- IPC adds latency and failure modes.
|
||||
- Routing-worker split requires strong tests.
|
||||
- Worker supervision and restart policy become part of the system design.
|
||||
|
||||
## Compatibility
|
||||
|
||||
FreeDMR 1.x/current code remains live until FreeDMR 2 is ready.
|
||||
|
||||
Initial FreeDMR 2 worker work should not change packet semantics.
|
||||
|
||||
Legacy dashboard/reporting compatibility, if required, belongs in reporting/export adapters, not in the packet core.
|
||||
|
||||
A single-process deployment must remain supported for small servers.
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
Before any packet-plane worker split:
|
||||
|
||||
- Deterministic harness coverage of the state machine.
|
||||
- Message-boundary equivalence tests.
|
||||
- UDP black-box equivalence tests.
|
||||
- Packet byte preservation tests.
|
||||
- Allowed rewrite-region tests.
|
||||
- Source quench/STUN/loop-control tests.
|
||||
- Duplicate/out-of-order tests.
|
||||
- Worker crash/restart tests.
|
||||
- Stale packet replay prevention tests.
|
||||
- Queue backpressure tests.
|
||||
- Live RF validation for protocol-visible behaviour.
|
||||
@ -0,0 +1,137 @@
|
||||
# FreeDMR 1.x Changelog
|
||||
|
||||
## Test Harnesses
|
||||
|
||||
- Added an in-process deterministic packet harness for `bridge_master.py`
|
||||
routing, state, expiry and packet rewrite checks without UDP.
|
||||
- Added a black-box UDP harness that starts FreeDMR with generated test configs,
|
||||
emulates HBP clients and FBP/OpenBridge peers, captures outbound UDP, supports
|
||||
venv bootstrap, and can model packet loss, duplicates and reordering.
|
||||
- Added synthetic and recorded packet fixture coverage for routing, slot rewrite,
|
||||
byte preservation, malformed packets, cadence and link impairment.
|
||||
|
||||
## Configuration and Options
|
||||
|
||||
- Hardened config parsing for booleans, alias stale time, missing session
|
||||
options and invalid numeric fields.
|
||||
- Added `DIAL_A_TG` to disable private-call dial-a-TG control.
|
||||
- Added `DYNAMIC_TG_ROUTING` to disable automatic creation of unknown
|
||||
conventional TG bridges.
|
||||
- Deprecated `DEFAULT_REFLECTOR` as the system default dial-a-TG setting, while
|
||||
keeping it as a TS2 compatibility alias.
|
||||
- Added canonical per-slot defaults: `DEFAULT_DIAL_TS1` and `DEFAULT_DIAL_TS2`.
|
||||
- Kept legacy OPTIONS aliases `DIAL`, `StartRef` and `DEFAULT_REFLECTOR` mapped
|
||||
to TS2; explicit `DEFAULT_DIAL_TS2` takes precedence.
|
||||
- Added validation/logging so invalid default dial values do not create bridge
|
||||
state and normalize to no default for the active runtime session.
|
||||
|
||||
## Dial-a-TG
|
||||
|
||||
- Made dial-a-TG private-call control slot-local: TS1 controls TS1, TS2 controls
|
||||
TS2. TS1 no longer retunes or disconnects TS2.
|
||||
- Preserved voice prompts on TG9 TS2.
|
||||
- Preserved TG9 as the RF-visible dial-a-TG talkgroup for both slots.
|
||||
- Rejected reserved/control targets consistently for live dial-a-TG and default
|
||||
startup/session configuration.
|
||||
- Preserved the current FreeDMR dial-a-TG policy cap of `999999`.
|
||||
- Ensured FBP route targets created for dial-a-TG remain active across local
|
||||
retunes/disconnects, in line with the mesh "everything everywhere" model.
|
||||
|
||||
## Data Path
|
||||
|
||||
- Preserved DMR data forwarding support.
|
||||
- Kept DATA-GATEWAY behavior for protocol-v1 SMS/GPS style handling.
|
||||
- Reported group-addressed data as data/control, not as voice lifecycle.
|
||||
- Suppressed false `GROUP VOICE` timeout reports for data/control packets.
|
||||
- Preserved data-sync/control payload bytes across HBP and FBP forwarding.
|
||||
- Kept `SUB_MAP` last-known-location behavior for unit data routed toward HBP.
|
||||
- Preserved FBP metadata such as source server, source repeater, BER, RSSI and
|
||||
hops according to protocol version.
|
||||
|
||||
## Voice Path
|
||||
|
||||
- Preserved real inbound LC where available and used explicit synthetic LC only
|
||||
as fallback.
|
||||
- Switched normal synthetic group voice LC service options to `0x00`; retained
|
||||
HBLink `0x20` only as an explicit legacy compatibility constant.
|
||||
- Reworked embedded LC handling so same-TG forwarding preserves embedded LC
|
||||
payloads where possible, while TG-mapped forwarding regenerates routing LC.
|
||||
- Added in-call Talker Alias and GPS embedded-LC logging without changing
|
||||
routing or packet mutation behavior.
|
||||
- Added generated prompt lifecycle handling so real RF voice can interrupt a
|
||||
prompt instead of being blocked as busy.
|
||||
- Fixed private dial-a-TG/AMI timeout reporting so private control calls do not
|
||||
emit unmatched group voice lifecycle events.
|
||||
- Made HBP and FBP voice sequence handling modulo-256 with explicit duplicate,
|
||||
loss and stale/out-of-order treatment.
|
||||
- Ensured voice terminators mark streams finished even when reporting is
|
||||
disabled, preventing late same-stream packets from reopening ended streams.
|
||||
|
||||
## Mesh and FBP/OpenBridge
|
||||
|
||||
- Added malformed/truncated `DMRD` and `DMRE` guards before fixed-offset parsing.
|
||||
- Corrected source-quench matching so BCSQ uses the TG namespace visible to the
|
||||
peer being quenched, including dial-a-TG reflector TGs.
|
||||
- Made STUN/BCST handling consistent as a broad FBP traffic gate.
|
||||
- Preserved protocol-version-sensitive FBP/OBP metadata layout.
|
||||
- Added tests for FBP keepalive gating, wrong network ID, bad hashes, stale
|
||||
timestamps, max-hop handling, v4 characterization and v1 refusal on v5 links.
|
||||
|
||||
## Reporting and Dashboard Compatibility
|
||||
|
||||
- Kept the legacy report socket opcode model unchanged.
|
||||
- Kept bridge event CSV field order unchanged.
|
||||
- Kept `DEFAULT_REFLECTOR` in runtime config as the effective TS2 default for
|
||||
compatibility with existing config/API/report consumers.
|
||||
- Kept prompt/ident generated audio visible as TG9 TS2.
|
||||
- The latest per-slot default-dial changes do not introduce new report event
|
||||
names or new report event fields.
|
||||
- Expected dashboard impact is low if the dashboard reads event fields and
|
||||
bridge entries by their existing keys.
|
||||
- Compatibility risk: `BRIDGE_SND` pickled bridge state may now include active
|
||||
TS1 `#reflector` entries. A dashboard that assumes every `#reflector` entry is
|
||||
TS2-only may need an update; a dashboard that already respects `TS`, `TGID`
|
||||
and `ACTIVE` should continue to parse it.
|
||||
- TS1 dial-a-TG activity may now appear as RF-visible TG9 on slot 1, which is
|
||||
intentional new behavior.
|
||||
|
||||
## Codec and Utility Cleanup
|
||||
|
||||
- Added `freedmr_dmr_codec.py` for locally tested DMR LC, embedded LC, slot type,
|
||||
BPTC, Hamming, Golay and RS parity helper behavior.
|
||||
- Moved runtime LC generation and byte/int helper usage away from older
|
||||
`dmrutils3` functions where covered.
|
||||
- Added standalone codec tests using fixed fixtures and MMDVMHost-style behavior.
|
||||
- Added focused utility tests for ID/byte helpers and alias lookup.
|
||||
|
||||
## API and Support Tools
|
||||
|
||||
- Replaced the Spyne-based API path with a small bounded HTTP/JSON control API.
|
||||
- Kept API operations as small in-memory control-plane actions.
|
||||
- Added API tests for request size limits, key validation, JSON responses,
|
||||
option storage and reset/kill behavior.
|
||||
- Tidied auxiliary tests for report receiver flags, SQL report insertion, AMI
|
||||
factory state and proxy environment booleans.
|
||||
|
||||
## Bridge.py Backports
|
||||
|
||||
- Backported only directly relevant, already-supported fixes from
|
||||
`bridge_master.py` to `bridge.py`.
|
||||
- Kept `bridge.py` focused on its existing conference-bridge role; did not add
|
||||
FreeDMR master-only features such as dial-a-TG.
|
||||
|
||||
## Documentation
|
||||
|
||||
- Added and updated harness, testing, API and architecture documentation.
|
||||
- Added FreeDMR 2 design/ADR documents separately, without changing current
|
||||
1.x runtime behavior.
|
||||
- Maintained `docs/codex-notes.md` as the engineering notebook for findings,
|
||||
assumptions, protocol-sensitive areas, invariants and unresolved questions.
|
||||
|
||||
## Validation
|
||||
|
||||
- Current non-UDP test discovery passes.
|
||||
- Focused UDP black-box tests for TS1 dial-a-TG and disabled dynamic TG routing
|
||||
pass when local UDP sockets are permitted.
|
||||
- Live RF validation is still required before treating protocol-visible behavior
|
||||
changes as release-ready.
|
||||
@ -0,0 +1,648 @@
|
||||
#!/usr/bin/env python
|
||||
#
|
||||
# Embedded LC codec helpers for FreeDMR.
|
||||
#
|
||||
# The embedded LC BPTC layout follows MMDVMHost CDMREmbeddedData,
|
||||
# CHamming::encode16114/decode16114, and CCRC::encodeFiveBit.
|
||||
# MMDVMHost is Copyright (C) Jonathan Naylor G4KLX and licensed GPLv2+
|
||||
# (or later). FreeDMR is GPLv3+, so this port is used under GPLv3+.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from bitarray import bitarray
|
||||
|
||||
|
||||
FULL_LC_BITS = 196
|
||||
FULL_LC_BYTES = 9
|
||||
FULL_LC_PARITY_BYTES = 3
|
||||
FULL_LC_CODE_BYTES = 12
|
||||
FULL_LC_INFO_BITS = 196
|
||||
EMBEDDED_LC_FRAGMENT_BITS = 32
|
||||
EMBEDDED_LC_RAW_BITS = 128
|
||||
EMBEDDED_LC_PAYLOAD_BITS = 72
|
||||
EMBEDDED_LC_PAYLOAD_BYTES = 9
|
||||
SLOT_TYPE_BITS = 20
|
||||
SLOT_TYPE_DATA_BITS = 8
|
||||
LC_FLCO_GROUP_VOICE = 0x00
|
||||
LC_FLCO_UNIT_VOICE = 0x03
|
||||
LC_FID_ETSI = 0x00
|
||||
LC_SERVICE_OPTIONS_NORMAL = 0x00
|
||||
LC_SERVICE_OPTIONS_OVCM = 0x04
|
||||
LC_SERVICE_OPTIONS_HBLINK_LEGACY = 0x20
|
||||
GROUP_VOICE_LC_OPT = b'\x00\x00\x00'
|
||||
UNIT_VOICE_LC_OPT = b'\x03\x00\x00'
|
||||
BS_VOICE_SYNC = bitarray('011101010101111111010111110111110111010111110111')
|
||||
BS_DATA_SYNC = bitarray('110111111111010101111101011101011101111101011101')
|
||||
EMB = {
|
||||
'BURST_B': bitarray('0001001110010001'),
|
||||
'BURST_C': bitarray('0001011101110100'),
|
||||
'BURST_D': bitarray('0001011101110100'),
|
||||
'BURST_E': bitarray('0001010100000111'),
|
||||
'BURST_F': bitarray('0001000111100010'),
|
||||
}
|
||||
SLOT_TYPE = {
|
||||
'PI_HEAD': bitarray('00010000001101100111'),
|
||||
'VOICE_LC_HEAD': bitarray('00010001101110001100'),
|
||||
'VOICE_LC_TERM': bitarray('00010010101001011001'),
|
||||
'CSBK': bitarray('00010011001010110010'),
|
||||
'MBC_HEAD': bitarray('00010100100111110000'),
|
||||
'MBC_CONT': bitarray('00010101000100011011'),
|
||||
'DATA_HEAD': bitarray('00010110000011001110'),
|
||||
'1/2_DATA': bitarray('00010111100000100101'),
|
||||
'3/4_DATA': bitarray('00011000111010100001'),
|
||||
'IDLE': bitarray('00011001011001001010'),
|
||||
'1/1_DATA': bitarray('00011010011110011111'),
|
||||
'RES_1': bitarray('00011011111101110100'),
|
||||
'RES_2': bitarray('00011100010000110110'),
|
||||
'RES_3': bitarray('00011101110011011101'),
|
||||
'RES_4': bitarray('00011110110100001000'),
|
||||
'RES_5': bitarray('00011111010111100011'),
|
||||
}
|
||||
|
||||
FULL_LC_INTERLEAVE_19696 = (
|
||||
0, 181, 166, 151, 136, 121, 106, 91, 76, 61, 46, 31, 16, 1, 182, 167, 152, 137,
|
||||
122, 107, 92, 77, 62, 47, 32, 17, 2, 183, 168, 153, 138, 123, 108, 93, 78, 63,
|
||||
48, 33, 18, 3, 184, 169, 154, 139, 124, 109, 94, 79, 64, 49, 34, 19, 4, 185, 170,
|
||||
155, 140, 125, 110, 95, 80, 65, 50, 35, 20, 5, 186, 171, 156, 141, 126, 111, 96,
|
||||
81, 66, 51, 36, 21, 6, 187, 172, 157, 142, 127, 112, 97, 82, 67, 52, 37, 22, 7,
|
||||
188, 173, 158, 143, 128, 113, 98, 83, 68, 53, 38, 23, 8, 189, 174, 159, 144, 129,
|
||||
114, 99, 84, 69, 54, 39, 24, 9, 190, 175, 160, 145, 130, 115, 100, 85, 70, 55, 40,
|
||||
25, 10, 191, 176, 161, 146, 131, 116, 101, 86, 71, 56, 41, 26, 11, 192, 177, 162,
|
||||
147, 132, 117, 102, 87, 72, 57, 42, 27, 12, 193, 178, 163, 148, 133, 118, 103, 88,
|
||||
73, 58, 43, 28, 13, 194, 179, 164, 149, 134, 119, 104, 89, 74, 59, 44, 29, 14,
|
||||
195, 180, 165, 150, 135, 120, 105, 90, 75, 60, 45, 30, 15,
|
||||
)
|
||||
|
||||
FULL_LC_PAYLOAD_INDEXES = (
|
||||
136, 121, 106, 91, 76, 61, 46, 31,
|
||||
152, 137, 122, 107, 92, 77, 62, 47, 32, 17, 2,
|
||||
123, 108, 93, 78, 63, 48, 33, 18, 3, 184, 169,
|
||||
94, 79, 64, 49, 34, 19, 4, 185, 170, 155, 140,
|
||||
65, 50, 35, 20, 5, 186, 171, 156, 141, 126, 111,
|
||||
36, 21, 6, 187, 172, 157, 142, 127, 112, 97, 82,
|
||||
7, 188, 173, 158, 143, 128, 113, 98, 83,
|
||||
)
|
||||
|
||||
RS129_HEADER_MASK = (0x96, 0x96, 0x96)
|
||||
RS129_TERMINATOR_MASK = (0x99, 0x99, 0x99)
|
||||
RS129_POLY = (64, 56, 14, 1)
|
||||
|
||||
GOLAY_2087_PARITY = (
|
||||
0x0000, 0xB08E, 0xE093, 0x501D, 0x70A9, 0xC027, 0x903A, 0x20B4,
|
||||
0x60DC, 0xD052, 0x804F, 0x30C1, 0x1075, 0xA0FB, 0xF0E6, 0x4068,
|
||||
0x7036, 0xC0B8, 0x90A5, 0x202B, 0x009F, 0xB011, 0xE00C, 0x5082,
|
||||
0x10EA, 0xA064, 0xF079, 0x40F7, 0x6043, 0xD0CD, 0x80D0, 0x305E,
|
||||
0xD06C, 0x60E2, 0x30FF, 0x8071, 0xA0C5, 0x104B, 0x4056, 0xF0D8,
|
||||
0xB0B0, 0x003E, 0x5023, 0xE0AD, 0xC019, 0x7097, 0x208A, 0x9004,
|
||||
0xA05A, 0x10D4, 0x40C9, 0xF047, 0xD0F3, 0x607D, 0x3060, 0x80EE,
|
||||
0xC086, 0x7008, 0x2015, 0x909B, 0xB02F, 0x00A1, 0x50BC, 0xE032,
|
||||
0x90D9, 0x2057, 0x704A, 0xC0C4, 0xE070, 0x50FE, 0x00E3, 0xB06D,
|
||||
0xF005, 0x408B, 0x1096, 0xA018, 0x80AC, 0x3022, 0x603F, 0xD0B1,
|
||||
0xE0EF, 0x5061, 0x007C, 0xB0F2, 0x9046, 0x20C8, 0x70D5, 0xC05B,
|
||||
0x8033, 0x30BD, 0x60A0, 0xD02E, 0xF09A, 0x4014, 0x1009, 0xA087,
|
||||
0x40B5, 0xF03B, 0xA026, 0x10A8, 0x301C, 0x8092, 0xD08F, 0x6001,
|
||||
0x2069, 0x90E7, 0xC0FA, 0x7074, 0x50C0, 0xE04E, 0xB053, 0x00DD,
|
||||
0x3083, 0x800D, 0xD010, 0x609E, 0x402A, 0xF0A4, 0xA0B9, 0x1037,
|
||||
0x505F, 0xE0D1, 0xB0CC, 0x0042, 0x20F6, 0x9078, 0xC065, 0x70EB,
|
||||
0xA03D, 0x10B3, 0x40AE, 0xF020, 0xD094, 0x601A, 0x3007, 0x8089,
|
||||
0xC0E1, 0x706F, 0x2072, 0x90FC, 0xB048, 0x00C6, 0x50DB, 0xE055,
|
||||
0xD00B, 0x6085, 0x3098, 0x8016, 0xA0A2, 0x102C, 0x4031, 0xF0BF,
|
||||
0xB0D7, 0x0059, 0x5044, 0xE0CA, 0xC07E, 0x70F0, 0x20ED, 0x9063,
|
||||
0x7051, 0xC0DF, 0x90C2, 0x204C, 0x00F8, 0xB076, 0xE06B, 0x50E5,
|
||||
0x108D, 0xA003, 0xF01E, 0x4090, 0x6024, 0xD0AA, 0x80B7, 0x3039,
|
||||
0x0067, 0xB0E9, 0xE0F4, 0x507A, 0x70CE, 0xC040, 0x905D, 0x20D3,
|
||||
0x60BB, 0xD035, 0x8028, 0x30A6, 0x1012, 0xA09C, 0xF081, 0x400F,
|
||||
0x30E4, 0x806A, 0xD077, 0x60F9, 0x404D, 0xF0C3, 0xA0DE, 0x1050,
|
||||
0x5038, 0xE0B6, 0xB0AB, 0x0025, 0x2091, 0x901F, 0xC002, 0x708C,
|
||||
0x40D2, 0xF05C, 0xA041, 0x10CF, 0x307B, 0x80F5, 0xD0E8, 0x6066,
|
||||
0x200E, 0x9080, 0xC09D, 0x7013, 0x50A7, 0xE029, 0xB034, 0x00BA,
|
||||
0xE088, 0x5006, 0x001B, 0xB095, 0x9021, 0x20AF, 0x70B2, 0xC03C,
|
||||
0x8054, 0x30DA, 0x60C7, 0xD049, 0xF0FD, 0x4073, 0x106E, 0xA0E0,
|
||||
0x90BE, 0x2030, 0x702D, 0xC0A3, 0xE017, 0x5099, 0x0084, 0xB00A,
|
||||
0xF062, 0x40EC, 0x10F1, 0xA07F, 0x80CB, 0x3045, 0x6058, 0xD0D6,
|
||||
)
|
||||
|
||||
SLOT_TYPE_NAMES = {
|
||||
0x0: "PI_HEAD",
|
||||
0x1: "VOICE_LC_HEAD",
|
||||
0x2: "VOICE_LC_TERM",
|
||||
0x3: "CSBK",
|
||||
0x4: "MBC_HEAD",
|
||||
0x5: "MBC_CONT",
|
||||
0x6: "DATA_HEAD",
|
||||
0x7: "1/2_RATE",
|
||||
0x8: "3/4_RATE",
|
||||
0x9: "IDLE",
|
||||
0xA: "1/1_RATE",
|
||||
0xB: "RES_1",
|
||||
0xC: "RES_2",
|
||||
0xD: "RES_3",
|
||||
0xE: "RES_4",
|
||||
0xF: "RES_5",
|
||||
}
|
||||
EMBEDDED_LC_PAYLOAD_RANGES = (
|
||||
(0, 11),
|
||||
(16, 27),
|
||||
(32, 42),
|
||||
(48, 58),
|
||||
(64, 74),
|
||||
(80, 90),
|
||||
(96, 106),
|
||||
)
|
||||
EMBEDDED_LC_CRC_POSITIONS = (42, 58, 74, 90, 106)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EmbeddedLC:
|
||||
data: bytes
|
||||
flco: int
|
||||
raw: bitarray
|
||||
corrected: int = 0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FullLC:
|
||||
data: bytes
|
||||
flco: int
|
||||
source_id: int
|
||||
target_id: int
|
||||
service_options: int
|
||||
is_group_call: bool
|
||||
is_unit_call: bool
|
||||
|
||||
|
||||
def build_group_voice_lc(
|
||||
dst_id: bytes,
|
||||
src_id: bytes,
|
||||
service_options: int = LC_SERVICE_OPTIONS_NORMAL,
|
||||
) -> bytes:
|
||||
if len(dst_id) != 3 or len(src_id) != 3:
|
||||
raise FullLCError("DMR LC target and source IDs must be three bytes")
|
||||
if service_options < 0 or service_options > 0xFF:
|
||||
raise FullLCError("DMR LC service options must fit in one byte")
|
||||
|
||||
return bytes([LC_FLCO_GROUP_VOICE, LC_FID_ETSI, service_options]) + dst_id + src_id
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SlotType:
|
||||
color_code: int
|
||||
data_type: int
|
||||
name: str
|
||||
corrected: int = 0
|
||||
|
||||
|
||||
class EmbeddedLCError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class FullLCError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class SlotTypeError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def bytes_to_bits(data: bytes) -> bitarray:
|
||||
bits = bitarray(endian="big")
|
||||
bits.frombytes(data)
|
||||
return bits
|
||||
|
||||
|
||||
def bits_to_int(bits: bitarray) -> int:
|
||||
value = 0
|
||||
for bit in bits:
|
||||
value = (value << 1) | int(bit)
|
||||
return value
|
||||
|
||||
|
||||
def _bits_from_int(value: int, length: int) -> bitarray:
|
||||
return bitarray((bool(value & (1 << bit)) for bit in range(length - 1, -1, -1)), endian="big")
|
||||
|
||||
|
||||
def _hamming_distance(a: int, b: int) -> int:
|
||||
return (a ^ b).bit_count()
|
||||
|
||||
|
||||
def _gf256_mul(left: int, right: int) -> int:
|
||||
result = 0
|
||||
while right:
|
||||
if right & 0x01:
|
||||
result ^= left
|
||||
right >>= 1
|
||||
left <<= 1
|
||||
if left & 0x100:
|
||||
left ^= 0x11D
|
||||
return result & 0xFF
|
||||
|
||||
|
||||
def encode_hamming_15113(data: bitarray) -> bitarray:
|
||||
if len(data) != 11:
|
||||
raise FullLCError("Hamming(15,11,3) input must be 11 bits")
|
||||
return bitarray((
|
||||
data[0] ^ data[1] ^ data[2] ^ data[3] ^ data[5] ^ data[7] ^ data[8],
|
||||
data[1] ^ data[2] ^ data[3] ^ data[4] ^ data[6] ^ data[8] ^ data[9],
|
||||
data[2] ^ data[3] ^ data[4] ^ data[5] ^ data[7] ^ data[9] ^ data[10],
|
||||
data[0] ^ data[1] ^ data[2] ^ data[4] ^ data[6] ^ data[7] ^ data[10],
|
||||
), endian="big")
|
||||
|
||||
|
||||
def encode_hamming_1393(data: bitarray) -> bitarray:
|
||||
if len(data) != 9:
|
||||
raise FullLCError("Hamming(13,9,3) input must be 9 bits")
|
||||
return bitarray((
|
||||
data[0] ^ data[1] ^ data[3] ^ data[5] ^ data[6],
|
||||
data[0] ^ data[1] ^ data[2] ^ data[4] ^ data[6] ^ data[7],
|
||||
data[0] ^ data[1] ^ data[2] ^ data[3] ^ data[5] ^ data[7] ^ data[8],
|
||||
data[0] ^ data[2] ^ data[4] ^ data[5] ^ data[8],
|
||||
), endian="big")
|
||||
|
||||
|
||||
def rs129_parity(data: bytes) -> bytes:
|
||||
if len(data) != FULL_LC_BYTES:
|
||||
raise FullLCError("RS(12,9) LC payload must be 9 bytes")
|
||||
|
||||
parity = [0x00, 0x00, 0x00]
|
||||
for byte in data:
|
||||
dbyte = byte ^ parity[2]
|
||||
parity[2] = parity[1] ^ _gf256_mul(RS129_POLY[2], dbyte)
|
||||
parity[1] = parity[0] ^ _gf256_mul(RS129_POLY[1], dbyte)
|
||||
parity[0] = _gf256_mul(RS129_POLY[0], dbyte)
|
||||
return bytes((parity[2], parity[1], parity[0]))
|
||||
|
||||
|
||||
def encode_full_lc_parity(data: bytes, terminator: bool = False) -> bytes:
|
||||
mask = RS129_TERMINATOR_MASK if terminator else RS129_HEADER_MASK
|
||||
return bytes(value ^ mask[index] for index, value in enumerate(rs129_parity(data)))
|
||||
|
||||
|
||||
def _encode_19696(data: bytes) -> bitarray:
|
||||
if len(data) != FULL_LC_CODE_BYTES:
|
||||
raise FullLCError("BPTC(196,96) input must be 12 bytes")
|
||||
|
||||
bits = bytes_to_bits(data)
|
||||
for _index in range(4):
|
||||
bits.insert(0, 0)
|
||||
|
||||
for index in range(9):
|
||||
start = (index * 15) + 1
|
||||
end = start + 11
|
||||
parity = encode_hamming_15113(bits[start:end])
|
||||
for pbit in range(4):
|
||||
bits.insert(end + pbit, parity[pbit])
|
||||
|
||||
for _index in range(60):
|
||||
bits.append(0)
|
||||
|
||||
column = bitarray(9, endian="big")
|
||||
for col in range(15):
|
||||
start = col + 1
|
||||
for index in range(9):
|
||||
column[index] = bits[start]
|
||||
start += 15
|
||||
parity = encode_hamming_1393(column)
|
||||
|
||||
target = 136 + col
|
||||
for pbit in range(4):
|
||||
bits[target] = parity[pbit]
|
||||
target += 15
|
||||
|
||||
return bits
|
||||
|
||||
|
||||
def interleave_19696(data: bitarray) -> bitarray:
|
||||
if len(data) != FULL_LC_BITS:
|
||||
raise FullLCError("BPTC(196,96) data must be 196 bits")
|
||||
interleaved = bitarray(FULL_LC_BITS, endian="big")
|
||||
for index in range(FULL_LC_BITS):
|
||||
interleaved[FULL_LC_INTERLEAVE_19696[index]] = data[index]
|
||||
return interleaved
|
||||
|
||||
|
||||
def encode_full_lc(data: bytes, terminator: bool = False) -> bitarray:
|
||||
if len(data) != FULL_LC_BYTES:
|
||||
raise FullLCError("full LC payload must be 9 bytes")
|
||||
code = data + encode_full_lc_parity(data, terminator=terminator)
|
||||
return interleave_19696(_encode_19696(code))
|
||||
|
||||
|
||||
def encode_header_lc(data: bytes) -> bitarray:
|
||||
return encode_full_lc(data, terminator=False)
|
||||
|
||||
|
||||
def encode_terminator_lc(data: bytes) -> bitarray:
|
||||
return encode_full_lc(data, terminator=True)
|
||||
|
||||
|
||||
def decode_full_lc(bits: bitarray) -> FullLC:
|
||||
if len(bits) != FULL_LC_INFO_BITS:
|
||||
raise FullLCError("full LC data must be 196 bits")
|
||||
payload = bitarray((bits[index] for index in FULL_LC_PAYLOAD_INDEXES), endian="big")
|
||||
data = payload.tobytes()
|
||||
flco = data[0] & 0x3F
|
||||
return FullLC(
|
||||
data=data,
|
||||
flco=flco,
|
||||
source_id=int.from_bytes(data[6:9], "big"),
|
||||
target_id=int.from_bytes(data[3:6], "big"),
|
||||
service_options=data[2],
|
||||
is_group_call=flco == LC_FLCO_GROUP_VOICE,
|
||||
is_unit_call=flco == LC_FLCO_UNIT_VOICE,
|
||||
)
|
||||
|
||||
|
||||
def _padded_bits_to_bytes(bits: bitarray) -> bytes:
|
||||
padded = bits.copy()
|
||||
add_bits = 8 - (len(padded) % 8)
|
||||
if add_bits < 8:
|
||||
for _bit in range(add_bits):
|
||||
padded.insert(0, 0)
|
||||
return padded.tobytes()
|
||||
|
||||
|
||||
def voice_head_term(payload: bytes) -> dict[str, object]:
|
||||
bits = bytes_to_bits(payload)
|
||||
info = bits[0:98] + bits[166:264]
|
||||
slot_type = bits[98:108] + bits[156:166]
|
||||
sync = bits[108:156]
|
||||
lc = decode_full_lc(info).data
|
||||
return {
|
||||
"LC": lc,
|
||||
"CC": _padded_bits_to_bytes(slot_type[0:4]),
|
||||
"DTYPE": _padded_bits_to_bytes(slot_type[4:8]),
|
||||
"SYNC": sync,
|
||||
}
|
||||
|
||||
|
||||
def voice_sync(payload: bytes) -> dict[str, object]:
|
||||
bits = bytes_to_bits(payload)
|
||||
return {
|
||||
"AMBE": [
|
||||
bits[0:72],
|
||||
bits[72:108] + bits[156:192],
|
||||
bits[192:264],
|
||||
],
|
||||
"SYNC": bits[108:156],
|
||||
}
|
||||
|
||||
|
||||
def voice(payload: bytes) -> dict[str, object]:
|
||||
bits = bytes_to_bits(payload)
|
||||
emb = bits[108:116] + bits[148:156]
|
||||
return {
|
||||
"AMBE": [
|
||||
bits[0:72],
|
||||
bits[72:108] + bits[156:192],
|
||||
bits[192:264],
|
||||
],
|
||||
"CC": _padded_bits_to_bytes(emb[0:4]),
|
||||
"LCSS": _padded_bits_to_bytes(emb[5:7]),
|
||||
"EMBED": bits[116:148],
|
||||
}
|
||||
|
||||
|
||||
def encode_slot_type(color_code: int, data_type: int) -> bitarray:
|
||||
if not 0 <= color_code <= 0x0F:
|
||||
raise SlotTypeError("slot color code must fit in four bits")
|
||||
if not 0 <= data_type <= 0x0F:
|
||||
raise SlotTypeError("slot data type must fit in four bits")
|
||||
|
||||
value = (color_code << 4) | data_type
|
||||
checksum = GOLAY_2087_PARITY[value]
|
||||
code = (value << 12) | ((checksum & 0xFF) << 4) | (checksum >> 12)
|
||||
return _bits_from_int(code, SLOT_TYPE_BITS)
|
||||
|
||||
|
||||
def decode_slot_type(bits: bitarray) -> SlotType:
|
||||
if len(bits) != SLOT_TYPE_BITS:
|
||||
raise SlotTypeError("slot type must be 20 bits")
|
||||
|
||||
observed = bits_to_int(bits)
|
||||
candidates = []
|
||||
for value in range(256):
|
||||
encoded = bits_to_int(encode_slot_type(value >> 4, value & 0x0F))
|
||||
candidates.append((_hamming_distance(observed, encoded), value))
|
||||
candidates.sort()
|
||||
|
||||
distance, value = candidates[0]
|
||||
if len(candidates) > 1 and candidates[1][0] == distance:
|
||||
raise SlotTypeError("ambiguous Golay(20,8,7) slot type")
|
||||
if distance > 3:
|
||||
raise SlotTypeError("slot type Golay(20,8,7) check failed")
|
||||
|
||||
data_type = value & 0x0F
|
||||
return SlotType(
|
||||
color_code=value >> 4,
|
||||
data_type=data_type,
|
||||
name=SLOT_TYPE_NAMES[data_type],
|
||||
corrected=distance,
|
||||
)
|
||||
|
||||
|
||||
def crc5(data: bytes | bitarray) -> int:
|
||||
bits = bytes_to_bits(data) if isinstance(data, bytes) else data
|
||||
if len(bits) != EMBEDDED_LC_PAYLOAD_BITS:
|
||||
raise EmbeddedLCError("embedded LC payload must be 72 bits")
|
||||
|
||||
total = 0
|
||||
for offset in range(0, EMBEDDED_LC_PAYLOAD_BITS, 8):
|
||||
total += bits_to_int(bits[offset:offset + 8])
|
||||
return total % 31
|
||||
|
||||
|
||||
def encode_hamming_16114(row: bitarray) -> None:
|
||||
if len(row) != 16:
|
||||
raise EmbeddedLCError("Hamming row must be 16 bits")
|
||||
|
||||
row[11] = row[0] ^ row[1] ^ row[2] ^ row[3] ^ row[5] ^ row[7] ^ row[8]
|
||||
row[12] = row[1] ^ row[2] ^ row[3] ^ row[4] ^ row[6] ^ row[8] ^ row[9]
|
||||
row[13] = row[2] ^ row[3] ^ row[4] ^ row[5] ^ row[7] ^ row[9] ^ row[10]
|
||||
row[14] = row[0] ^ row[1] ^ row[2] ^ row[4] ^ row[6] ^ row[7] ^ row[10]
|
||||
row[15] = row[0] ^ row[2] ^ row[5] ^ row[6] ^ row[8] ^ row[9] ^ row[10]
|
||||
|
||||
|
||||
def decode_hamming_16114(row: bitarray) -> bool:
|
||||
if len(row) != 16:
|
||||
raise EmbeddedLCError("Hamming row must be 16 bits")
|
||||
|
||||
c0 = row[0] ^ row[1] ^ row[2] ^ row[3] ^ row[5] ^ row[7] ^ row[8]
|
||||
c1 = row[1] ^ row[2] ^ row[3] ^ row[4] ^ row[6] ^ row[8] ^ row[9]
|
||||
c2 = row[2] ^ row[3] ^ row[4] ^ row[5] ^ row[7] ^ row[9] ^ row[10]
|
||||
c3 = row[0] ^ row[1] ^ row[2] ^ row[4] ^ row[6] ^ row[7] ^ row[10]
|
||||
c4 = row[0] ^ row[2] ^ row[5] ^ row[6] ^ row[8] ^ row[9] ^ row[10]
|
||||
|
||||
syndrome = 0
|
||||
syndrome |= 0x01 if c0 != row[11] else 0
|
||||
syndrome |= 0x02 if c1 != row[12] else 0
|
||||
syndrome |= 0x04 if c2 != row[13] else 0
|
||||
syndrome |= 0x08 if c3 != row[14] else 0
|
||||
syndrome |= 0x10 if c4 != row[15] else 0
|
||||
|
||||
corrections = {
|
||||
0x01: 11,
|
||||
0x02: 12,
|
||||
0x04: 13,
|
||||
0x08: 14,
|
||||
0x10: 15,
|
||||
0x19: 0,
|
||||
0x0B: 1,
|
||||
0x1F: 2,
|
||||
0x07: 3,
|
||||
0x0E: 4,
|
||||
0x15: 5,
|
||||
0x1A: 6,
|
||||
0x0D: 7,
|
||||
0x13: 8,
|
||||
0x16: 9,
|
||||
0x1C: 10,
|
||||
}
|
||||
|
||||
if syndrome == 0:
|
||||
return True
|
||||
if syndrome not in corrections:
|
||||
return False
|
||||
row[corrections[syndrome]] = not row[corrections[syndrome]]
|
||||
return True
|
||||
|
||||
|
||||
def encode_embedded_lc(lc: bytes | bitarray) -> tuple[bitarray, bitarray, bitarray, bitarray]:
|
||||
payload = bytes_to_bits(lc) if isinstance(lc, bytes) else lc.copy()
|
||||
if len(payload) != EMBEDDED_LC_PAYLOAD_BITS:
|
||||
raise EmbeddedLCError("embedded LC must be 9 bytes / 72 bits")
|
||||
|
||||
data = bitarray([0] * EMBEDDED_LC_RAW_BITS, endian="big")
|
||||
checksum = crc5(payload)
|
||||
data[106] = bool(checksum & 0x01)
|
||||
data[90] = bool(checksum & 0x02)
|
||||
data[74] = bool(checksum & 0x04)
|
||||
data[58] = bool(checksum & 0x08)
|
||||
data[42] = bool(checksum & 0x10)
|
||||
|
||||
position = 0
|
||||
for start, end in EMBEDDED_LC_PAYLOAD_RANGES:
|
||||
data[start:end] = payload[position:position + (end - start)]
|
||||
position += end - start
|
||||
|
||||
for row_start in range(0, 112, 16):
|
||||
row = data[row_start:row_start + 16]
|
||||
encode_hamming_16114(row)
|
||||
data[row_start:row_start + 16] = row
|
||||
|
||||
for column in range(16):
|
||||
data[column + 112] = (
|
||||
data[column + 0]
|
||||
^ data[column + 16]
|
||||
^ data[column + 32]
|
||||
^ data[column + 48]
|
||||
^ data[column + 64]
|
||||
^ data[column + 80]
|
||||
^ data[column + 96]
|
||||
)
|
||||
|
||||
raw = bitarray([0] * EMBEDDED_LC_RAW_BITS, endian="big")
|
||||
position = 0
|
||||
for index in range(EMBEDDED_LC_RAW_BITS):
|
||||
raw[index] = data[position]
|
||||
position += 16
|
||||
if position > 127:
|
||||
position -= 127
|
||||
|
||||
return tuple(
|
||||
raw[index:index + EMBEDDED_LC_FRAGMENT_BITS]
|
||||
for index in range(0, EMBEDDED_LC_RAW_BITS, EMBEDDED_LC_FRAGMENT_BITS)
|
||||
)
|
||||
|
||||
|
||||
def encode_emblc(lc: bytes | bitarray) -> dict[int, bitarray]:
|
||||
fragments = encode_embedded_lc(lc)
|
||||
return {
|
||||
1: fragments[0],
|
||||
2: fragments[1],
|
||||
3: fragments[2],
|
||||
4: fragments[3],
|
||||
}
|
||||
|
||||
|
||||
def decode_embedded_lc(fragments: tuple[bitarray, bitarray, bitarray, bitarray] | list[bitarray]) -> EmbeddedLC:
|
||||
if len(fragments) != 4:
|
||||
raise EmbeddedLCError("embedded LC requires four fragments")
|
||||
|
||||
raw = bitarray(endian="big")
|
||||
for fragment in fragments:
|
||||
if len(fragment) != EMBEDDED_LC_FRAGMENT_BITS:
|
||||
raise EmbeddedLCError("embedded LC fragments must be 32 bits")
|
||||
raw += fragment
|
||||
|
||||
data = bitarray([0] * EMBEDDED_LC_RAW_BITS, endian="big")
|
||||
position = 0
|
||||
for index in range(EMBEDDED_LC_RAW_BITS):
|
||||
data[position] = raw[index]
|
||||
position += 16
|
||||
if position > 127:
|
||||
position -= 127
|
||||
|
||||
corrected = 0
|
||||
for row_start in range(0, 112, 16):
|
||||
row = data[row_start:row_start + 16]
|
||||
before = row.copy()
|
||||
if not decode_hamming_16114(row):
|
||||
raise EmbeddedLCError("embedded LC Hamming check failed")
|
||||
if row != before:
|
||||
corrected += 1
|
||||
data[row_start:row_start + 16] = row
|
||||
|
||||
for column in range(16):
|
||||
parity = (
|
||||
data[column + 0]
|
||||
^ data[column + 16]
|
||||
^ data[column + 32]
|
||||
^ data[column + 48]
|
||||
^ data[column + 64]
|
||||
^ data[column + 80]
|
||||
^ data[column + 96]
|
||||
^ data[column + 112]
|
||||
)
|
||||
if parity:
|
||||
raise EmbeddedLCError("embedded LC column parity check failed")
|
||||
|
||||
payload = bitarray(endian="big")
|
||||
for start, end in EMBEDDED_LC_PAYLOAD_RANGES:
|
||||
payload += data[start:end]
|
||||
|
||||
checksum = 0
|
||||
if data[42]:
|
||||
checksum += 16
|
||||
if data[58]:
|
||||
checksum += 8
|
||||
if data[74]:
|
||||
checksum += 4
|
||||
if data[90]:
|
||||
checksum += 2
|
||||
if data[106]:
|
||||
checksum += 1
|
||||
|
||||
if crc5(payload) != checksum:
|
||||
raise EmbeddedLCError("embedded LC 5-bit checksum failed")
|
||||
|
||||
decoded = payload.tobytes()
|
||||
return EmbeddedLC(data=decoded, flco=decoded[0] & 0x3F, raw=raw, corrected=corrected)
|
||||
|
||||
|
||||
def embedded_lc_fragment_from_payload(payload: bytes) -> bitarray:
|
||||
if len(payload) != 33:
|
||||
raise EmbeddedLCError("DMR payload must be 33 bytes")
|
||||
bits = bytes_to_bits(payload)
|
||||
return bits[116:148]
|
||||
|
||||
|
||||
def payload_with_embedded_lc_fragment(payload: bytes, fragment: bitarray) -> bytes:
|
||||
if len(payload) != 33:
|
||||
raise EmbeddedLCError("DMR payload must be 33 bytes")
|
||||
if len(fragment) != EMBEDDED_LC_FRAGMENT_BITS:
|
||||
raise EmbeddedLCError("embedded LC fragment must be 32 bits")
|
||||
bits = bytes_to_bits(payload)
|
||||
bits = bits[:116] + fragment + bits[148:264]
|
||||
return bits.tobytes()
|
||||
@ -0,0 +1,265 @@
|
||||
import unittest
|
||||
|
||||
from freedmr_dmr_codec import (
|
||||
EmbeddedLCError,
|
||||
FullLCError,
|
||||
LC_SERVICE_OPTIONS_HBLINK_LEGACY,
|
||||
LC_SERVICE_OPTIONS_NORMAL,
|
||||
build_group_voice_lc,
|
||||
SlotTypeError,
|
||||
decode_embedded_lc,
|
||||
decode_full_lc,
|
||||
decode_slot_type,
|
||||
embedded_lc_fragment_from_payload,
|
||||
encode_embedded_lc,
|
||||
encode_full_lc,
|
||||
encode_full_lc_parity,
|
||||
encode_slot_type,
|
||||
payload_with_embedded_lc_fragment,
|
||||
rs129_parity,
|
||||
voice,
|
||||
voice_head_term,
|
||||
)
|
||||
|
||||
|
||||
GROUP_VOICE_LC = bytes.fromhex("000000000c302f9be5")
|
||||
HEADER_LC_BITS = (
|
||||
"000010111100001000000100110101100001111000001100001010111001100000001000"
|
||||
"010100000111010001100001000000000000001011110100100001110110011100000000"
|
||||
"0000000000000010111000000000111111001000010110100111"
|
||||
)
|
||||
TERMINATOR_LC_BITS = (
|
||||
"000010111010110100000100000000100001111010111100001010111110000000001000"
|
||||
"001000000111010011000001000100010100001011000111000001110001000100000000"
|
||||
"1100010000000010011000000000111001011000010110010100"
|
||||
)
|
||||
EMBEDDED_LC_BITS = (
|
||||
"00001111000011110000011000000110",
|
||||
"00010111000100010000000000000110",
|
||||
"00001100000000110011010100011011",
|
||||
"00010111001101100010001000100010",
|
||||
)
|
||||
VOICE_HEAD_TERM_PAYLOAD = bytes.fromhex(
|
||||
"2b6004101f842dd00df07d41046dff57d75df5de30152e2070b20f803f88c695e2"
|
||||
)
|
||||
VOICE_BURST_PAYLOAD = bytes.fromhex(
|
||||
"b9e881526173002a6bb9e881526134e0f060691173002a6bb9e881526173002a6a"
|
||||
)
|
||||
|
||||
|
||||
class FreeDmrDmrCodecTest(unittest.TestCase):
|
||||
def test_full_lc_header_encode_matches_current_codec_fixture(self):
|
||||
lc = GROUP_VOICE_LC
|
||||
|
||||
encoded = encode_full_lc(lc)
|
||||
|
||||
self.assertEqual(encoded.to01(), HEADER_LC_BITS)
|
||||
|
||||
def test_full_lc_terminator_encode_matches_current_codec_fixture(self):
|
||||
lc = GROUP_VOICE_LC
|
||||
|
||||
encoded = encode_full_lc(lc, terminator=True)
|
||||
|
||||
self.assertEqual(encoded.to01(), TERMINATOR_LC_BITS)
|
||||
|
||||
def test_embedded_lc_group_voice_encode_matches_current_codec(self):
|
||||
lc = GROUP_VOICE_LC
|
||||
|
||||
encoded = encode_embedded_lc(lc)
|
||||
|
||||
self.assertEqual(tuple(fragment.to01() for fragment in encoded), EMBEDDED_LC_BITS)
|
||||
|
||||
def test_current_runtime_compatibility_function_names(self):
|
||||
import freedmr_dmr_codec as dmr_codec
|
||||
|
||||
lc = GROUP_VOICE_LC
|
||||
|
||||
self.assertEqual(dmr_codec.encode_header_lc(lc).to01(), HEADER_LC_BITS)
|
||||
self.assertEqual(dmr_codec.encode_terminator_lc(lc).to01(), TERMINATOR_LC_BITS)
|
||||
self.assertEqual(
|
||||
tuple(dmr_codec.encode_emblc(lc)[index].to01() for index in (1, 2, 3, 4)),
|
||||
EMBEDDED_LC_BITS,
|
||||
)
|
||||
|
||||
def test_voice_head_term_decode_matches_current_codec(self):
|
||||
decoded = voice_head_term(VOICE_HEAD_TERM_PAYLOAD)
|
||||
|
||||
self.assertEqual(decoded["LC"], bytes.fromhex("001020000c302f9be5"))
|
||||
self.assertEqual(decoded["CC"], b"\x01")
|
||||
self.assertEqual(decoded["DTYPE"], b"\x01")
|
||||
self.assertEqual(decoded["SYNC"].to01(), "110111111111010101111101011101011101111101011101")
|
||||
|
||||
def test_voice_burst_decode_matches_current_codec(self):
|
||||
decoded = voice(VOICE_BURST_PAYLOAD)
|
||||
|
||||
self.assertEqual(
|
||||
[ambe.to01() for ambe in decoded["AMBE"]],
|
||||
[
|
||||
"101110011110100010000001010100100110000101110011000000000010101001101011",
|
||||
"101110011110100010000001010100100110000101110011000000000010101001101011",
|
||||
"101110011110100010000001010100100110000101110011000000000010101001101010",
|
||||
],
|
||||
)
|
||||
self.assertEqual(decoded["CC"], b"\x01")
|
||||
self.assertEqual(decoded["LCSS"], b"\x01")
|
||||
self.assertEqual(decoded["EMBED"].to01(), "01001110000011110000011000000110")
|
||||
|
||||
def test_full_lc_decode_classifies_group_voice(self):
|
||||
lc = GROUP_VOICE_LC
|
||||
|
||||
decoded = decode_full_lc(encode_full_lc(lc))
|
||||
|
||||
self.assertEqual(decoded.data, lc)
|
||||
self.assertEqual(decoded.flco, 0x00)
|
||||
self.assertTrue(decoded.is_group_call)
|
||||
self.assertFalse(decoded.is_unit_call)
|
||||
self.assertEqual(decoded.target_id, 3120)
|
||||
self.assertEqual(decoded.source_id, 3120101)
|
||||
|
||||
def test_build_group_voice_lc_defaults_to_normal_service_options(self):
|
||||
lc = build_group_voice_lc(bytes.fromhex("00005b"), bytes.fromhex("2f9be5"))
|
||||
|
||||
self.assertEqual(lc, bytes.fromhex("00000000005b2f9be5"))
|
||||
self.assertEqual(decode_full_lc(encode_full_lc(lc)).service_options, LC_SERVICE_OPTIONS_NORMAL)
|
||||
|
||||
def test_build_group_voice_lc_can_represent_legacy_hblink_options_explicitly(self):
|
||||
lc = build_group_voice_lc(
|
||||
bytes.fromhex("00005b"),
|
||||
bytes.fromhex("2f9be5"),
|
||||
service_options=LC_SERVICE_OPTIONS_HBLINK_LEGACY,
|
||||
)
|
||||
|
||||
self.assertEqual(lc, bytes.fromhex("00002000005b2f9be5"))
|
||||
self.assertEqual(decode_full_lc(encode_full_lc(lc)).service_options, LC_SERVICE_OPTIONS_HBLINK_LEGACY)
|
||||
|
||||
def test_full_lc_decode_classifies_unit_voice(self):
|
||||
lc = bytes.fromhex("030000000c302f9be5")
|
||||
|
||||
decoded = decode_full_lc(encode_full_lc(lc))
|
||||
|
||||
self.assertEqual(decoded.flco, 0x03)
|
||||
self.assertFalse(decoded.is_group_call)
|
||||
self.assertTrue(decoded.is_unit_call)
|
||||
|
||||
def test_full_lc_rs129_parity_matches_header_and_terminator_masks(self):
|
||||
lc = bytes.fromhex("001020000c302f9be5")
|
||||
|
||||
self.assertEqual(rs129_parity(lc), bytes.fromhex("4c42cc"))
|
||||
self.assertEqual(encode_full_lc_parity(lc), bytes.fromhex("dad45a"))
|
||||
self.assertEqual(encode_full_lc_parity(lc, terminator=True), bytes.fromhex("d5db55"))
|
||||
|
||||
def test_full_lc_requires_nine_byte_lc(self):
|
||||
with self.assertRaises(FullLCError):
|
||||
encode_full_lc(b"\x00" * 8)
|
||||
|
||||
def test_slot_type_encode_matches_current_codec_fixtures(self):
|
||||
self.assertEqual(encode_slot_type(1, 0x1).to01(), "00010001101110001100")
|
||||
self.assertEqual(encode_slot_type(1, 0x2).to01(), "00010010101001011001")
|
||||
self.assertEqual(encode_slot_type(1, 0x3).to01(), "00010011001010110010")
|
||||
self.assertEqual(encode_slot_type(1, 0x6).to01(), "00010110000011001110")
|
||||
|
||||
def test_slot_type_decode_corrects_three_bits(self):
|
||||
bits = encode_slot_type(1, 0x2)
|
||||
bits[0] = not bits[0]
|
||||
bits[7] = not bits[7]
|
||||
bits[19] = not bits[19]
|
||||
|
||||
decoded = decode_slot_type(bits)
|
||||
|
||||
self.assertEqual(decoded.color_code, 1)
|
||||
self.assertEqual(decoded.data_type, 0x2)
|
||||
self.assertEqual(decoded.name, "VOICE_LC_TERM")
|
||||
self.assertEqual(decoded.corrected, 3)
|
||||
|
||||
def test_slot_type_decode_rejects_uncorrectable_error(self):
|
||||
bits = encode_slot_type(1, 0x2)
|
||||
for index in (0, 1, 2, 3):
|
||||
bits[index] = not bits[index]
|
||||
|
||||
with self.assertRaises(SlotTypeError):
|
||||
decode_slot_type(bits)
|
||||
|
||||
def test_slot_type_rejects_values_outside_four_bits(self):
|
||||
with self.assertRaises(SlotTypeError):
|
||||
encode_slot_type(16, 0x1)
|
||||
with self.assertRaises(SlotTypeError):
|
||||
encode_slot_type(1, 16)
|
||||
|
||||
def test_embedded_lc_round_trips_talker_alias_header(self):
|
||||
lc = bytes.fromhex("04004c43414c4c3132")
|
||||
|
||||
encoded = encode_embedded_lc(lc)
|
||||
decoded = decode_embedded_lc(encoded)
|
||||
|
||||
self.assertEqual(decoded.data, lc)
|
||||
self.assertEqual(decoded.flco, 0x04)
|
||||
self.assertEqual(decoded.corrected, 0)
|
||||
|
||||
def test_embedded_lc_round_trips_gps_info(self):
|
||||
lc = bytes.fromhex("080007fcfae048b57b")
|
||||
|
||||
encoded = encode_embedded_lc(lc)
|
||||
decoded = decode_embedded_lc(encoded)
|
||||
|
||||
self.assertEqual(decoded.data, lc)
|
||||
self.assertEqual(decoded.flco, 0x08)
|
||||
|
||||
def test_encoded_talker_alias_fragments_match_mmdvmhost_layout_fixture(self):
|
||||
lc = bytes.fromhex("04004c43414c4c3132")
|
||||
payload = b"\x55" * 33
|
||||
|
||||
encoded_payloads = [
|
||||
payload_with_embedded_lc_fragment(payload, fragment).hex()
|
||||
for fragment in encode_embedded_lc(lc)
|
||||
]
|
||||
|
||||
self.assertEqual(
|
||||
encoded_payloads,
|
||||
[
|
||||
"555555555555555555555555555550517092855555555555555555555555555555",
|
||||
"555555555555555555555555555550382441d55555555555555555555555555555",
|
||||
"5555555555555555555555555555522717b8e55555555555555555555555555555",
|
||||
"5555555555555555555555555555522b73ae255555555555555555555555555555",
|
||||
],
|
||||
)
|
||||
|
||||
def test_payload_fragment_helpers_extract_and_apply_embedded_lc(self):
|
||||
lc = bytes.fromhex("080007fcfae048b57b")
|
||||
fragment = encode_embedded_lc(lc)[0]
|
||||
original = b"\x55" * 33
|
||||
|
||||
updated = payload_with_embedded_lc_fragment(original, fragment)
|
||||
extracted = embedded_lc_fragment_from_payload(updated)
|
||||
|
||||
self.assertEqual(extracted, fragment)
|
||||
self.assertEqual(updated[:14], original[:14])
|
||||
self.assertEqual(updated[19:], original[19:])
|
||||
|
||||
def test_embedded_lc_decode_corrects_single_bit_error(self):
|
||||
lc = bytes.fromhex("04004c43414c4c3132")
|
||||
fragments = list(encode_embedded_lc(lc))
|
||||
fragments[0] = fragments[0].copy()
|
||||
fragments[0][0] = not fragments[0][0]
|
||||
|
||||
decoded = decode_embedded_lc(fragments)
|
||||
|
||||
self.assertEqual(decoded.data, lc)
|
||||
self.assertEqual(decoded.corrected, 1)
|
||||
|
||||
def test_embedded_lc_decode_rejects_uncorrectable_error(self):
|
||||
lc = bytes.fromhex("04004c43414c4c3132")
|
||||
fragments = list(encode_embedded_lc(lc))
|
||||
fragments[0] = fragments[0].copy()
|
||||
fragments[0][0] = not fragments[0][0]
|
||||
fragments[0][8] = not fragments[0][8]
|
||||
|
||||
with self.assertRaises(EmbeddedLCError):
|
||||
decode_embedded_lc(fragments)
|
||||
|
||||
def test_embedded_lc_requires_nine_byte_lc(self):
|
||||
with self.assertRaises(EmbeddedLCError):
|
||||
encode_embedded_lc(b"\x00" * 8)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@ -0,0 +1,24 @@
|
||||
import unittest
|
||||
|
||||
from utils import bytes_3, bytes_4, get_alias, int_id
|
||||
|
||||
|
||||
class FreeDmrUtilsTest(unittest.TestCase):
|
||||
def test_integer_byte_helpers_are_big_endian(self):
|
||||
self.assertEqual(bytes_3(0x010203), b"\x01\x02\x03")
|
||||
self.assertEqual(bytes_4(0x01020304), b"\x01\x02\x03\x04")
|
||||
self.assertEqual(int_id(b"\x01\x02\x03\x04"), 0x01020304)
|
||||
|
||||
def test_get_alias_returns_matching_record_or_original_id(self):
|
||||
aliases = {
|
||||
3120001: {"callsign": "M0ABC", "name": "Test"},
|
||||
235: "TG235",
|
||||
}
|
||||
|
||||
self.assertEqual(get_alias(bytes_3(3120001), aliases, "callsign"), ["M0ABC"])
|
||||
self.assertEqual(get_alias(235, aliases), "TG235")
|
||||
self.assertEqual(get_alias(999, aliases), 999)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Reference in new issue