FreeDMR/docs/hdstack.md

# HDStack multi-process deployment

HDStack raises the aggregate hotspot capacity of one FreeDMR container by
running up to five independent FreeDMR HBP workers. It does not raise the
connection limit of an individual worker. The default worker count is the
available CPU count minus one, clamped to the supported range of one through
five.

Omitting `HDSTACK` selects that CPU-derived default. Setting `HDSTACK=1`
explicitly preserves the existing standard single-FreeDMR process layout.
Setting `HDSTACK` to a value from 2 through 5 selects the generated
multi-process layout:

```text
hotspots -> proxy -> HBP workers -> no-HBP aggregator -> external FBP/OBP
                       |              |
                       +-> Loro       +-> optional bridge.py over FBP v5/FBCP
                       +-> Data Gateway over FBP v5/FBCP

worker reports -> reporting MUX -> existing dashboard/reporting consumer
```

The aggregator owns all OpenBridge/FBP links from the main `freedmr.cfg` and
has no HBP `MASTER`, `PEER` or `XLXPEER` system. The optional `bridge.py`
process retains its own configuration and rules. The reporting MUX reads only
worker reports; aggregator and bridge reporting are not included.

## Installer and container configuration

The maintained installer asks for the main server ID, proposes the CPU-derived
worker count, and asks whether the separate bridge process is required. It
writes the resulting values directly into `/etc/freedmr/docker-compose.yml`.

```yaml
environment:
  - HDSTACK=5
  - HDSTACK_BASEID=23400
  - HDSTACK_BRIDGE=0
```

`HDSTACK_BASEID` is the aggregator server ID. Worker IDs are allocated by
adding their one-based worker number. For the example above, the aggregator is
`23400` and worker IDs continue from `23401` through `23405`.

The separate `bridge.py` process is disabled by default. Enable it when the
deployment requires bridge rules or HBP peer/XLX connections:

```yaml
environment:
  - HDSTACK_BRIDGE=1
```

`HDSTACK_BRIDGE` accepts only `0` or `1`. It affects only multi-process mode.
When disabled, no `bridge.py` process is started.

The installer provides `freedmr-bridge.cfg` and `rules-bridge.py` skeletons.
The latter contains the required `BRIDGES = {}` structure, so the optional
process starts idle until routing rules are configured. Enabling the bridge
also requires a separate bridge server ID which must not collide with the
aggregator or worker IDs.

The standard Compose file mounts these inputs:

```yaml
volumes:
  - '/etc/freedmr/freedmr.cfg:/opt/freedmr/freedmr.cfg'
  - '/etc/freedmr/freedmr-bridge.cfg:/opt/freedmr/freedmr-bridge.cfg:ro'
  - '/etc/freedmr/rules-bridge.py:/opt/freedmr/rules.py:ro'
```

The paths can be changed with:

```yaml
environment:
  - HDSTACK_BRIDGE_CONFIG=/opt/freedmr/freedmr-bridge.cfg
  - HDSTACK_BRIDGE_RULES=/opt/freedmr/rules.py
```

The main `freedmr.cfg` remains the sysop-facing aggregator and worker-template
configuration. It must contain one enabled `[SYSTEM]` stanza in `MASTER` mode.
All enabled and disabled `OPENBRIDGE` sections except `[DATA-GATEWAY]` are
copied to the aggregator; this includes external FBP/OBP links. Other protocol
systems, including `[SYSTEM]` and `[ECHO]`, are not copied to the aggregator.

Each worker receives the same `[SYSTEM]` policy. Only its base HBP port is
derived. When the native Data Gateway is enabled, each worker also receives a
direct `[DATA-GATEWAY]` FBP v5/FBCP relationship. Worker and aggregator
configuration files are generated privately in `/dev/shm/freedmr-hdstack` when
the container starts. The mounted source configuration and bridge files are
not changed.

When `[ECHO]` is enabled, each worker receives its own generated ECHO peer and
Loro playback process on a private loopback port pair. Calls cannot leak
between worker playback state, and each Loro accepts only its corresponding
worker.

When enabled, the bridge configuration supplies its distinct server ID and any
`PEER` or `XLXPEER` systems used by `bridge.py`. HDStack generates the
reciprocal private FBP v5/FBCP connection to the aggregator. Its `rules.py` remains
the authority for bridge routing. Bridge reporting must either be disabled or
use a port which does not collide with the worker reports or MUX.

## Worker ports

The `[SYSTEM]` `PORT` and `GENERATOR` values define each worker range. With the
standard values `PORT=54000` and `GENERATOR=100`:

```text
worker 1 -> 54000..54099
worker 2 -> 54100..54199
worker 3 -> 54200..54299
worker 4 -> 54300..54399
worker 5 -> 54400..54499
```

The generated ranges must fit within the UDP port range and must not collide
with configured bridge, external OpenBridge or reporting ports.

The internal worker path consumes one additional FBP hop. FreeDMR accepts a
maximum of 20 FBP hops before discarding traffic; this ceiling applies to all
FBP relationships rather than being a special HDStack exception.

Workers connect to the aggregator over generated loopback FBP v5 links with
FBCP enabled. FBP v5 preserves origin metadata and carries unit data as well as
group traffic; FBCP provides source quench and the existing link-control
behaviour. Strict reciprocal endpoint checks remain enabled because both ends
use fixed loopback addresses and ports. Worker ports are `7001` through
`7005`; reciprocal aggregator ports are `7101` through `7105`. These links
are private to the container.

The generated section names retain the historical `OBP-HDSTACK-*` prefix even
though their configured wire version is FBP v5. Existing
`bridge_master.py` dynamic bridge construction recognises that prefix.

When the optional bridge process is enabled, it connects to the aggregator over
the same strict private FBP v5/FBCP profile. The aggregator binds UDP port
`7200` and the bridge binds UDP port `7201`. The generated peer identity on
each side is the other process's configured server ID. External FBP and OBP
relationships remain on the aggregator.

Each generated Loro pair uses successive private ports beginning with the
existing `[ECHO]` peer and master ports, normally `54916` and `54915`.

## Data Gateway

New installations include a separate FreeDMR Data Gateway container for
D-APRS and future packet-data services. The installer asks for the APRS-IS
login callsign and passcode, generates a private FBP key, enables the
`[DATA-GATEWAY]` relationship, and writes matching relationships to
`/etc/freedmr/data-gateway-relationships.json`. The Compose service mounts
that document read-only. During the compatibility transition the same value
also remains in the established environment setting, allowing the current
stable Data Gateway image to run while development images verify that both
forms describe the same relationships. Existing configurations keep
`DATA_GATEWAY: False` and the disabled section by default, so replacing only
the FreeDMR image does not remove the legacy D-APRS path.

Each worker connects directly to the gateway. The aggregator and optional
`bridge.py` process do not. The default ports are:

```text
worker 1 62041 -> gateway 62031
worker 2 62042 -> gateway 62032
worker 3 62043 -> gateway 62033
worker 4 62044 -> gateway 62034
worker 5 62045 -> gateway 62035
```

FreeDMR sends every admitted, locally originated HBP DMR packet to this link,
including unit data, group data and voice packets. Peer or mesh ingress is not
copied back to the gateway. The complete 33-byte DMR burst is preserved; the
gateway owns decoding, APRS consent and application policy. Enabling the native
gateway disables legacy destination-`900999` D-APRS delivery to avoid duplicate
processing. It is not an automatic fallback if the native link is unavailable.

The default installation runs the gateway locally, but this is not a required
one-gateway-per-server architecture. A sysop may point each worker relationship
at a shared remote gateway by configuring matching contiguous target ports,
peer IDs and keys, then removing or disabling the local Compose service. The
gateway endpoint must be reachable from the FreeDMR container; `localhost`
would address the FreeDMR container itself, not a separate gateway container.

### Migrating the legacy D-APRS container

Running the maintained installer on a directory which already contains both
`freedmr.cfg` and `docker-compose.yml` delegates directly to the upgrader. It
does not reinstall Docker, rewrite the Docker daemon configuration, ask
fresh-install questions or replace either file with a distributed template.
An incomplete deployment containing only one of those files is rejected for
manual review.

`upgrade-daprs-to-data-gateway.sh` migrates the documented legacy `D-APRS`
Compose service to the native Data Gateway without replacing unrelated
FreeDMR configuration or Compose services. This deployment transition is not
performed by `docker pull`: current images require the corresponding deployment
version marker in the mounted `freedmr.cfg`. Download the script, inspect it,
and run a dry run before pulling and starting the new images:

```sh
curl -fsSL \
  https://gitlab.hacknix.net/hacknix/FreeDMR/-/raw/master/docker-configs/upgrade-daprs-to-data-gateway.sh \
  -o /tmp/upgrade-daprs-to-data-gateway.sh
sudo /bin/bash /tmp/upgrade-daprs-to-data-gateway.sh --dry-run
sudo /bin/bash /tmp/upgrade-daprs-to-data-gateway.sh
```

The script reads the server ID and any existing Data Gateway ports and
non-placeholder passphrase from `freedmr.cfg`. It reads the HDStack settings
and APRS credentials from the legacy or native service in
`docker-compose.yml`. If `HDSTACK` is omitted, it applies the container's
CPU-count-minus-one default, clamped to one through five. A private FBP key is
generated only when no usable one exists. An existing native relationship
array is preserved as deployment configuration rather than rebuilt from a
canonical HDStack assumption. Missing, conflicting or invalid identity, key
or port values still cause the migration to stop rather than guess.

Before changing the deployment files, the script validates the proposed Compose
configuration and asks for confirmation. It uses a deployment lock and creates
timestamped backups beside `/etc/freedmr/freedmr.cfg`,
`/etc/freedmr/docker-compose.yml` and an existing relationship document. The
operation is idempotent. It recognises a complete native deployment and can
resume when either the configuration or Compose half of this migration was
installed first. A complete native service also takes precedence over a stale
legacy service left beside it. Conflicting keys, ports, malformed services and
unrecognised layouts still stop for manual review. Deployment format 2 records
the mounted relationship document and is written only as part of a validated
upgrade.

The script does not pull images or restart containers; follow the commands it
prints after reviewing the generated files. If a current FreeDMR image is
started against an unversioned deployment, the container logs the required
upgrade and exits before starting FreeDMR. A sysop maintaining a deliberately
custom deployment may temporarily bypass that guard with:

```yaml
environment:
  - FREEDMR_ALLOW_UNMIGRATED_START=1
```

The override logs a warning on every startup. It does not modify or advance the
deployment marker, and it does not assert that the custom configuration is
compatible. The guided installer validates its generated Compose file and
writes the same marker for new deployments. Merely copying an example
`freedmr.cfg` does not record a completed deployment upgrade.

## Session assignment

The proxy rotates new DMR IDs through the configured worker ranges. With two
workers the assignments are:

```text
new session 1 -> worker 1
new session 2 -> worker 2
new session 3 -> worker 1
new session 4 -> worker 2
```

Within the selected worker range, the proxy retains its existing random choice
of an available generated system port.
Subscriber location remains best-effort state. A worker records the originating
HBP repeater or hotspot ID, slot and observation time rather than a worker-local
generated system name. Authenticated FBP v5 traffic received through the
private aggregator relationship also updates this map when its source server
and hop path identify another worker in the same HDStack deployment.

Each worker resolves a unit-data destination against its currently connected
HBP peers before transmitting. This allows a hotspot or repeater to reconnect
on another worker without requiring the subscriber to transmit again first,
provided the subscriber-to-access-peer observation has already propagated.
Workers retain separate persistence files; they do not concurrently write one
shared pickle file.


A DMR ID remains pinned to that exact generated port for the lifetime of its
active proxy session. Moving it would start a new HBP login and reset its
BRIDGES and OPTIONS state. The proxy does not rebalance or fail over an active
session. Once the session expires, a later login by the same DMR ID is a new
assignment. Affinity is not persisted across proxy restarts.

Existing explicit proxy configurations using `DestportStart` and `DestportEnd`
continue to describe one range. The equivalent direct multi-range setting is:

```ini
[PROXY]
BackendPortRanges: [[54000, 54099], [54100, 54199]]
```

or:

```sh
FDPROXY_BACKEND_PORT_RANGES='[[54000,54099],[54100,54199]]'
```

The standard HDStack entrypoint supplies this setting automatically.

## Proxy diagnostics

In multi-worker mode the proxy logs each backend number and port range during
startup. With the standard two-worker layout this includes:

```text
(PROXY)(HDSTACK) Backend:1 ports:54000-54099.
(PROXY)(HDSTACK) Backend:2 ports:54100-54199.
```

When `FDPROXY_CLIENTINFO=1`, as in the shipped Compose configuration, client
assignment and removal messages include both the backend number and exact
generated port. This makes round-robin assignment and retained session
affinity visible without enabling packet-level debug logging. Single-backend
startup output remains unchanged.

## Reporting MUX

When reporting is enabled in the main configuration, worker report ports are
derived from its `REPORT_PORT`. With the default output port `4321`:

```text
MUX output -> 4321
worker 1   -> 4322
worker 2   -> 4323
worker 3   -> 4324
worker 4   -> 4325
worker 5   -> 4326
```

The MUX preserves the existing netstring reporting protocol and the main
configuration's `REPORT_CLIENTS` allow-list. It merges worker configuration
and bridge snapshots and relays worker bridge events. System names are
prefixed with the stable worker number to prevent collisions:

```text
1:SYSTEM-001
2:SYSTEM-001
```

Only the system-name field is changed. The reporting protocol contains trusted
Python pickle data and must remain on a trusted network.

A failed or malformed worker reporting source is isolated from healthy
sources. Reporting failure does not affect the proxy or DMR traffic. If main
reporting is disabled, worker reporting and the MUX are both disabled.

## Limitations

HDStack deliberately provides no least-connections balancing, active backend
failover, live migration, persistent affinity or automatic reconfiguration.
A failed worker does not cause its active sessions to move to another worker.
The separate bridge config and rules remain sysop-managed. The runtime
generator creates the private transport link but does not create routing
policy.