A save is now one upsert per message in the list being saved, and costs the
same with 50,000 messages of history as with 50. That is the Feature's
"done when", and it is the part #919 could not deliver.
Also gone: a delete is a single-row DELETE instead of rewriting the whole
history, and a load reads only its own key's rows, ordered and pageable in
SQLite.
Load order matters and is deliberate. Rows are consulted first, but ONLY
when rows actually exist. A key marked migrated with no rows still falls
through to the legacy slot-index and unscoped-key adoption (#194, #306),
because those run on blobs; short-circuiting on the marker alone would strand
history stored under an older key. Adoption then hands its result to the
migration, so the key ends up on rows either way.
The blob path stays as the fallback for a key whose migration was refused.
Refused keys keep working exactly as they did after #919: off the UI isolate,
merge-not-overwrite, failing closed on an undecodable blob. Tested, including
that a refused key does not accumulate half a history in rows.
One unreadable row now costs one message. On the blob a single bad byte took
the whole conversation with it.
Clear paths had to learn about rows: a cleared channel or conversation that
only dropped its blob would have stayed fully readable. clearAll also sweeps
row-only keys, which no longer appear in a blob key listing.
`encodeHistoryForTest` is new on both stores. The #919 tests built their
blob fixtures by saving through the store and reading the blob back, which
cannot work once saves write rows, and hand-written fixtures are what rotted
the first time. They now call the stores' own encoder directly.
Tests: 24 added, through the public store API only, because that is what an
existing install hits on its first launch after this ships. Covered: blob
adoption on first load and on first save, the windowed-save truncation case
(#343), per-radio and per-contact scoping (#472), delete, clear, clearAll on
row-only keys, a refused migration still serving reads and writes, and the
payload being the stores' own encoding.
Suite: 1,344 pass, analyze clean, format clean.
Citadel: meshcore-open-tfq
Refs #920, #917, #334
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Per key, once: count the blob's records, convert it to rows off the UI
isolate, insert them in one transaction, read the stored count back, and
only then mark the key as reading from rows.
**The blob is never deleted**, not even on a verified match. Rows become the
read and write path; the blob is left untouched. Reclaiming that space is a
separate owner-gated step, because the moment the delete runs is the moment
a mistake stops being recoverable, and a downgrade to an older build still
finds its history where it looks for it.
Every failure leaves the blob authoritative and says so loudly: an
undecodable blob, a converter that throws, a count mismatch, a failed
insert. None of them retries, because a count that did not match once will
not match on a loop.
The migrated set is its own marker key, not "this key has rows". Those are
different statements: a key can have rows and an incomplete migration at the
same time, and treating rows as proof of completeness is how half a history
would read as a whole one. An unreadable marker treats every key as
unmigrated, which is the safe direction because the blobs are all still
there.
The first version of the readback gate was a check that could not fail. It
compared the stored row count against the number of rows the converter
produced, so a converter that silently dropped records matched itself and
passed. Caught by a test built to drop a record. The blob's element count is
now measured by a separate decode that does nothing but count, so the
converter cannot report its own homework: `rows.length` must equal that
count, and the stored count must equal the distinct identities. Both can
fail independently, and together they close the loop from blob element to
stored row.
Duplicate identities in a source blob are deduped rather than refused. The
old merge should not have produced any, but the legacy slot-index adoption
path (#194) could have, and that must not read as data loss.
Tests: 15, nine of them failure paths. The lossy-converter case is the one
that caught the broken gate.
Citadel: meshcore-open-x9q
Refs #920, #917, #334
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Schema 2 adds one row per message beside StoredBlobs, which stays exactly as
it is. A save becomes an upsert of one row, so its cost stops depending on
how much history is already stored. That is what #919 could not do: it moved
the whole-history re-encode to another isolate, it did not remove it.
Shape, and why it is this shape:
store_key the EXISTING blob key, byte-identical
merge_key the stores' existing _mergeKey identity
timestamp_ms INTEGER, so ORDER BY and LIMIT run in SQLite
payload the per-message JSON the stores already emit
PK (store_key, merge_key), index (store_key, timestamp_ms)
Reusing the blob key means every per-radio and per-channel scoping rule
(#194, #472) carries over untouched, and the migration becomes a per-key
operation with an exact record count to verify against.
The payload is JSON rather than a column per field. A Message has 25 fields
including three maps, two byte arrays and the translation set; hand-mapping
them would mean trusting a new field-by-field converter against 41 MB of
real history on its first run, and #355 cost 574 messages the last time a
storage migration was hurried. With a payload column the migration is a pure
re-key, the stores' existing encoders are reused verbatim, and a new message
field needs no second migration. The trade is that message fields are not
SQL-queryable; nothing asks for that today, and columns can be added beside
the payload later without another data migration. Flagged for the owner on
the issue before building it.
The row API lives on BlobStore rather than in a second class because both
APIs must share one OffbandDatabase: opening it twice is an error on the web
backends, and a second singleton would be one mistake away from doing that.
`limit` with no `offset` returns the NEWEST n, still oldest-first, because
that is what a chat screen opens with.
Tests: 18. The row set covers store-key scoping (a leak there would be a
history bleed between radios), merge-key upsert, paging, per-row delete, and
rows coexisting with a retained blob. The schema set builds a database that
looks like one an older build left behind, v1 table with real data and
user_version 1, and proves the upgrade runs, creates a usable table, leaves
the blobs byte-identical and lands the index on both an upgraded and a fresh
database.
Citadel: meshcore-open-g6j
Refs #920, #917, #334
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deleting one message rewrites the entire blob, so on the owner's largest
channel a delete blocked the UI for as long as a save did, right after the
user tapped something. Both delete paths now go through
`mergeOffThread`.
Two call shapes, one helper. `mergeOffThread`'s second argument is renamed
`incomingJson` to `payload`, because the callback decides what that string
means: the incoming batch when saving, the target merge key when deleting.
The method only carries it across the boundary.
`dropFrom{Channel,Dm}HistoryJson` return null when the target is not in the
history, so nothing is written at all. Re-encoding 5 MB to change nothing is
exactly the cost this Epic exists to stop paying, and the old code did it on
every miss.
`MessageStore.removeMessage` also gains per-key serialisation it never had:
it was a read-modify-write outside `BlobStore.synchronized`, so a save
landing after a delete could resurrect the deleted message. There is a test
for that race.
Tests: 8 added in their own file, including both stores' public delete API,
an absent target, an absent blob and a corrupt blob left byte-identical.
Citadel: meshcore-open-z7c
Refs #919, #917, #334
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`MessageStore.saveMessages` had the same read-modify-write as the channel
store: decode the whole DM history, rebuild every record, re-encode it, all
on the UI isolate, for every single message that arrives.
Applies A1's helper. Only the incoming batch is encoded on the UI isolate;
the decode, merge and re-encode go to another isolate via
`mergeDmHistoryJson`, a top-level function so `compute` can send it. The
five JSON helpers it needs are now static for the same reason.
`mergeDmHistoryJson` throws on undecodable stored JSON rather than
swallowing it, which `mergeOffThread` turns into a refused write. That is
the same refusal the inline code made, and it is what stops a 5 MB history
being replaced by the one message that happened to arrive.
Tests: 6 added. Fixtures encode through the store's own encoder, never a
hand-written map, because a hand-rolled fixture rots the moment a non-null
field joins the schema. Includes a save round trip proving the second save
merges rather than overwrites (#343).
Citadel: meshcore-open-8sg
Refs #919, #917, #334
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BlobStore.mergeOffThread runs the whole read-modify-write through compute(): stored JSON and incoming JSON in, merged JSON out, so only strings cross the isolate boundary and the object graph never does. Keeps the per-key serialisation that stops concurrent saves clobbering each other.
channel_message_store.saveChannelMessages now encodes only the INCOMING messages on the UI isolate (normally one message, measured 0.002 ms) and hands the 5.38 MB decode-merge-encode to another isolate, where it measured 134 ms per save and crossed Android's 5s input-dispatch watchdog in a burst (#334).
Fails closed: a stored blob that will not decode throws out of the merge, is logged, and the key is NOT written, which is the same refusal the inline code made. compute runs inline on web, which has no isolates; that is documented behaviour, not a silent fallback, and #920 removes the cost for every platform.
_messageToJson, _messageFromJson, _mergeKey, _repeatToJson and _repeatFromJson became static so the top-level merge can call them; none touched instance state.
9 new tests cover merge equivalence, timestamp ordering, empty/absent blobs, the throw on corrupt JSON, fail-closed leaving the blob byte-identical, and three concurrent saves all surviving. Fixtures go through the store's own encoder rather than hand-written maps, after a hand-rolled fixture rotted on a non-null field.
Refs #334, #917
Agent: TealGlen (session c61ce804)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ChannelRegionScopeStore persists a channel-index -> region-name map per device (first 10 hex of the device key, matching the other channel stores). set/clear/scopeFor, null = unscoped. Tests cover round-trip, per-device and per-channel isolation, persistence, and no-device-key guard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ContactStore.loadContacts returned an empty list when the stored JSON
failed to decode, and the next save wrote the in-memory list over the
stored contacts. A read failure was treated as "no contacts".
A decode failure is now logged as an error, the raw value is kept
under contacts<key>.unreadable-<timestamp>, and saves to that key are
refused (and logged) until a load succeeds. Other radios' keys are
unaffected.
Tests cover the round trip, the failure report, the raw copy, the
refused save, recovery after a good load, and another radio's key;
with the refusal disabled the save test fails.
Epic #668.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Children of #471 (parent stays open for AAB hardware validation, #507).
The block list was one global unscoped list on the phone, unioned onto
every radio on connect. A stale block for one of the owner's own radios
therefore rode onto every fresh/erased radio, and clearing a radio never
stuck: any other radio still holding it re-seeded the global list on
connect, which re-pushed it back.
- #505 block_store.dart: scope keys/names by connected device key (like
the app's other stores); dropLegacyGlobal() deletes the legacy
block_keys_v1/block_names_v1 (drop-and-start-fresh, owner-approved).
No global list.
- #505 block_service.dart: load() drops the legacy global and starts
empty; loadForDevice(deviceKey) swaps the in-memory set per radio and
runs the #250 self-heal. All mutating ops serialized through a Future
chain so loadForDevice and importKeys (different connector frame
handlers, both unawaited) cannot interleave and wipe each other.
- #506 meshcore_connector.dart: the device-key hook loads the connected
radio's list, so the offload union reconciles within that one radio.
Existing radio-side firmware blocks left untouched (no auto-CLEAR on
migration, owner-approved). Tests: per-radio isolation, clear-sticks
across reconnect, legacy-drop, disconnect-clears, load/import race.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ben's style rule: no em-dash character in copy, docs, or code comments
(it reads as an AI tell), and the repo is going public. Replaced the
em-dashes in code comments and English UI strings with commas, colons,
or periods, whichever reads best. User-facing strings were hand-tuned
for natural punctuation rather than a blanket comma.
Preserved the lone "no data" glyph placeholders (a standalone dash used
as a not-available indicator in status displays); those are a design
element, not prose.
Regenerated app_localizations*.dart from app_en.arb (the English
fallback for untranslated keys propagates to every locale's generated
file). Non-English ARB translations left untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Smaz is a client-side text convention with no MeshCore protocol or firmware
support (findings: GH-315). When enabled it compressed ordinary prose into
`s:`+base64, which any non-lineage client renders as garbage — the same
cross-client interop failure as the old `g:<code>` GIF token.
Phase 1 of the Smaz-removal epic (GH-314): stop sending Smaz and remove the
user-facing surface, while KEEPING decode so messages from lineage peers still
on Smaz, and any legacy compressed rows, keep rendering. Decode removal is
deferred to Phase 2 (GH-421).
Removed: the Smaz branch in prepareContact/ChannelOutboundText (Cyr2Lat branch
and the structured-payload guard preserved); connector state/API (the enabled
maps, is*/set*SmazEnabled, ensureContactSmazSettingLoaded, the warm-up and
channel loaders); the per-channel and per-contact toggles plus their
mutual-exclusion; loadSmazEnabled/saveSmazEnabled and the `*_smaz_` key prefixes
(stores kept, they also hold Cyr2Lat); l10n `channels_smazCompression` and the
orphaned `chat_compressOutgoingMessages` across 18 locales (+ regenerated
app_localizations).
Retained for Phase 2: the 5 Smaz.tryDecodePrefixed decode sites and
helpers/smaz.dart.
Safety (verified): no storage migration, the app already persists plaintext
(receive decodes before store; send stores the pre-compression text), so the
`s:` form was wire-only. ACK matching is unaffected, the expected hash derives
from prepare*'s output, so dropping compression keeps both sides hashing
plaintext.
Behavior change: the composer byte-counter now reflects raw size, so anyone who
had Smaz on can type slightly fewer chars per message.
Tests: gif_url_outbound_guard_test rewritten to pin plaintext passthrough and
decode retention. Full suite green, analyze clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tapping a DM opens the Path screen; it now shows the RF details the app
was already receiving but discarding:
- SNR: captured from the v3 contact-msg-recv frame (was skipBytes(1)).
Firmware sends (int8)(snr_dB*4), so dB = byte/4.0 (MyMesh.cpp:512) —
the old commented-out code multiplied by 4, which was 16x wrong.
- Path type: firmware sends path_len 0xFF for a direct/routed frame and
the hop count for a flood (MyMesh.cpp:545). The app collapsed 0xFF->0,
colliding "direct" with "flood, 0 hops". Capture the distinction into
Message.isFloodRoute so the Path row reads "direct (routed)" vs
"flood, N hops".
- RSSI: row wired to Message.rssi but left null — the wire byte is a
hardcoded reserved 0 today; populated once firmware ships it (#439).
Message gains snr/rssi/isFloodRoute (nullable, additive, persisted like
rxTime). 10 tests cover scaling, the 0xFF discriminator, and copyWith.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wadamesh keys its history watermark (last_delivered_seq) by client_id. We
sent none, so we shared the empty-string slot with every other MeshCore
client on the machine: whichever connected first drained the device history
ring and the next app got NO_MORE_MESSAGES for frames it never received.
cid_len is 6 by necessity, not preference. Stock reads cmd_frame[1..7] as
reserved with the app name at a fixed offset 8; Wadamesh reads the name at
2 + cid_len. Only 6 puts the name at 8 on both, so one frame serves both
firmwares with no firmware change. Covered by app_start_frame_test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reactions stored only a Map<String,int> of emoji to count, so "who
reacted" was unanswerable and MeshCore One's tap-to-see-who had nothing
to show against. This captures the reactor for every reaction, both our
own r: format and the PocketMesh / MeshCore One format.
Data layer only. The tap-a-badge-to-see-who UI is a separate follow-on;
this makes the data available and does not change any screen.
Additive by design. A new reactionSenders field (Map<String,List<String>>,
emoji -> reactor names) sits alongside the existing count map, serialised
under a new JSON key. Both maps are always written. An older build reading
a newer store ignores the unknown key and still gets correct counts from
reactions; it never hits the hard `value as int` cast that would fail the
whole message-list load (the #355 data-loss shape). Records written before
this field load with an empty sender map, so their counts survive with
names simply absent.
- ChannelMessage + Message: new reactionSenders field, constructor,
copyWith
- both stores: serialise the new key; deserialise via a shared
ReactionHelper.reactionSendersFromJson that returns empty for a missing
key and skips malformed entries rather than throwing
- applyReaction: records the reactor and dedups per reactor per emoji.
This is persistent dedup (survives restart), unlike the connector's
in-memory processed-set. A pre-existing count with no sender list is
incremented from its stored value, not recomputed from the partial
list, so old counts are preserved.
- connector: threads the reactor name into all reaction paths. Channel
uses the frame sender; a room resolves the author via
fourByteRoomContactKey; an outgoing reaction is attributed to self.
- pending queue: retry now hands the stored reactor to its callback
Tests: reactor capture, two-reactor count, same-reactor no-double-count,
pre-#383 count preservation, and JSON round-trip + backward-compat +
malformed-entry handling for the new field. Full suite 604 passing.
Epic #376.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the database can't open (e.g. the native sqlite library fails to load),
every read/write silently failed and the app opened to an empty, normal-looking
screen — the user thinks their history was wiped (SAFELANE §6 violation).
Probe the storage layer in main() before any store reads
(BlobStore.verifyReadWrite). On failure, a StorageHealthService one-way latch
records it, and a persistent, non-dismissable banner is shown above the whole
app: history is not lost, storage is unavailable, messages are NOT being saved,
restart after fixing. Cross-platform (probe goes through drift, covers web too).
Tests: service state/latch + banner show/hide widget test. Gemini-reviewed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#363 pins the DB to one directory and migrates ONE prior store in, but a user
who ran differently-built copies can have data split across several stores.
On startup (native only, after the prefs->drift migration) this discovers
every store on the machine and unions its bulk blobs (messages by id, contacts
by public key) into the current one, so nothing shows as a gap. Sources are
read-only and never deleted.
Not a one-shot: instead of a permanent flag it tracks each store's signature
(mtime+size), so a store that a stray older build later creates or grows is
re-merged rather than stranded. A store over a 200 MB guard, or one that
cannot be read, is skipped and NOT recorded as done, so it retries later.
Identity dedup is key-order independent. sqlite3 (dart:ffi) stays out of the
web build via a conditional import.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The VACUUM INTO snapshot import pulled dart:ffi via package:sqlite3, which
does not compile on web. Isolate it behind a conditional import (native impl
+ web stub); the migration is native-only and never runs on web.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DB directory came from getApplicationSupportDirectory(), which on Windows
is derived from the executable's Company/Product version metadata. Two builds
with different metadata (or a future rebrand) resolved DIFFERENT %APPDATA%
folders and read DIFFERENT databases, so a user's history appeared to vanish
when they ran a different build.
Pin the desktop DB to a constant path (Windows: %APPDATA%\Offband MeshCore;
Linux pinned too, its support dir derives from the exe name). On first run at
the pinned location, migrate an existing DB in via SQLite VACUUM INTO (a
consistent snapshot that is safe even under a concurrent writer), choosing the
DB from the known canonical locations first and falling back to a bounded
scan of the app-data roots (no name blocklist) so a DB under an unknown
folder is still found. On any snapshot failure it aborts cleanly, leaving the
source intact and opening a fresh DB - never a raw file copy of a live WAL DB.
Adds sqlite3 as a direct dependency (pinned to drift 2.34.2's resolved 3.5.0).
Tests cover source selection, the folder-agnostic scan, and the snapshot
happy + abort paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The migration's already-present branch called prefs.remove(key) and threw
the prefs copy away when drift already held the key. A build that writes to
SharedPreferences (a non-drift build, or any in-between test build) collects
new messages there, so the next drift run silently gapped that history out.
It cost 574 real channel messages, recovered from backups.
Union the prefs copy into drift by element identity (messageId, else
publicKey, else canonical JSON), keeping drift's live copy on a collision and
appending prefs-only elements. Verify the merged write before removing the
source. New `merged` counter in the report.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the stopgap (53f87f5) that committed
lib/storage/drift/offband_database.g.dart to green CI. Restores the
repo's "generated files are not committed" convention:
- dropped the .gitignore negation; the .g.dart is gitignored + untracked
(git rm --cached, working copy kept)
- added `dart run build_runner build --delete-conflicting-outputs`
immediately after `flutter pub get` in every Dart job: flutter_dart
(analyze), build.yml (all 6), release-signed (android-signed),
deploy-web (before build_pipe)
- keeps the drift_dev 2.34.0 / drift 2.34.2 / build_runner ^2.15.1 pins
from the stopgap for deterministic generation
Verified locally on the bench toolchain (3.44.1): deleted the working
.g.dart, ran build_runner from scratch (regenerated cleanly), confirmed
the generated file passes `dart format --set-exit-if-changed` and full
`flutter analyze --fatal-infos --fatal-warnings` is clean with the file
generated (not tracked).
For #335 / PR #348. Agent: SapphireCompass (session 8d755b5e)
CI failed on every job: offband_database.g.dart (drift's build_runner output)
was gitignored and no workflow runs build_runner, so analyze and all six
platform builds hit "Target of URI hasn't been generated". drift is the first
code in this repo needing codegen, which is why dev's CI never had to.
The workflows are not matrixed - six separate build jobs plus analyze, web
deploy and release-signed, each with its own `flutter pub get`. Adding a
codegen step to all of them is wide and easy to miss one. Instead the generated
file is committed (drift supports this): one file, works across every job and
workflow with no CI edits, and reproducible.
drift_dev is pinned (2.34.0) alongside the already-pinned drift, so a future
regeneration produces the same file rather than drifting from the committed
copy. Regenerated against the pinned versions before committing.
Follow-up worth having: a CI check that regenerates and diffs, so a schema
change without regen fails loudly rather than shipping a stale .g.dart.
Adversarial review (standards#145) of the storage/migration surface found four
real defects, all data-integrity. All confirmed against the code and fixed; no
false positives.
BLOCKER - concurrent save race. saveChannelMessages / saveMessages are
read-modify-write with an await gap, so two saves to the same key raced and the
second clobbered the first, silently losing messages (e.g. a message and its
delivery ack arriving together). BlobStore now provides synchronized(key, ...),
a per-key operation chain; every RMW - merge-save, remove, and the load-path
legacy migration - runs through it. Different keys stay concurrent. New test
fires two concurrent saves and asserts the union survives.
BLOCKER - non-atomic DB relocation. The documents->support move copied and
deleted each file (.sqlite/-wal/-shm) in turn, so a failure after the main file
moved stranded the DB across two locations and corrupted it. Now copies all
files, verifies each by size, and only then deletes the sources; on any failure
it rolls back the destination and leaves the original intact.
MAJOR - load-path legacy migration raced save. The #194 index->PSK adoption did
a blind write to the PSK key that could clobber a save that landed first. It now
runs under the key lock and MERGES (union) instead of overwriting, so both the
adopted history and any fresh message survive.
MINOR - channel merge key lacked a sender. Two senders posting identical text at
the same timestamp without a messageId would collide and lose one. The key now
includes the sender, matching MessageStore.
flutter analyze clean, dart format clean, 509 tests pass. Full Gemini log in
docs/llm-consultations/.
The persisted store was being overwritten with the windowed in-memory list, so
any channel or DM with more than _messageWindowSize (200) messages lost
everything older than the newest 200 on the first save after load. Observed
live: Public went 232 -> 201 in one session on a build that already had the
#333 fix, so this was not the race - it was windowing truncating the store.
This is the slow-erosion cause behind the whole 566 -> 231 -> 206 -> 201
history.
saveChannelMessages / saveMessages now MERGE into the persisted set instead of
overwriting: upsert by message identity so older persisted messages are kept,
new ones added, and the in-memory copy wins for edits/reactions/status. If the
existing history fails to decode, the save aborts loudly rather than merging
into an empty base and truncating (SAFELANE 6).
Deletion is now an explicit path - removeChannelMessage / removeMessage - since
the app deletes individual messages. Routing delete through the merging save
would resurrect them; the connector's deleteChannelMessage / deleteMessage now
call the explicit remove.
Windowing stays for display and memory; it no longer dictates what is stored.
Tests: 250-message history survives a 200-window save; new message appends;
edit is captured not duplicated; delete does not resurrect. 508 pass, analyze
and format clean.
Cost: a save now re-reads and re-encodes the channel/contact history. Cheap
with drift's per-key writes (#335); a future append-only schema removes even
that.
drift_flutter's native default is getApplicationDocumentsDirectory(), which on
Windows resolves to the user's Documents folder - redirected into OneDrive on
most machines (verified: drift_flutter 0.3.1 connect.dart). A live SQLite file
syncing to OneDrive risks lock contention and corruption, and it is simply the
wrong place for app data.
The DB now opens in getApplicationSupportDirectory() (%APPDATA% on Windows),
the same place SharedPreferences already lives, so all app data sits together.
Existing installs already have a DB in the old location, so opening a fresh one
there would abandon their migrated history. _appSupportDatabaseDirectory()
relocates it once on first run: it moves offband_store.sqlite and its -wal/-shm
sidecars from the documents dir to the support dir before drift opens, and
never deletes a source without a successful copy. If relocation fails it is
logged loudly and a fresh DB is created, with the migration re-running from
SharedPreferences rather than silently losing anything.
Web is unaffected: path_provider has no web backend and drift ignores
databaseDirectory there, using OPFS/IndexedDB.
Adds `path` as a direct dependency (was transitive).
flutter analyze clean, tests pass.
Wires the migration into startup and points the four bulk stores at drift.
SharedPreferences keeps the settings, which is what it is for.
Startup runs the migration after prefs are up and BEFORE any store reads, and
awaits it. Letting stores race a half-finished migration is precisely the
shape of #333, where a storage path silently chose the wrong key and 566 real
messages read as empty.
Converted: message_store, channel_message_store, contact_store,
contact_discovery_store. All four now have ZERO prefs get/set for bulk data.
Three safety properties, deliberately built in:
1. readWithPrefsFallback - if a key is somehow not in drift, the prefs copy is
still served rather than reading as empty. It logs a WARNING when it fires,
because a fallback during normal operation means the migration is
incomplete and someone needs to know. Silence here is what made #333 look
like data loss.
2. deleteEverywhere / keysWithPrefix span BOTH backends. A clear that only
removed the drift row would leave a pre-migration prefs copy to reappear
through the fallback, resurrecting deleted history.
3. The legacy-key migrations inside the stores now check both backends and
only touch prefs when a legacy key actually exists. This also carries the
#306 fix into this branch: the unconditional prefs.remove was still present
here, since this branched from dev rather than from the #306 work.
Verified: analyze clean, format clean, 504 tests pass, Windows and web both
build. Migration rehearsed earlier against a copy of a real 7 MB store: 69
keys, 5.21 MB, zero failures.
NOT yet run against live data - that happens on first launch of this build,
and the owner should have a prefs backup before that.
Moves message history, contacts and discovered contacts out of the settings
store. Settings stay in SharedPreferences, which is what it is for.
Ordering is the whole safety argument: WRITE, VERIFY BY READING BACK, and only
then remove the source. #333 was a storage path that chose a key silently and
made 566 real messages read as empty; deleting before verifying would make
that class of mistake permanent instead of cosmetic. On any failure the source
is left intact and the error is logged - never a silent drop (SAFELANE 6).
Idempotent by construction: a key already present in drift is not overwritten,
so re-running is a no-op. If an older build re-writes a migrated key into
prefs, the migrated copy wins and the stale prefs copy is discarded rather
than promoted.
Rehearsed against a COPY of a real 7 MB store, as the plan required before
touching live data:
REHEARSAL: 69 migrated, 0 failed, 5.21 MB, 69 bulk keys expected
Every key checked for exact length, confirmed removed from prefs, and every
settings key confirmed untouched. The live store was never opened.
Two things the tests caught that review would not have:
1. getString THROWS on a non-string value rather than returning null, so a
non-string under a bulk prefix was counted as a migration FAILURE. It now
type-checks with prefs.get() and skips. Alarming falsely is its own bug.
2. The `contacts` prefix was checked against the real store rather than
assumed: it matches only the 6 bulk contact blobs, and correctly does NOT
match contact_unread_count*.
Not yet wired into app startup - that is the switchover, and it is deliberately
a separate commit so this can be reviewed on its own.
flutter analyze clean, dart format clean, 504 tests pass.
Step 1 gate, web half. Building is not evidence, so this was run in a browser.
Assets: sqlite3.wasm (748 KB) and drift_worker.js (355 KB), both taken from
the SAME drift-2.34.2 release so they are built against each other. Validated
before use - the wasm has correct WebAssembly magic (0061 736d) and carries
SQLite 3.53.3; the worker is genuine compiled Dart JS.
Note the filename: the release ships `drift_worker.js`, not the
`drift_worker.dart.js` the docs name. The database URI was corrected to match,
which would otherwise have been a silent 404 at runtime.
Browser result, served over HTTP with COOP/COEP and Content-Type:
application/wasm:
Using WasmStorageImplementation.opfsLocks
PROBE-OK open+write+read
PROBE-OK 6MiB round-trip (localStorage cap is 5MiB)
drift selected OPFS, the top storage tier, and round-tripped 6 MiB - data the
current localStorage-backed web build physically cannot hold. No exceptions.
Adds web/_headers for Cloudflare Pages: forces application/wasm on the wasm
asset and sets COOP/COEP for cross-origin isolation. Verified to ship into
build/web. Without COOP/COEP drift still works, falling back to IndexedDB,
which is slower but still unbounded next to localStorage - a performance tier,
not a correctness requirement.
web_probe/ is a standalone entrypoint that exercises ONLY the drift stack, so
a pass or fail is unambiguous without BLE and the rest of the app in the way.
It is not part of the app build.
Still no user data touched and no store switched over. The migration is the
next step.
flutter analyze clean, dart format clean, 497 tests pass.
Step 1 of #335, deliberately before any data migration: prove the store works
before trusting it with message history.
Adds drift + drift_flutter, a key/value StoredBlobs table mapping 1:1 onto the
existing SharedPreferences keys, and verifyReadWrite() as a runtime probe.
Key/value rather than relational tables on purpose: it maps directly onto the
current store interfaces, so callers do not change and the migration can be
verified row-for-row against the old keys. Relational schemas come later, once
this is proven and querying is actually wanted.
Verified here:
- 3 tests pass, including a 6 MiB round-trip - larger than the 5 MiB
localStorage cap that makes the web build impossible today
- flutter analyze clean
- Windows release build succeeds
- Web release build compiles
Two findings from the gate that a plan-on-paper would have missed:
1. sqlite3_flutter_libs 0.6.0+eol is a DEPRECATED no-op. From sqlite3 v3.x it
is unnecessary and drift_flutter already covers it, so it is not a direct
dependency here.
2. The web build COMPILES BUT WOULD FAIL AT RUNTIME: drift needs sqlite3.wasm
and drift_worker.dart.js shipped in web/, and neither is present. Per drift
docs they are downloaded from GitHub releases, version-matched to
pubspec.lock (drift 2.34.2, sqlite3 3.5.0), and the server must serve .wasm
as Content-Type: application/wasm. Not done here - downloading files needs
explicit human approval.
Also noted: the drift_dev CLI does not compile at these versions
(allSchemaEntities missing from the drift3_preview GeneratedDatabase), so its
asset tooling is unavailable. build_runner code generation is unaffected.
No user data is touched. No store is switched over. The migration is the next
step and remains gated on the web assets question.
The stores violated the no-silent-failures rule, and that is why tonight's
diagnosis took as long as it did. Two classes:
1. Ten catch blocks swallowed decode errors and returned an empty collection.
`catch (_) { return []; }` means a corrupt or unreadable store presents to
the caller as "no data", so the user sees no contacts, no channels or no
history with nothing anywhere explaining why. Indistinguishable from data
loss. Every one now logs what failed, with the store named, before
returning.
2. ChannelMessageStore._storageKey fell back from the PSK-identity key to the
legacy slot-index key without a word. That fallback is correct before a
channel list exists, but once history has been migrated to the PSK key it
reads a DIFFERENT key and returns empty. That is exactly #333: a user's 566
Public messages were intact on disk while the app showed nothing, with no
error, no parse failure and no log line to follow.
It now warns when it falls back while a resolver is installed, since that
combination specifically means the channel list was not loaded first.
Had either of these been loud, #333 would have been a one-line log read
instead of a multi-hour investigation that looked like data loss.
flutter analyze clean, dart format clean, 494 tests pass.
The previous commit fixed loadSmazEnabled and killed the post-connect stall
(startup-load channelSettings: 37792ms -> 0ms, measured twice). The contact
pull still choked, in multi-second bursts, each landing immediately after an
"Added new contact" line.
No frame handler exceeded the 100ms threshold, because the work is not in a
handler: the contact handler fires _loadMessagesForContact unawaited, so it
runs after the handler returns and escapes that timing.
message_store.loadMessages had the same unconditional prefs.remove(oldKey),
and it runs ONCE PER CONTACT during a pull. On Windows every prefs mutation
rewrites the entire file, so a 234-contact pull meant 234 full multi-MB writes
for legacy keys that do not exist.
A sweep found the identical pattern in six more stores: contact_store,
unread_store, channel_store, community_store, contact_group_store and
channel_order_store. Those run once per load rather than per item, so they are
far cheaper, but it is the same bug and they are fixed the same way.
In every case the removal now happens only when a legacy key actually exists,
paired with the write that migrates it, so the cleanup still occurs on a real
migration.
flutter analyze clean, dart format clean, 494 tests pass.
The connect freeze is a legacy-key migration that writes even when there is
nothing to migrate.
loadSmazEnabled called prefs.remove(oldKey) unconditionally whenever the
scoped key was absent, which is the normal case for a channel that has never
had the setting changed. On Windows shared_preferences re-serialises and
rewrites the ENTIRE prefs file on any mutation, so each of those no-op removes
cost a full multi-MB write. Repeated across every channel slot on every
connect, that is tens of full-file rewrites back to back.
Measured with the instrumentation from the previous commit, on the reporting
device (6.9 MB prefs, ~42 channel slots):
startup-load channelOrder 0ms
startup-load contactCache 12ms
startup-load channelSettings 37792ms <-- here
startup-load cachedChannels 0ms
startup-load channelMessages 21ms
startup-load unreadState 0ms
startup-load discoveredContacts 11ms
No frame handler exceeded 100ms, confirming the stall was local storage work
and not radio traffic.
The remove now happens only when a legacy key actually exists, alongside the
write that migrates it. contact_settings_store had the identical pattern per
contact and is fixed the same way.
This also explains the platform gap in the original report: Android's backend
batches into native storage, so the same code costs almost nothing there,
while every no-op remove on Windows is a full file rewrite.
flutter analyze clean, dart format clean, 469 tests pass.
The firmware path-len byte is packed and self-describing (src/Packet.h:79-84):
high 2 bits = hash size - 1, low 6 bits = hash COUNT (hops), byte length =
count * size. The app treated the low 6 bits as a byte count and ignored the
width, on the strength of a comment asserting the high bits were "not reliably
populated". Both halves of that comment were wrong.
Receive: Contact.fromFrame read `count` bytes where firmware meant `count`
hops, so at 2-byte width it kept HALF of every path and discarded the rest.
DAYTON VA's 2-hop route became a 2-byte fragment, which is why every repeater
login in Bandit's capture timed out.
Send: buildUpdateContactPathFrame wrote the count raw, leaving mode bits 00.
That tells the radio "1-byte hashes" while handing it 2-byte hash data, so it
routed to nodes never on the route. This is the likelier cause of the observed
"routes via c6 and 5c separately" behaviour.
Also fixes a third unit inconsistency: setContactPath wrote customPath.length
(bytes) into pathLength (hops) after every path set.
Contact now carries pathHashWidth per path, so a stored path can never be
re-sliced at a width it was not captured at. realHopCount() is removed rather
than adjusted: it divided an already-correct hop count, halving it at 2-byte
width, and every call site was compensating for a bug that no longer exists.
Migration (Ben's call, option A): records written before this change have no
pathHashWidth and hold truncated bytes that cannot be recovered by
reinterpretation, so their path is dropped and the contact reverts to flood
until the radio re-supplies it. Logged, not silent.
Tests asserted the old semantics (expect(realHopCount(2, 2), 1)) and were
rewritten against firmware truth rather than adjusted to keep passing. Added
send-side encoding coverage, which did not exist. 456 tests pass.
Refs #240, #279, #299
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every received DM/channel message now logs arrival wall-clock vs the
sender-claimed payload timestamp (delta + TIME ANOMALY marker beyond
24h), routed directly to AppDebugLogService so the rotating on-disk
trail is unconditional. Models persist a nullable rxTime set at every
live ingest construction site; legacy stored records stay null.
Diagnostic child of #280.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
App-global (NOT per-device) block list, source of truth for the client:
- BlockStore persists blockedKeys (pubkey hex) + blockedNames (name->first-seen
epoch, for A7 promote-and-prune) via PrefsManager, global keys.
- BlockService (ChangeNotifier): isBlocked/isNameBlocked/block/unblock/
blockName/unblockName + unmodifiable accessors.
- Wired into main.dart MultiProvider.
Epic A #165 / A1 #168. Design: docs/architecture/block-contract-as-built.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gemini review flagged setPublicKeyHex using `value.length > 10`, which would
store an empty scope for an exactly-10-char device key (unreachable today —
MeshCore public keys are 64 hex — but inconsistent with ChannelStore, which
uses >=10). Align to >=10 for consistency and to remove the latent off-by-one.
Channel message history was persisted keyed by the reusable slot index, so a
channel added to a slot that once held another channel inherited its messages.
Storage now keys by the channel's PSK identity via a resolver on the store
(index -> channel PSK), leaving all call sites unchanged. A one-time migration
adopts pre-existing slot-index history into the PSK key on first load (only when
the PSK is known); clearChannelMessages now wipes both the PSK key and the raw
slot-index key so Build-B's reuse-clear can't be re-migrated onto a new occupant.
Build-A (durable) of epic #190; supersedes Build-B's (#193) guard. Also removes
the dead device-wide fallback read in loadChannelMessages.
this adds the actual last modified timestamp when present, before we used
last advert time as last modified in error
also sets _pendingInitialContactsSync to true on first connect over BLE
- Introduced translation functionality in chat screen, allowing users to translate messages before sending.
- Added MessageTranslationButton to the input bar for enabling/disabling translation.
- Implemented translation service to handle incoming and outgoing text translations using llama models.
- Enhanced message storage to include original and translated text, language codes, and translation status.
- Created UI components for displaying translated messages and managing translation options.
- Added translation model management, including downloading and storing models locally.
- Updated app settings to manage translation preferences and model selections.
- Added storageUsedKb and storageTotalKb properties to MeshCoreConnector.
- Updated battery and storage frame parsing with improved error handling.
- Refactored log RX data handling to use BufferReader for better readability and error management.
- Enhanced message parsing in ChannelMessage and Message classes to utilize BufferReader.
- Introduced new text type for signed messages in meshcore_protocol.dart.
- Updated BLE debug log screen to use BufferReader for payload parsing.
- Refactored message retry service to handle ACK hashes as integers instead of Uint8List.
- Improved message storage serialization and deserialization to accommodate new expectedAckHash type.
- Added wasPulled property to Contact model for better state management.
* Refactor contact handling: replace DiscoveryContact with Contact, update related methods and settings
* Enhance contact handling: include latitude, longitude, and last modified timestamp in contact updates; refactor path handling to accommodate discovered contacts across multiple screens
* Enhance SNRIndicator: include discovered contacts in name resolution for repeaters
* Refactor path handling: replace addReturnPath with buildPath to improve path construction logic and handle target contact types
* Update lib/screens/map_screen.dart
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add localization for "Show Discovery Contacts" in multiple languages and refactor location plausibility check in map screen
* Enhance contact management: update discovered contacts' active status and improve contact handling with flags and raw packet data
* Refactor ChannelsScreen: pass ChannelMessageStore to buildExpandedContent and ensure messages are cleared after channel creation
* Update MapScreen: adjust label zoom threshold and refactor guessed marker building to include labels
* Refactor ChannelsScreen: change channelMessageStore to a private getter and update its usage in buildExpandedContent calls
* Enhance location plausibility check: add latitude and longitude bounds to ensure valid coordinates
* Update lib/connector/meshcore_connector.dart
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor MeshCoreConnector and related stores: update discovered contacts handling, migrate legacy keys, and set public key in community store
* Refactor MeshCoreConnector and ChannelsScreen: update discovered contacts handling and set public key in community store; enhance location plausibility check in MapScreen
* Update CMD_ADD_UPDATE_CONTACT frame format to include optional latitude and longitude fields
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Translated contact settings and related strings in Slovenian, Swedish, Ukrainian, Chinese, Dutch, Polish, Portuguese, Russian, and Slovak.
- Added new strings for discovered contacts actions such as adding, copying, and deleting contacts.
- Enhanced the DiscoveryContact model to include a rawPacket field for better data handling.
- Updated the contacts screen to support new actions in the context menu for discovered contacts.
- Improved the contact discovery store to handle the serialization of the new rawPacket field.
- Implemented contact settings in localization files for Swedish, Ukrainian, and Chinese.
- Added new DiscoveryContact model to handle discovered contacts.
- Created DiscoveryScreen to display discovered contacts with filtering and sorting options.
- Integrated contact discovery storage to persist discovered contacts.
- Updated settings screen to include options for automatic contact addition.
- Enhanced app bar and list filter widgets for better user experience.
- Fixed variable naming inconsistencies in contact model.