27 Commits (ee01c1917526eeec8c511693179a87c314387bec)

Author	SHA1	Message	Date
diffwireauto	5b061924c7	feat(#966): mock-ups, a confirmation that states the scope, and the eligibility bug it found ... The owner asked why there were no UI mock-ups to review. There was no good answer: this Epic's card was built and installed on both his machines without him ever seeing the design, and the plan had no step requiring otherwise. So the plan now has B0: mock-ups first, owner-approved before any of it is built, as a committed HTML file beside the plan in the format his firmware initiative already uses. epic-b-storage-mockups.html is that file: eleven numbered variants with real Material 3 tokens, a 360-wide pane, example values from his own install, and the behaviour to build as bullets under each. Owner-approved 2026-10-23. The confirmation (B1a, variants 8 to 10). It states how many copies, roughly how much space, how many messages are NOT affected, and that it cannot be undone, with the count on the action button so the scope is the last thing read. Cancel, a tap outside and back all decline. Nothing eligible opens no dialog at all, because a dialog that only says zero is not worth opening. That needed a preview pass: verify every eligible key and total it without deleting, so the numbers are real rather than estimated. Verification now runs twice for a confirmed clear, once to preview and once under the key lock. That is the cost of saying what will go before it goes. approximateBytes is named for what it is: the sum of blob lengths, not what VACUUM returns, because page rounding moves the real figure; the dialog says "about" and the result reports what actually changed. Writing the preview found a real bug, and it was about to bite him. The "skip anything stamped this pass" guard exists so installing an update never deletes a blob on sight, and it was being applied to the button as well. Every install that migrated before #962 has an empty __msgrows_migrated_at, so the first press stamped everything and cleared NOTHING. His install is exactly that: 47 migrated keys, none stamped. He would have pressed the button on a 70.9 MB database with 16.6 MB of reclaimable blobs and been told "nothing to clear". Fixed as eligibleByAge, one static rule both preview and run share, so the dialog cannot promise what the sweep will not do. An explicit press includes newly stamped keys; the automatic sweep still refuses them. Three tests on the rule directly, plus one proving the automatic path refuses while the press clears. Label stays "Clear Stale History Now" rather than "Compact", owner's call and the better argument: the name generalises to clearing stale history for other reasons later, such as #963's retention limit. 14 widget tests and 34 storage tests. Full suite 1,502 pass, 1 skipped. analyze clean, format clean. Task: meshcore-open-gan Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	3168c413c9	feat(#966): a durable audit of what the sweep deleted ... The owner asked whether the delete and compact are logged or silent. Checked rather than assumed, and the answer was bad on two counts: - A delete logged only a COUNT. "Cleared 40" names nothing, so there was no record of WHICH keys were reclaimed. Refusals were already logged properly, with the key and up to three missing records; deletions were not. - All of it goes through appLogger, which is gated on appDebugLogEnabled. That is off by default and off on his install, so on a normal install an irreversible delete left no trail at all. Both fixed. The summary now names the cleared and kept keys, and the sweep also writes a durable entry into the database itself, under __msgrows_reclaim_audit: timestamp, the cleared keys in full, the refused keys, and the byte counts either side. In the database because that survives restarts and the logging toggle, and can be read back without the app running, which is also how it gets verified on hardware. Key lists are stored in full. Truncating them would recreate the "Cleared 40" problem the entry exists to solve. Capped at the last 5 sweeps instead, which is generous given a key is cleared once and then has no blob to consider again, so this is close to a once-per-install operation. A failure to write the audit is logged and tolerated rather than failing the sweep: the deletes already happened and were each verified, so losing the record is worse than nothing but far better than reporting a correct sweep as broken. An unreadable audit likewise starts a fresh list and is logged, never silently replaced. scoping_inventory gained the key, globalOnPurpose: one sweep spans every radio and the entries name store keys that already carry the radio. Six tests: a sweep records which keys it deleted; the entry survives a fresh store object over the same database, so it is really on disk; refusals are recorded too; a sweep that did nothing writes no entry; the cap drops the oldest and keeps the newest; and an unreadable audit does not stop a sweep. Lands on Epic B's branch rather than Epic A's, where the sweep lives, to avoid restacking B over a changed A for a change that reaches dev through both either way. Origin is #965's engine. Full suite 1,489 pass, 1 skipped. analyze clean, format clean. Task: meshcore-open-gan Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	84c2dbd9da	fix(#965): act on the Gemini gate findings ... Three findings, all addressed. One was a real data-loss hole. 1. CRITICAL, fixed. missingFromRows trusted the converter's own output as the measure of the converter. A callback that silently dropped a record, rather than throwing, would have had every record it DID produce found present, returned "nothing missing", and the blob holding the only copy of the dropped one would have been deleted. The migration gate has always guarded this with an independent countJsonArrayElements in a separate compute; the verify path I added did not, which left the two halves of the same rule unequal. It now does the same cross-check and refuses as unverifiable on a mismatch. New test: a converter that drops its last row is refused with missing == null, not missing == []. 2. Measured, not redesigned. VACUUM takes an exclusive lock, and the review called it a UI freeze. It is not: drift_flutter runs SQLite in a spawned isolate (_isolateEntrypoint -> DriftIsolate.inCurrent, verified in the package), so database access queues behind it while the UI thread stays free. That is a stall on history-backed reads, not an ANR, which is the distinction that matters for an app whose storage work began as an ANR investigation. It is also close to once per install, since a cleared key has no blob to consider again. The duration is now logged, because 0.6 s on the owner's 70.9 MiB database is a number worth watching in the field rather than assuming. 3. Fixed, and the review was right about why. The non-history-key test asserted only that the isHistoryKey helper returns false, which would still pass if run() stopped calling it: the loop visits keys that HAVE rows, and contacts and friends normally have none, so they were never reachable anyway. Those keys now get a planted row and a 99-day-old timestamp, so the prefix guard inside run() is the only thing standing between them and a delete. Also carried over from the previous round: the check-then-delete window is closed by BlobStore.reclaimVerifiedHistoryBlob, which holds the per-key lock across both the verify and the delete and so also serialises the sweep against a migration or a blob merge on the same key. Two tests cover it, including that a second call on a blobless key reports neither a delete nor a problem, which is what makes a repeat sweep silent. On the owner's database 7 of 47 aged keys are in that state, having migrated through the empty-blob path. 18 tests in the file, full suite 1,472 pass, 1 skipped. Re-verified against a copy of the owner's real database after the changes: blob keys: 477, migrated history keys aged: 47 cleared 40, refused 0, 17362944 bytes returned (16.56 MiB) kept keys missing: 0, byte-changed: 0 integrity_check: ok, rows 29950 -> 29950 Task: meshcore-open-5vt Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	1b38c674e4	feat(#965): per-key migration timestamps and the stale-blob sweep ... Epic A of #962. The engine only; the manual action is #966. A1 wrote the diagnosis as failing tests first, and they were red for the right reason: no sweep existed and six BlobStore methods did not. Also lands a control test asserting that DELETE alone does not shrink the database, which passes today and must keep passing, since it is the entire reason a VACUUM step exists. A2, the timestamps. A second blob, __msgrows_migrated_at, keyed by store key. Deliberately NOT a reshape of __msgrows_migrated: that marker is read on every save and every load through a cache, so changing its shape would put a migration of its own on the hottest storage path. Owner decision 2026-10-22. An install that migrated before this ships has no timestamp, so backfillMigratedAt stamps it on FIRST SIGHT and the sweep skips anything it stamped this pass: updating the app can never delete a blob on the spot, and every key gets the full 30 days measured from when we first knew about it. A3, the sweep. Eligibility is age plus a fresh verification, never the marker alone, which is the difference between "we recorded that this migrated" and "we just confirmed it". missingFromRows returns null for "could not tell" and an empty list for "verified nothing missing", and those are deliberately different values: collapsing them is how the only copy of an unreadable history gets deleted. A4, one VACUUM, only when something was deleted. A failure there is logged and tolerated: the deletes already happened and were each verified, so it costs space and not data, and the next sweep retries it. The comparison rule now lives in ONE place, _absentFrom, used by both the migration gate and the sweep. The gate still decodes its own blob, because a second compute over multiple megabytes on the migration path would cost more than the duplication saves, so the rule is shared even though the decode is not. Wired into main.dart's existing post-frame callback, beside the #906 exit-info collector, for the same reason: storage hygiene must never be on the path to a usable screen. The catchError sits at that fire-and-forget call site rather than inside run(), so #966's button still gets a failure it can report. scoping_inventory gained the new key, because the #853 gate caught it exactly as designed. Its verdict is globalOnPurpose for the same reason as the marker: it is keyed BY store keys that already carry the radio. 15 tests, the plan's whole acceptance bar: the 29-day and 30-day boundaries; one missing row refuses AND the same key clears once its rows are complete, which is what proves the gate discriminates rather than never deleting; an undecodable blob refuses; an unstamped key is stamped not deleted; an unmigrated key is untouched; contacts, discovered_contacts and path_history_ are asserted non-candidates BY NAME; a DM key sweeps through the DM converter; and every kept blob is byte-identical across the sweep and VACUUM with the row count unchanged. A5, verification against a copy of the owner's real database through the production sweep: blob keys: 470, migrated history keys aged: 47 cleared 40, refused 0, 17367040 bytes returned (16.56 MiB) kept keys missing: 0, byte-changed: 0 integrity_check: ok, rows 29949 -> 29949 47 aged and 40 cleared because the other 7 migrated through the empty-blob path and never had a blob to reclaim. Full suite 1,469 pass, 1 skipped, up from 1,454. analyze clean, format clean. Task: meshcore-open-5vt Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	5f79c6e1db	fix(#943,#952): act on the Gemini gate findings ... Two of five findings were real. Log the downgrade refusal BEFORE the path lookup and the throw. onUpgrade runs inside drift's beforeOpen, so it fires on whichever isolate issued the first query, and whoever catches the exception decides what the user sees. A downgrade that only stopped background work, with nothing in the log, would be the quiet failure this branch exists to replace. The log line now lands first, so the record exists however the exception is handled. Strengthen the recovery bench. It asserted row COUNTS, so a converter producing the right number of rows with wrong contents would have passed. It now also checks that every identified blob record has a row carrying its identity, and that no payload is empty, both derived from the raw blob rather than from the converter under test so the check is not circular. Records with no messageId are not identity-checked, because reproducing the `x:` fallback key here would make it circular again; the count assertion still covers them. Verified on a copy of the owner's database with the trap reconstructed (the 6,308-message channel removed from the marker plus one planted row the blob does not contain): the key migrates, and all 6,308 identities are present. Then verified the new assertion can FAIL, by planting an empty payload the migration does not overwrite: Expected: empty Actual: WhereIterable<String>:['{}'] Three findings were rejected, with reasons in the PR body: the duplicate merge-key collapse is pre-existing and unchanged by this diff, the claimed migration/blob-save race cannot occur because of the re-check inside the per-key lock, and the path_provider call is bounded and already proven. Task: meshcore-open-lkp Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	b08b42bff8	fix(#943): refuse a newer schema loudly instead of rewriting the version ... Owner decision, 2026-10-22: downgrades stay unsupported, and must fail deliberately rather than silently. Before this, onUpgrade guarded on `from < 2`, so a database from a FUTURE schema fell straight through it. The open succeeded, and drift then stamped user_version back down to this build's 2, because it writes schemaVersion whenever the versions differed. The database's only record of having been on a newer build was destroyed, and a later run of that newer build would re-run its own upgrade against tables already in their new shape. onUpgrade now throws OffbandSchemaDowngrade when `from > to`, naming both schemas, saying the upgrade is one-way, and giving the database path when it can be resolved (path_provider has no web implementation, so the path is best-effort and the message stands without it). Not made to work, deliberately. Once a key is on rows its blob is frozen, so an older build would read stale history and write into the blob while this one reads rows, orphaning everything it received. Supporting a downgrade would mean dual-writing the blob on every save, which is the cost #917 exists to remove. The two tests that documented the old silent behaviour now assert the refusal, including that user_version is left at 3 on a file-backed database, which is the whole point of throwing. Three more guard the upgrade direction: an equal version takes neither branch, a fresh database still lands on 2, and a real 1 to 2 upgrade still carries its blobs across, since that is the path every existing install takes. Release-note disclosure and rollout staging are tracked separately. Task: meshcore-open-3e1 Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	eef192ea7f	fix(#952): the migration gate asserts a superset, not equal totals ... Four of the owner's store keys could never migrate to rows. Each held rows whose merge_key is absent from its blob (1, 3, 7 and 5 of them), and the readback gate compared the key's TOTAL row count against the number of distinct records in the blob. Those two numbers can never be brought level again: nothing removes the extra rows, and the frozen blob can never come to contain them. Every attempt refused, for good. His two largest channels were among them, 6,313 and 5,648 messages, so they kept re-encoding 3.77 MB and 3.55 MB of history on every save. That is the cost #920 removed and they were not getting it. No data loss: the blob stays complete and authoritative for a refused key. The gate asserted the wrong invariant. What it exists to catch is a record from the blob failing to reach a row, which is a SUBSET question. Equality became wrong the moment rows could legitimately hold messages the frozen blob does not, which is true for every migrated key the instant a new message arrives. - messageMergeKeys(storeKey) reads back the stored identities, and the gate now requires every deduped blob key to be among them. Rows beyond that are logged and accepted rather than treated as corruption. - The refusal log names how many records are missing and samples three, which is more use than the two totals it printed before. - The four keys need no healing migration: they migrate normally at next access, pinned by "a key stuck under the old gate recovers on next access". The step can still fail, and both halves of the loop are proved: a BlobStore whose insert silently drops a row is refused and stays on the blob, and a lossy converter is still refused before anything is written. Renamed the test file, since it now guards the gate rather than describing the trap it was written to demonstrate. Task: meshcore-open-lkp Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	e2b8970f1f	fix(#943): establish what governs a downgrade, and correct the claim in code ... #943 says drift "refuses to open a database whose user_version is newer than the binary's schemaVersion". That is wrong, and the real mechanism points somewhere else. drift has no newer-schema refusal. It compares the stored user_version against schemaVersion for INEQUALITY and routes both directions into onUpgrade ("Schema version upgrades and downgrades will both be run here"). The pre-#917 build declared schema 1 and no migration getter at all, so it got drift's default onUpgrade, which throws. That is why the owner's old install would not start: not a version check, but a missing strategy, so it never reached the blobs it would have found. The consequence matters for the decision. This build does NOT throw on a downgrade: the guard is `from < 2`, so a database from a future schema opens silently, and drift then stamps user_version back down to 2, erasing the only record that it has been through a newer build. The next schema bump inherits a silent failure rather than #943's loud one. - schema_downgrade_test.dart, six tests: a newer version reports as hadUpgrade rather than being refused; a bare strategy throws (the #943 failure); this build opens a schema-3 database without complaint; and it rewrites the version backwards. Two more pin that an equal version takes neither branch and that a fresh database still lands on 2, so the downgrade cases are genuinely testing the downgrade path. - Corrects the stale claim in offband_database.dart, which still told readers a downgrade finds its history. The merged plan was corrected in 4b8f5731; this is the same correction in the code it describes. Also lands migration_refusal_trap_test.dart for #952, found while gathering this evidence: four of the owner's store keys hold rows the blob does not, so the readback gate's equality check refuses them on every attempt, for good. Five tests, including the clean-key control and a check that no message is lost. Fix deliberately not included; #952 carries the decision. The three decisions in #943 are untouched and remain the owner's. Task: meshcore-open-3e1 Agent Mail: TealGlen Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	d20d567bba	task(#920): B3 blob-to-rows migration with a readback gate that can fail ... Per key, once: count the blob's records, convert it to rows off the UI isolate, insert them in one transaction, read the stored count back, and only then mark the key as reading from rows. **The blob is never deleted**, not even on a verified match. Rows become the read and write path; the blob is left untouched. Reclaiming that space is a separate owner-gated step, because the moment the delete runs is the moment a mistake stops being recoverable, and a downgrade to an older build still finds its history where it looks for it. Every failure leaves the blob authoritative and says so loudly: an undecodable blob, a converter that throws, a count mismatch, a failed insert. None of them retries, because a count that did not match once will not match on a loop. The migrated set is its own marker key, not "this key has rows". Those are different statements: a key can have rows and an incomplete migration at the same time, and treating rows as proof of completeness is how half a history would read as a whole one. An unreadable marker treats every key as unmigrated, which is the safe direction because the blobs are all still there. The first version of the readback gate was a check that could not fail. It compared the stored row count against the number of rows the converter produced, so a converter that silently dropped records matched itself and passed. Caught by a test built to drop a record. The blob's element count is now measured by a separate decode that does nothing but count, so the converter cannot report its own homework: `rows.length` must equal that count, and the stored count must equal the distinct identities. Both can fail independently, and together they close the loop from blob element to stored row. Duplicate identities in a source blob are deduped rather than refused. The old merge should not have produced any, but the legacy slot-index adoption path (#194) could have, and that must not read as data loss. Tests: 15, nine of them failure paths. The lossy-converter case is the one that caught the broken gate. Citadel: meshcore-open-x9q Refs #920, #917, #334 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	4d4d44a61a	task(#920): B1 message_records table and the per-message row API ... Schema 2 adds one row per message beside StoredBlobs, which stays exactly as it is. A save becomes an upsert of one row, so its cost stops depending on how much history is already stored. That is what #919 could not do: it moved the whole-history re-encode to another isolate, it did not remove it. Shape, and why it is this shape: store_key the EXISTING blob key, byte-identical merge_key the stores' existing _mergeKey identity timestamp_ms INTEGER, so ORDER BY and LIMIT run in SQLite payload the per-message JSON the stores already emit PK (store_key, merge_key), index (store_key, timestamp_ms) Reusing the blob key means every per-radio and per-channel scoping rule (#194, #472) carries over untouched, and the migration becomes a per-key operation with an exact record count to verify against. The payload is JSON rather than a column per field. A Message has 25 fields including three maps, two byte arrays and the translation set; hand-mapping them would mean trusting a new field-by-field converter against 41 MB of real history on its first run, and #355 cost 574 messages the last time a storage migration was hurried. With a payload column the migration is a pure re-key, the stores' existing encoders are reused verbatim, and a new message field needs no second migration. The trade is that message fields are not SQL-queryable; nothing asks for that today, and columns can be added beside the payload later without another data migration. Flagged for the owner on the issue before building it. The row API lives on BlobStore rather than in a second class because both APIs must share one OffbandDatabase: opening it twice is an error on the web backends, and a second singleton would be one mistake away from doing that. `limit` with no `offset` returns the NEWEST n, still oldest-first, because that is what a chat screen opens with. Tests: 18. The row set covers store-key scoping (a leak there would be a history bleed between radios), merge-key upsert, paging, per-row delete, and rows coexisting with a retained blob. The schema set builds a database that looks like one an older build left behind, v1 table with real data and user_version 1, and proves the upgrade runs, creates a usable table, leaves the blobs byte-identical and lands the index on both an upgraded and a fresh database. Citadel: meshcore-open-g6j Refs #920, #917, #334 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	1ed7581a51	task(#919): A3 both explicit delete paths off the UI isolate ... Deleting one message rewrites the entire blob, so on the owner's largest channel a delete blocked the UI for as long as a save did, right after the user tapped something. Both delete paths now go through `mergeOffThread`. Two call shapes, one helper. `mergeOffThread`'s second argument is renamed `incomingJson` to `payload`, because the callback decides what that string means: the incoming batch when saving, the target merge key when deleting. The method only carries it across the boundary. `dropFrom{Channel,Dm}HistoryJson` return null when the target is not in the history, so nothing is written at all. Re-encoding 5 MB to change nothing is exactly the cost this Epic exists to stop paying, and the old code did it on every miss. `MessageStore.removeMessage` also gains per-key serialisation it never had: it was a read-modify-write outside `BlobStore.synchronized`, so a save landing after a delete could resurrect the deleted message. There is a test for that race. Tests: 8 added in their own file, including both stores' public delete API, an absent target, an absent blob and a corrupt blob left byte-identical. Citadel: meshcore-open-z7c Refs #919, #917, #334 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
diffwireauto	4ddb6a4358	feat(#919): move the channel-history merge off the UI isolate ... BlobStore.mergeOffThread runs the whole read-modify-write through compute(): stored JSON and incoming JSON in, merged JSON out, so only strings cross the isolate boundary and the object graph never does. Keeps the per-key serialisation that stops concurrent saves clobbering each other. channel_message_store.saveChannelMessages now encodes only the INCOMING messages on the UI isolate (normally one message, measured 0.002 ms) and hands the 5.38 MB decode-merge-encode to another isolate, where it measured 134 ms per save and crossed Android's 5s input-dispatch watchdog in a burst (#334). Fails closed: a stored blob that will not decode throws out of the merge, is logged, and the key is NOT written, which is the same refusal the inline code made. compute runs inline on web, which has no isolates; that is documented behaviour, not a silent fallback, and #920 removes the cost for every platform. _messageToJson, _messageFromJson, _mergeKey, _repeatToJson and _repeatFromJson became static so the top-level merge can call them; none touched instance state. 9 new tests cover merge equivalence, timestamp ordering, empty/absent blobs, the throw on corrupt JSON, fail-closed leaving the blob byte-identical, and three concurrent saves all surviving. Fixtures go through the store's own encoder rather than hand-written maps, after a hand-rolled fixture rotted on a non-null field. Refs #334, #917 Agent: TealGlen (session c61ce804) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	3 weeks ago
Strycher	b96f20140f	chore(#457): remove em-dash characters from source ... Ben's style rule: no em-dash character in copy, docs, or code comments (it reads as an AI tell), and the repo is going public. Replaced the em-dashes in code comments and English UI strings with commas, colons, or periods, whichever reads best. User-facing strings were hand-tuned for natural punctuation rather than a blanket comma. Preserved the lone "no data" glyph placeholders (a standalone dash used as a not-available indicator in status displays); those are a design element, not prose. Regenerated app_localizations*.dart from app_en.arb (the English fallback for untranslated keys propagates to every locale's generated file). Non-English ARB translations left untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	4 months ago
Strycher	d68c6dfdd8	fix(#385): surface storage-init failure with a persistent warning ... When the database can't open (e.g. the native sqlite library fails to load), every read/write silently failed and the app opened to an empty, normal-looking screen — the user thinks their history was wiped (SAFELANE §6 violation). Probe the storage layer in main() before any store reads (BlobStore.verifyReadWrite). On failure, a StorageHealthService one-way latch records it, and a persistent, non-dismissable banner is shown above the whole app: history is not lost, storage is unavailable, messages are NOT being saved, restart after fixing. Cross-platform (probe goes through drift, covers web too). Tests: service state/latch + banner show/hide widget test. Gemini-reviewed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	4 months ago
Strycher	61b0eab423	feat(#367): consolidate all found message stores into the pinned DB ... #363 pins the DB to one directory and migrates ONE prior store in, but a user who ran differently-built copies can have data split across several stores. On startup (native only, after the prefs->drift migration) this discovers every store on the machine and unions its bulk blobs (messages by id, contacts by public key) into the current one, so nothing shows as a gap. Sources are read-only and never deleted. Not a one-shot: instead of a permanent flag it tracks each store's signature (mtime+size), so a store that a stray older build later creates or grows is re-merged rather than stranded. A store over a 200 MB guard, or one that cannot be read, is skipped and NOT recorded as done, so it retries later. Identity dedup is key-order independent. sqlite3 (dart:ffi) stays out of the web build via a conditional import. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	4 months ago
Strycher	b1fb227b4c	fix(#363): keep sqlite3 (dart:ffi) out of the web build ... The VACUUM INTO snapshot import pulled dart:ffi via package:sqlite3, which does not compile on web. Isolate it behind a conditional import (native impl + web stub); the migration is native-only and never runs on web. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	4 months ago
Strycher	e45946a3d3	fix(#363): pin the drift DB to one fixed directory, migrate old locations in ... The DB directory came from getApplicationSupportDirectory(), which on Windows is derived from the executable's Company/Product version metadata. Two builds with different metadata (or a future rebrand) resolved DIFFERENT %APPDATA% folders and read DIFFERENT databases, so a user's history appeared to vanish when they ran a different build. Pin the desktop DB to a constant path (Windows: %APPDATA%\Offband MeshCore; Linux pinned too, its support dir derives from the exe name). On first run at the pinned location, migrate an existing DB in via SQLite VACUUM INTO (a consistent snapshot that is safe even under a concurrent writer), choosing the DB from the known canonical locations first and falling back to a bounded scan of the app-data roots (no name blocklist) so a DB under an unknown folder is still found. On any snapshot failure it aborts cleanly, leaving the source intact and opening a fresh DB - never a raw file copy of a live WAL DB. Adds sqlite3 as a direct dependency (pinned to drift 2.34.2's resolved 3.5.0). Tests cover source selection, the folder-agnostic scan, and the snapshot happy + abort paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	4 months ago
Strycher	8af0e0d5dc	fix(#355): merge prefs into drift on migration instead of discarding ... The migration's already-present branch called prefs.remove(key) and threw the prefs copy away when drift already held the key. A build that writes to SharedPreferences (a non-drift build, or any in-between test build) collects new messages there, so the next drift run silently gapped that history out. It cost 574 real channel messages, recovered from backups. Union the prefs copy into drift by element identity (messageId, else publicKey, else canonical JSON), keeping drift's live copy on a collision and appending prefs-only elements. Verify the merged write before removing the source. New `merged` counter in the report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	4 months ago
Strycher	b7584bd0c2	chore(#352): generate drift code in CI instead of committing it ... Replaces the stopgap (53f87f5) that committed lib/storage/drift/offband_database.g.dart to green CI. Restores the repo's "generated files are not committed" convention: - dropped the .gitignore negation; the .g.dart is gitignored + untracked (git rm --cached, working copy kept) - added `dart run build_runner build --delete-conflicting-outputs` immediately after `flutter pub get` in every Dart job: flutter_dart (analyze), build.yml (all 6), release-signed (android-signed), deploy-web (before build_pipe) - keeps the drift_dev 2.34.0 / drift 2.34.2 / build_runner ^2.15.1 pins from the stopgap for deterministic generation Verified locally on the bench toolchain (3.44.1): deleted the working .g.dart, ran build_runner from scratch (regenerated cleanly), confirmed the generated file passes `dart format --set-exit-if-changed` and full `flutter analyze --fatal-infos --fatal-warnings` is clean with the file generated (not tracked). For #335 / PR #348. Agent: SapphireCompass (session 8d755b5e)	4 months ago
Strycher	53f87f51e3	build(#335): commit the drift generated code so CI can analyze/build ... CI failed on every job: offband_database.g.dart (drift's build_runner output) was gitignored and no workflow runs build_runner, so analyze and all six platform builds hit "Target of URI hasn't been generated". drift is the first code in this repo needing codegen, which is why dev's CI never had to. The workflows are not matrixed - six separate build jobs plus analyze, web deploy and release-signed, each with its own `flutter pub get`. Adding a codegen step to all of them is wide and easy to miss one. Instead the generated file is committed (drift supports this): one file, works across every job and workflow with no CI edits, and reproducible. drift_dev is pinned (2.34.0) alongside the already-pinned drift, so a future regeneration produces the same file rather than drifting from the committed copy. Regenerated against the pinned versions before committing. Follow-up worth having: a CI check that regenerates and diffs, so a schema change without regen fails loudly rather than shipping a stale .g.dart.	4 months ago
Strycher	3ff272ad8c	fix(#343): close the data-loss holes found by Gemini review ... Adversarial review (standards#145) of the storage/migration surface found four real defects, all data-integrity. All confirmed against the code and fixed; no false positives. BLOCKER - concurrent save race. saveChannelMessages / saveMessages are read-modify-write with an await gap, so two saves to the same key raced and the second clobbered the first, silently losing messages (e.g. a message and its delivery ack arriving together). BlobStore now provides synchronized(key, ...), a per-key operation chain; every RMW - merge-save, remove, and the load-path legacy migration - runs through it. Different keys stay concurrent. New test fires two concurrent saves and asserts the union survives. BLOCKER - non-atomic DB relocation. The documents->support move copied and deleted each file (.sqlite/-wal/-shm) in turn, so a failure after the main file moved stranded the DB across two locations and corrupted it. Now copies all files, verifies each by size, and only then deletes the sources; on any failure it rolls back the destination and leaves the original intact. MAJOR - load-path legacy migration raced save. The #194 index->PSK adoption did a blind write to the PSK key that could clobber a save that landed first. It now runs under the key lock and MERGES (union) instead of overwriting, so both the adopted history and any fresh message survive. MINOR - channel merge key lacked a sender. Two senders posting identical text at the same timestamp without a messageId would collide and lose one. The key now includes the sender, matching MessageStore. flutter analyze clean, dart format clean, 509 tests pass. Full Gemini log in docs/llm-consultations/.	4 months ago
Strycher	2fefb6aa91	fix(#343): keep full message history; window only in memory ... The persisted store was being overwritten with the windowed in-memory list, so any channel or DM with more than _messageWindowSize (200) messages lost everything older than the newest 200 on the first save after load. Observed live: Public went 232 -> 201 in one session on a build that already had the #333 fix, so this was not the race - it was windowing truncating the store. This is the slow-erosion cause behind the whole 566 -> 231 -> 206 -> 201 history. saveChannelMessages / saveMessages now MERGE into the persisted set instead of overwriting: upsert by message identity so older persisted messages are kept, new ones added, and the in-memory copy wins for edits/reactions/status. If the existing history fails to decode, the save aborts loudly rather than merging into an empty base and truncating (SAFELANE 6). Deletion is now an explicit path - removeChannelMessage / removeMessage - since the app deletes individual messages. Routing delete through the merging save would resurrect them; the connector's deleteChannelMessage / deleteMessage now call the explicit remove. Windowing stays for display and memory; it no longer dictates what is stored. Tests: 250-message history survives a 200-window save; new message appends; edit is captured not duplicated; delete does not resurrect. 508 pass, analyze and format clean. Cost: a save now re-reads and re-encodes the channel/contact history. Cheap with drift's per-key writes (#335); a future append-only schema removes even that.	4 months ago
Strycher	fa470c1442	fix(#335): store the drift DB in app-support, not OneDrive-synced Documents ... drift_flutter's native default is getApplicationDocumentsDirectory(), which on Windows resolves to the user's Documents folder - redirected into OneDrive on most machines (verified: drift_flutter 0.3.1 connect.dart). A live SQLite file syncing to OneDrive risks lock contention and corruption, and it is simply the wrong place for app data. The DB now opens in getApplicationSupportDirectory() (%APPDATA% on Windows), the same place SharedPreferences already lives, so all app data sits together. Existing installs already have a DB in the old location, so opening a fresh one there would abandon their migrated history. _appSupportDatabaseDirectory() relocates it once on first run: it moves offband_store.sqlite and its -wal/-shm sidecars from the documents dir to the support dir before drift opens, and never deletes a source without a successful copy. If relocation fails it is logged loudly and a fresh DB is created, with the migration re-running from SharedPreferences rather than silently losing anything. Web is unaffected: path_provider has no web backend and drift ignores databaseDirectory there, using OPFS/IndexedDB. Adds `path` as a direct dependency (was transitive). flutter analyze clean, tests pass.	4 months ago
Strycher	dd67391a0e	feat(#335): switch bulk stores over to drift ... Wires the migration into startup and points the four bulk stores at drift. SharedPreferences keeps the settings, which is what it is for. Startup runs the migration after prefs are up and BEFORE any store reads, and awaits it. Letting stores race a half-finished migration is precisely the shape of #333, where a storage path silently chose the wrong key and 566 real messages read as empty. Converted: message_store, channel_message_store, contact_store, contact_discovery_store. All four now have ZERO prefs get/set for bulk data. Three safety properties, deliberately built in: 1. readWithPrefsFallback - if a key is somehow not in drift, the prefs copy is still served rather than reading as empty. It logs a WARNING when it fires, because a fallback during normal operation means the migration is incomplete and someone needs to know. Silence here is what made #333 look like data loss. 2. deleteEverywhere / keysWithPrefix span BOTH backends. A clear that only removed the drift row would leave a pre-migration prefs copy to reappear through the fallback, resurrecting deleted history. 3. The legacy-key migrations inside the stores now check both backends and only touch prefs when a legacy key actually exists. This also carries the #306 fix into this branch: the unconditional prefs.remove was still present here, since this branched from dev rather than from the #306 work. Verified: analyze clean, format clean, 504 tests pass, Windows and web both build. Migration rehearsed earlier against a copy of a real 7 MB store: 69 keys, 5.21 MB, zero failures. NOT yet run against live data - that happens on first launch of this build, and the owner should have a prefs backup before that.	4 months ago
Strycher	a2c7302aad	feat(#335): migrate bulk data from SharedPreferences into drift ... Moves message history, contacts and discovered contacts out of the settings store. Settings stay in SharedPreferences, which is what it is for. Ordering is the whole safety argument: WRITE, VERIFY BY READING BACK, and only then remove the source. #333 was a storage path that chose a key silently and made 566 real messages read as empty; deleting before verifying would make that class of mistake permanent instead of cosmetic. On any failure the source is left intact and the error is logged - never a silent drop (SAFELANE 6). Idempotent by construction: a key already present in drift is not overwritten, so re-running is a no-op. If an older build re-writes a migrated key into prefs, the migrated copy wins and the stale prefs copy is discarded rather than promoted. Rehearsed against a COPY of a real 7 MB store, as the plan required before touching live data: REHEARSAL: 69 migrated, 0 failed, 5.21 MB, 69 bulk keys expected Every key checked for exact length, confirmed removed from prefs, and every settings key confirmed untouched. The live store was never opened. Two things the tests caught that review would not have: 1. getString THROWS on a non-string value rather than returning null, so a non-string under a bulk prefix was counted as a migration FAILURE. It now type-checks with prefs.get() and skips. Alarming falsely is its own bug. 2. The `contacts` prefix was checked against the real store rather than assumed: it matches only the 6 bulk contact blobs, and correctly does NOT match contact_unread_count*. Not yet wired into app startup - that is the switchover, and it is deliberately a separate commit so this can be reviewed on its own. flutter analyze clean, dart format clean, 504 tests pass.	4 months ago
Strycher	847a28df3e	feat(#335): prove drift/SQLite-WASM works in a real browser ... Step 1 gate, web half. Building is not evidence, so this was run in a browser. Assets: sqlite3.wasm (748 KB) and drift_worker.js (355 KB), both taken from the SAME drift-2.34.2 release so they are built against each other. Validated before use - the wasm has correct WebAssembly magic (0061 736d) and carries SQLite 3.53.3; the worker is genuine compiled Dart JS. Note the filename: the release ships `drift_worker.js`, not the `drift_worker.dart.js` the docs name. The database URI was corrected to match, which would otherwise have been a silent 404 at runtime. Browser result, served over HTTP with COOP/COEP and Content-Type: application/wasm: Using WasmStorageImplementation.opfsLocks PROBE-OK open+write+read PROBE-OK 6MiB round-trip (localStorage cap is 5MiB) drift selected OPFS, the top storage tier, and round-tripped 6 MiB - data the current localStorage-backed web build physically cannot hold. No exceptions. Adds web/_headers for Cloudflare Pages: forces application/wasm on the wasm asset and sets COOP/COEP for cross-origin isolation. Verified to ship into build/web. Without COOP/COEP drift still works, falling back to IndexedDB, which is slower but still unbounded next to localStorage - a performance tier, not a correctness requirement. web_probe/ is a standalone entrypoint that exercises ONLY the drift stack, so a pass or fail is unambiguous without BLE and the rest of the app in the way. It is not part of the app build. Still no user data touched and no store switched over. The migration is the next step. flutter analyze clean, dart format clean, 497 tests pass.	4 months ago
Strycher	19858698ee	feat(#335): add drift and prove it opens on the targets buildable here ... Step 1 of #335, deliberately before any data migration: prove the store works before trusting it with message history. Adds drift + drift_flutter, a key/value StoredBlobs table mapping 1:1 onto the existing SharedPreferences keys, and verifyReadWrite() as a runtime probe. Key/value rather than relational tables on purpose: it maps directly onto the current store interfaces, so callers do not change and the migration can be verified row-for-row against the old keys. Relational schemas come later, once this is proven and querying is actually wanted. Verified here: - 3 tests pass, including a 6 MiB round-trip - larger than the 5 MiB localStorage cap that makes the web build impossible today - flutter analyze clean - Windows release build succeeds - Web release build compiles Two findings from the gate that a plan-on-paper would have missed: 1. sqlite3_flutter_libs 0.6.0+eol is a DEPRECATED no-op. From sqlite3 v3.x it is unnecessary and drift_flutter already covers it, so it is not a direct dependency here. 2. The web build COMPILES BUT WOULD FAIL AT RUNTIME: drift needs sqlite3.wasm and drift_worker.dart.js shipped in web/, and neither is present. Per drift docs they are downloaded from GitHub releases, version-matched to pubspec.lock (drift 2.34.2, sqlite3 3.5.0), and the server must serve .wasm as Content-Type: application/wasm. Not done here - downloading files needs explicit human approval. Also noted: the drift_dev CLI does not compile at these versions (allSchemaEntities missing from the drift3_preview GeneratedDatabase), so its asset tooling is unavailable. build_runner code generation is unaffected. No user data is touched. No store is switched over. The migration is the next step and remains gated on the web assets question.	4 months ago