Qortal-Hub/docs/reticulum-chat-sync-architecture.md

# Reticulum Chat Sync Architecture

## Purpose

This document describes the current Reticulum chat sync model, why it does not scale well, and the target architecture we want for long-term correctness and performance.

The goal is not only to reduce noisy logs. The goal is to make Reticulum chat work well when:

- a user belongs to many groups,
- each group may have many active participants,
- overlay peers appear, disappear, or change often,
- Reticulum wire payloads are small,
- attachments and resource downloads are active,
- background unread/mention summaries must stay fresh,
- the active open chat must stay responsive.

## Current System

Reticulum chat currently uses a custom `RCHAT` protocol over the Reticulum overlay.

At a high level:

```text
Reticulum overlay
  -> RCHAT wire messages
  -> ReticulumChatManager
  -> SQLite event DB
  -> UI summaries / active chat / resource downloads
```

Events are signed and stored locally. The UI reads local history from SQLite and receives live events from the manager.

### Current Background Subscription Flow

When the app loads the user's joined groups, the renderer sends all local group memberships to the main process and subscribes to each group.

```text
Group.tsx
  -> setLocalGroupMemberships(all joined group IDs)
  -> subscribeGroup(groupId) for each joined group
```

That means Reticulum chat subscribes to groups before the user opens a specific group chat. This is intentional because background summaries, unread counts, and mentions need to work from the group list.

### Current Subscribe Behavior

Today, `subscribeGroup()` is too eager. A group subscription sends a full sync starter pack:

```text
subscribeGroup(group)
  -> sub
  -> author_heads_req
  -> sync_req after latest local event
  -> sync_req before earliest local event
```

The peer replies with:

```text
author_heads
sync_hints
sync_hints
sync_hints
...
```

If any hinted event is missing locally, we request the full event:

```text
event_hint / sync_hints
  -> event_req for missing event
  -> event_offer / resource transfer for full event payload
```

### Current Active Chat Flow

When the user opens a group/channel:

```text
useReticulumGroupChat(group, channel)
  -> subscribeGroup(group)
  -> subscribeChannel(group, channel)
  -> getHistory(group, channel)
```

The active chat gets local DB history immediately. But the network sync path still overlaps with the same group subscription machinery used by background groups.

### Current Resource Flow

Attachments and large event payloads use the resource transfer path:

```text
resource_find (rf)
  -> resource_have from a verified provider
  -> direct linked byte-range resource transfer
  -> local resource store
```

This is conceptually separate from chat sync. RCHAT overlay messages only discover providers and route provider hints; the byte-range request is sent on the dedicated resource link after a provider is known.

## Problems With The Current Model

### 1. Subscription Means Too Much

`sub` currently means:

```text
I am interested in this group.
Also send author heads.
Also send recent history.
Also send older history.
```

That is too much work for a background subscription.

### 2. Background Sync Can Behave Like Active Chat Sync

The app may be joined to many groups. Background group subscriptions are needed for summaries and mentions, but they should not trigger full history sync for every group.

Current behavior can scale like:

```text
joined groups * overlay peers * subscription replay * sync windows
```

That creates unnecessary `sub`, `author_heads_req`, `sync_req`, and `sync_hints` traffic.

### 3. Peer Churn Replays Too Much

When overlay peers change or the bridge recovers, subscriptions can be reannounced. Today that can replay full sync intent, not just liveness.

```text
peer changed
  -> reannounce subscriptions
  -> sub + author_heads_req + sync_req + sync_req
```

Peer churn should not cause history sync storms.

### 4. Sync Happens Before We Prove Anything Is Missing

The current system asks for history windows first, then dedupes after receiving hints.

```text
ask for history
  -> receive many hints
  -> discover most are already known
```

The better model is to exchange state first, then request exact missing data.

### 5. It Does Not Scale To Large Groups Cleanly

A group can have many users. The sync system must never depend on:

- full group membership lists,
- all authors in a group,
- all events in a group,
- all channels in one packet.

Sync must scale with actual missing event activity, not group size or network size.

## Target Architecture

The target architecture is:

```text
Local-first signed event-log replication
  + compact checkpoints/digests
  + cursor-based feed sync
  + exact missing-range repair
  + adaptive peer/source planner
  + strict priority scheduling
  + isolated resource transfer
```

In simpler terms:

```text
Reticulum = transport
RCHAT = replication protocol
SQLite = local source of truth
UI = local projection
resources = large object transfer
```

RCHAT should sync signed event logs, not chat screens.

## Core Concept

Every chat event is immutable and signed.

```text
eventId
groupId
channelId
authorAddress
authorSeq
timestamp
eventType
payloadHash
signature
```

Each author has an append-only event sequence within a group:

```text
Alice in group 716:
  seq 1
  seq 2
  seq 3

Bob in group 716:
  seq 1
  seq 2
```

Nodes do not sync users. They sync events.

There are two sync tools:

```text
Normal sync:
  group/channel feed cursors
  "I have this timeline up to event X"

Gap repair:
  author sequence ranges
  "I saw Bob seq 42 but only have Bob seq 39"
```

Author sequence state is important for integrity repair, but it must not be the main digest for large groups. A group can have many chatters, so the primary sync primitive must be bounded group/channel cursors.

## Cursor And Continuity Semantics

Cursor semantics must be deterministic. Every peer must agree on what "after this event" and "before this event" means.

### Feed Cursor

A feed cursor is a position in a group/channel timeline.

```text
feedCursor
  feedTimestamp
  eventId
```

The canonical feed order is:

```sql
ORDER BY feed_timestamp ASC, event_id ASC
```

All feed queries, feed responses, hashes, and comparisons must use this exact ordering. A cursor means "the position at this feedTimestamp/eventId in the canonical feed order".

The feed cursor is cross-author. It should not include `authorAddress` or `authorSeq`, because those fields describe a single author's append-only sequence, not the mixed channel timeline.

### Timestamp Trust

The signed event timestamp must not be trusted blindly for feed ordering.

The system should distinguish:

```text
signedTimestamp:
  timestamp inside the signed event
  retained as event metadata

acceptedAt:
  local receive/insert time
  useful for abuse handling and local projections

feedTimestamp:
  canonical timestamp used for feed ordering and cursors
```

Rules:

```text
if signedTimestamp is within accepted past/future bounds:
  feedTimestamp = signedTimestamp

if signedTimestamp is too far in the future or past:
  reject or quarantine the event

if suspicious events are accepted for diagnostics:
  feedTimestamp = acceptedAt
  keep signedTimestamp as metadata only
```

The existing event timestamp validation rules should be tightened around this model. A malicious or broken client must not be able to force an event far into the past or future and distort feed cursors, unread summaries, or sync windows.

### Author Cursor

An author cursor is used only for per-author integrity repair.

```text
authorCursor
  authorAddress
  authorSeq
```

Author cursors answer a different question:

```text
Do I have every event for this author up to this sequence?
```

Feed cursors and author cursors must not be treated as interchangeable.

### Latest Does Not Mean Complete

A latest cursor only proves freshness:

```text
I have seen an event at least as new as this cursor.
```

It does not prove continuity:

```text
I have every event before this cursor.
```

The implementation must not assume that having the latest event means there are no holes behind it.

Continuity must be tracked separately:

```text
background summaries:
  latest cursor is enough for freshness

active visible chat:
  visible window should be verified as continuous

author sequence gaps:
  repaired with author range requests
```

### Page Metadata

Every feed response should include page metadata:

```text
pageStartCursor
pageEndCursor
hasMore
windowHash
```

`windowHash` is a deterministic hash of the ordered event IDs in the returned page. It lets peers detect divergence when they believe they are talking about the same cursor window but have different local contents.

## Proposed Protocol Model

### 1. Hello

Peers announce the protocol version and runtime capabilities.

```text
hello
  version: 1
  features:
    digest
    feed_req
    range_req
    event_batch
    resource_v2
```

This is not a backward-compatibility layer. It is a sanity check so peers can reject unknown or incomplete protocol implementations instead of silently falling back to noisy behavior.

### 2. Group Subscription

`sub` becomes liveness and interest only.

```text
group_sub
  groups: [716, 812]
  mode: summary
```

It should not automatically mean "send me history".

### 3. State Digest

A digest says what local event state we already have.

```text
group_digest
  groupId: 716
  latestCursor:
    eventId: abc123
    feedTimestamp: 1782340000
  channels:
    general:
      latestCursor:
        eventId: abc123
        feedTimestamp: 1782340000
      oldestCursor:
        eventId: def456
        feedTimestamp: 1782330000
      visibleWindowHash: ...
  digestHash: ...
```

This is cursor based. A cursor is a bookmark that means "I have events up to this point" or "I have visible history back to this point".

The latest cursor is freshness state only. It is not a promise that all previous events are present locally.

For large groups, the digest remains bounded:

```text
group_digest
  groupId: 716
  latestCursor:
    eventId: abc123
    feedTimestamp: 1782340000
  channelCount: 100
  channels: limited page only
  moreChannels: true
```

The system never sends all authors for a huge group by default.

A small `recentAuthorSample` can exist as an optimization, but it must be optional, bounded, and never required for correctness.

```text
recentAuthorSample:
  max 10-50 authors
  recently observed locally
  used only to detect likely gaps faster
```

### 4. Feed Request

After comparing digests, request only missing feed pages.

```text
feed_req
  groupId: 716
  channelId: general
  after:
    eventId: abc123
    feedTimestamp: 1782340000
  limit: 25
```

The receiver must answer according to the canonical feed order:

```sql
WHERE (feed_timestamp > cursor.feedTimestamp)
   OR (feed_timestamp = cursor.feedTimestamp AND event_id > cursor.eventId)
ORDER BY feed_timestamp ASC, event_id ASC
LIMIT pageLimit
```

For older visible history:

```text
feed_req
  groupId: 716
  channelId: general
  before:
    eventId: def456
    feedTimestamp: 1782330000
  limit: 25
```

Older history uses the inverse cursor comparison but still returns events in canonical ascending order.

### 5. Gap Repair Request

Author sequence ranges are used only when a received event or hint proves a gap for a known author.

```text
range_req
  groupId: 716
  ranges:
    Bob: 19-20
    Carol: 8-12
```

This should not require enumerating every author in the group.

### 6. Event Batch

The peer returns compact event envelopes or full events only when they fit.

```text
event_batch
  groupId: 716
  channelId: general
  pageStartCursor:
    eventId: event-a
    feedTimestamp: 1782340100
  pageEndCursor:
    eventId: event-z
    feedTimestamp: 1782340200
  hasMore: true
  windowHash: ...
  events:
    ...
```

Oversized event payloads continue through the resource path.

The `windowHash` is computed from the ordered event IDs in `events`. It is not a security primitive; event signatures still provide authenticity. It is a cheap divergence detector.

### 7. Resource Transfer

Attachments and large event blobs remain separate:

```text
resource_find (rf)
  -> resource_have
  -> linked byte-range resource transfer
```

Resource transfer should not be starved by background sync traffic. Overlay discovery stays bounded, and bulk/request bytes stay on the direct resource link.

## Background Vs Active Sync

The target design splits sync into two tiers.

### Background Group Sync

Used for:

- unread counts,
- mentions,
- latest summaries,
- notification readiness.

Behavior:

```text
background group membership
  -> group_sub summary
  -> compact cursor digest pages
  -> newest missing feed pages only when needed
```

Background sync must be cheap, low priority, and bounded.

It must not pull full history for every joined group.

### Active Chat Sync

Used when the user opens a group/channel.

Behavior:

```text
active channel opened
  -> load local history immediately
  -> active channel cursor digest
  -> request missing visible feed pages
  -> page older history only when user scrolls
```

Active chat gets higher priority than background group sync.

## Priority Scheduler

RCHAT should have a small internal scheduler so not all work competes equally.

Suggested priorities:

```text
P0 live active chat hints
P1 active chat missing event pulls
P2 channel metadata for open group
P3 resource transfer control
P4 background unread/mention sync
P5 periodic subscription refresh
```

Rules:

- Active chat should stay responsive.
- Background sync must yield during resource transfers.
- Periodic refresh must never flood the bridge.
- Duplicate control messages should be suppressed before sending.

## Peer Planner

Overlay peers are sources, not state owners.

The planner should track:

```text
peer health
groups peer appears useful for
last successful event/resource response
recent failed requests
known digest/cursor state
in-flight requests
```

Then decide:

```text
what is missing?
which peer likely has it?
how urgent is it?
have we asked recently?
will it fit in one wire?
is a resource transfer active?
```

This prevents repeated requests to bad peers and avoids duplicating work across peers.

## Wire Size Strategy

Every control message must be pageable and checked before sending.

```text
Build page:
add next group/channel/range
  check wireFitsReticulum()
  if too large, flush current page
```

If even a single group digest is too large:

```text
send minimal group checkpoint:
  groupId
  latest cursor
  channelCount
  digestHash
```

The peer can request detailed pages only if needed.

## Cursor Edge Cases And Conflict Rules

The cursor rules must be explicit so different implementations do not diverge.

### Same Feed Timestamp

If two events have the same `feedTimestamp`, the canonical tie-breaker is `eventId`.

```text
ORDER BY feed_timestamp ASC, event_id ASC
```

This rule applies everywhere:

- local DB queries,
- digest generation,
- feed requests,
- event batches,
- window hashes,
- cursor comparisons.

### Late Older Event

If a valid event arrives later with a `feedTimestamp` older than the current latest cursor:

```text
insert it into the local event DB normally
do not move latest cursor backward
update or invalidate affected verified windows
trigger repair only if it affects an active visible window or summary range
```

The latest cursor is monotonic forward. It represents freshness, not complete history.

### Overlapping Feed Page

If a feed page overlaps events we already have:

```text
dedupe by eventId
validate any duplicate event if needed
verify windowHash if this is a known comparable window
continue from pageEndCursor
```

The sync cursor advances from the page metadata, not from the last newly inserted event. This prevents getting stuck on pages that are mostly duplicates.

### Peer Sends Events Outside Requested Bounds

If a peer returns events outside the requested `feed_req` bounds:

```text
ignore out-of-bounds events for that response
do not update peer cursor from the invalid page
record a peer protocol violation
penalize the peer if violations repeat
```

This prevents one bad peer from poisoning local cursor state.

### Duplicate Event

If the same `eventId` arrives again:

```text
if stored event hash/signature matches:
  ignore as duplicate

if stored event hash/signature differs:
  reject as conflict/malicious
  record diagnostic evidence
```

The event ID is immutable. A different payload or signature for the same `eventId` must never replace local state.

### Window Hash Mismatch

If two peers claim the same cursor window but provide different `windowHash` values:

```text
do not mark the window verified
request the page from another peer if available
fall back to exact event ID/range repair if needed
penalize peers that repeatedly disagree with valid majority data
```

The window hash is a divergence detector. Event signatures remain the authority for individual event validity.

## Initial Implementation Constants

The protocol must define concrete limits before implementation. Without shared constants, two correct-looking implementations can behave very differently under load.

These are initial defaults. They should be tuned from live metrics, but every implementation should start from the same bounded behavior.

```text
MAX_FEED_PAGE_EVENTS = 25
MAX_DIGEST_GROUPS_PER_PAGE = 20
MAX_GROUPS_PER_SUB_PAGE = 50
MAX_DIGEST_CHANNELS_PER_GROUP = 16
MAX_RECENT_AUTHOR_SAMPLE = 20

MAX_IN_FLIGHT_PER_PEER = 4
MAX_IN_FLIGHT_ACTIVE = 8
MAX_IN_FLIGHT_BACKGROUND = 8
MAX_BACKGROUND_WORK_PER_TICK = 10
SYNC_TICK_MS = 250

DIGEST_DEDUPE_TTL_MS = 30_000
BACKGROUND_DIGEST_REFRESH_MS = 60_000
ACTIVE_DIGEST_REFRESH_MS = 10_000

TIMESTAMP_FUTURE_TOLERANCE_MS = 5 * 60_000
TIMESTAMP_PAST_TOLERANCE_MS = 30 * 24 * 60 * 60_000

PEER_VIOLATION_COOLDOWN_MS = 5 * 60_000
MAX_PEER_VIOLATIONS_BEFORE_COOLDOWN = 3
```

Meaning:

- `MAX_FEED_PAGE_EVENTS` bounds event batch size.
- `MAX_DIGEST_GROUPS_PER_PAGE` and `MAX_GROUPS_PER_SUB_PAGE` prevent group membership replay from becoming one huge packet.
- `MAX_DIGEST_CHANNELS_PER_GROUP` prevents a large group with many channels from filling one digest.
- `MAX_RECENT_AUTHOR_SAMPLE` is only an optimization hint. Correctness must not depend on it.
- `MAX_IN_FLIGHT_*` prevents one peer or background sync from occupying the whole scheduler.
- `MAX_BACKGROUND_WORK_PER_TICK` and `SYNC_TICK_MS` make background sync cooperative.
- timestamp tolerances protect feed ordering from broken or malicious device clocks.
- peer violation cooldown prevents repeated malformed pages from wasting bandwidth.

All message builders must still call `wireFitsReticulum()`. These constants are upper bounds, not permission to exceed the real wire limit.

## Data Model Additions

The immutable event table remains the source of truth.

Useful additions:

```text
rchat_peer_group_state
  peer_hash
  group_id
  latest_event_id
  latest_feed_timestamp
  digest_hash
  updated_at

rchat_peer_channel_state
  peer_hash
  group_id
  channel_id
  latest_event_id
  latest_feed_timestamp
  oldest_event_id
  oldest_feed_timestamp
  visible_window_hash
  updated_at

rchat_verified_windows
  group_id
  channel_id
  start_event_id
  start_feed_timestamp
  end_event_id
  end_feed_timestamp
  window_hash
  verified_at

rchat_missing_ranges
  group_id
  author_address
  from_seq
  to_seq
  preferred_peer
  attempts
  next_attempt_at

rchat_sync_queue
  priority
  group_id
  channel_id
  peer_hash
  operation
  dedupe_key
  next_attempt_at
```

Projection data remains local:

```text
summaries
mentions
search index
channels/categories
read watermarks
```

These are derived from events, not synced as primary state.

## Required Database Indexes

The sync design depends on fast cursor and repair queries. These indexes should exist before enabling the new protocol.

The existing immutable event table should keep `event_id` as its unique event identity. If the table remains named `reticulum_chat_events`, use indexes like:

```sql
CREATE INDEX IF NOT EXISTS idx_reticulum_chat_events_feed
ON reticulum_chat_events(group_id, channel_id, feed_timestamp, event_id);

CREATE INDEX IF NOT EXISTS idx_reticulum_chat_events_author_seq
ON reticulum_chat_events(group_id, author_address, author_seq);
```

The new sync state tables should have:

```sql
CREATE INDEX IF NOT EXISTS idx_rchat_sync_queue_ready
ON rchat_sync_queue(priority, next_attempt_at);

CREATE INDEX IF NOT EXISTS idx_rchat_peer_channel_state
ON rchat_peer_channel_state(peer_hash, group_id, channel_id);

CREATE INDEX IF NOT EXISTS idx_rchat_peer_group_state
ON rchat_peer_group_state(peer_hash, group_id);

CREATE INDEX IF NOT EXISTS idx_rchat_missing_ranges_ready
ON rchat_missing_ranges(group_id, author_address, next_attempt_at);

CREATE INDEX IF NOT EXISTS idx_rchat_verified_windows_lookup
ON rchat_verified_windows(group_id, channel_id, start_feed_timestamp, end_feed_timestamp);
```

If the event table is renamed during the rewrite, keep the same index shapes on the replacement table.

## Current Vs Proposed

### Current

```text
subscribe / replay
  -> send sub
  -> send author_heads_req
  -> send sync_req after
  -> send sync_req before
  -> receive many sync_hints
  -> dedupe after traffic already happened
```

### Proposed

```text
subscribe / replay
  -> send liveness sub
  -> send compact cursor digest
  -> compare state
  -> request missing feed pages or exact repair ranges
  -> transfer only missing events/resources
```

## Implementation Plan

This design should replace the current eager sync model directly. Since this is not production protocol compatibility work, the implementation should remove the old behavior instead of carrying it indefinitely.

### 1. Define The New Wire Protocol

- Define implementation constants and database indexes first.
- Add `hello`.
- Add `group_sub`.
- Add `group_digest`.
- Add `feed_req`.
- Add `range_req`.
- Add `event_batch`.
- Use `resource_find` / `resource_have` for resource provider discovery.
- Send byte-range resource requests only after opening the dedicated provider link.
- Every message builder must use `wireFitsReticulum()` while paging.

### 2. Replace Subscription Semantics

- `subscribeGroup()` becomes background-light.
- Group subscription sends only `group_sub` and bounded summary digest work.
- Subscription replay sends liveness/digest only.
- Remove automatic `author_heads_req`, `sync_req after`, and `sync_req before` from group subscription.
- `subscribeChannel()` becomes the active chat sync trigger.

### 3. Build Cursor Digest Sync

- Add group-level latest cursors.
- Add channel-level latest and oldest cursors.
- Build bounded digest pages by group/channel.
- Store peer digest/cursor state.
- Define and enforce canonical feed ordering everywhere.
- Use validated `feedTimestamp`, not raw signed timestamps, for feed cursors and ordering.
- Reject or quarantine events with timestamps outside accepted bounds.
- Track latest freshness separately from verified continuity.
- Suppress duplicate digest pages per peer/group/channel.

### 4. Build Feed Requests And Event Batches

- Compare peer cursors with local cursors.
- Request newer feed pages with `feed_req after`.
- Request older visible pages with `feed_req before` only for active chat or scrollback.
- Return `event_batch` pages that always fit the wire limit.
- Include `pageStartCursor`, `pageEndCursor`, `hasMore`, and `windowHash` in every event batch.
- Verify active visible windows independently from latest freshness.
- Use resource transfer for oversized event payloads.

### 5. Keep Author Sequence Repair

- Keep `authorSeq` validation on every event.
- When an inbound event proves a known author's sequence gap, enqueue `range_req`.
- Do not enumerate all authors for a group.
- Treat author heads as optional bounded repair/diagnostic data only.

### 6. Add Scheduler And Peer Planner

- Prioritize active chat and resource control.
- Track peer usefulness and failures.
- Add per-peer/per-group dedupe keys.
- Cap background work per tick.
- Keep background sync low priority.
- Make resource transfer control higher priority than background digest chatter.

### 7. Update Resource Transfer Integration

- Keep resource bytes on the existing resource transfer path.
- Keep event/resource manifests as signed chat events.
- Ensure resource progress/events are coalesced so transfer UI does not flood the bridge.
- Ensure background sync yields while large resource transfers are active.

### 8. Remove Legacy Eager Sync Paths

- Remove subscription-triggered `author_heads_req`.
- Remove subscription-triggered `sync_req after`.
- Remove subscription-triggered `sync_req before`.
- Remove automatic continuation behavior that can create unbounded `sync_hints` bursts.
- Replace `sync_hints` with bounded `event_batch` responses.

### 9. Add Diagnostics And Tests

- Log digest pages sent/skipped.
- Log feed requests and event batches.
- Log duplicate suppression.
- Log scheduler queue depth by priority.
- Test many groups, large groups, peer churn, active chat, background summaries, and resource transfer under load.

## Success Criteria

- Subscribing to many groups does not create a sync storm.
- Overlay peer churn causes small digest/liveness traffic only.
- Background summaries remain fresh.
- Active chat receives missing visible events quickly.
- Large resource downloads are not slowed by background sync chatter.
- `sync_hints` volume drops sharply for already-synced peers.
- Bridge event queue does not build multi-second backlogs during normal chat/resource use.
- Wire-size violations are impossible by construction.
- No protocol step depends on enumerating all group members or all group authors.
- Author heads are optional repair/optimization data, not the primary sync state.

## Final Principle

The final system should behave like this:

```text
Local DB is truth.
Peers exchange compact state.
Planner fetches missing feed pages and exact repair gaps.
UI renders local projections.
Large bytes use resource transfer.
```

That is the scalable foundation for Reticulum chat.