Skip to content

feat(network): announce send chunks as an all-or-nothing batch#53

Merged
andinux merged 1 commit into
codex/chunked-payloads-networkfrom
feature/apply-batch-send
Jul 3, 2026
Merged

feat(network): announce send chunks as an all-or-nothing batch#53
andinux merged 1 commit into
codex/chunked-payloads-networkfrom
feature/apply-batch-send

Conversation

@andinux

@andinux andinux commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

/apply coverage on the server is db_version-granular, but a send chunk is not a db_version: one chunk can span several db_versions (no db_version-boundary flush in the chunk builder), and a value larger than max_chunk_size fragments across several chunks within one db_version. So the first chunk touching db_version N marks N covered before N is complete — a later chunk's failure is masked (failures.apply hidden, columns silently missing) and lastOptimisticVersion advances on a partially received db_version.

Fix (client side)

One send_changes call = one all-or-nothing batch:

  • Every chunk announces the same global window dbVersionMin = send_checkpoint+1, dbVersionMax = watermark, plus three new fields: batchId (UUIDv7 per call), chunkIndex (0-based), isFinal (true on the last chunk; already computed by the cloudsync_payload_chunks vtab).
  • This replaces the per-chunk announce-window tiling (network_announce_min removed) — the fixed window covers db_versions consumed by non-local-change transactions by construction.
  • The server (companion change in the cloudsync server repo) advances optimistic only when the final chunk is received, and confirms + appends the whole window only when every chunk of the batch applied without error. A failed batch is never merged: the window reads back as a gap, the failure is surfaced, and the client re-sends it whole under a new batchId. The client checkpoint already follows lastOptimisticVersion backward, so the regress-on-failure signal works unchanged.
  • Old clients (monolithic payloads) send no batchId → the server keeps today's per-payload behavior, which is correct because a monolithic payload is a complete db_version set.

Changes

  • src/network/network.c: batch UUID per send call; fixed global window announce; batchId/chunkIndex/isFinal in the /apply JSON (both blob and url transports); send-loop SELECT reads is_final instead of db_version_min/max; tiling logic removed.
  • src/network/network_private.h: network_announce_min helper removed; network_apply_json_payload exposed for the network unit test.
  • test/network_unit.c: announce-tiling test replaced by test_apply_json_payload_batch (fixed window, batch fields, both transports, missing-batch-id rejection).

Compatibility

The current server ignores the new fields and appends the announced window per chunk, so this client can ship ahead of the server change. Response wire format is unchanged (lastOptimisticVersion/lastConfirmedVersion stay integer db_versions).

Testing

  • make unittest — all pass (incl. memory-leaks check)
  • make network-unittest — all pass (incl. the new batch-payload test)
  • make e2e against the live (pre-batch) server — all pass, incl. Send Gap From Clock Hole and all Chunked tests

🤖 Generated with Claude Code

One send_changes call = one batch: every chunk announces the same global
window [send_checkpoint+1 .. watermark] plus batchId (UUIDv7 per call),
chunkIndex and isFinal, replacing the per-chunk announce-window tiling.
The server can then advance optimistic only when the final chunk is
received and confirm the whole window only when every chunk of the batch
applied - a failed batch is never merged, so a partially applied
db_version can no longer be masked as covered; the client re-sends the
window under a new batchId.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@andinux andinux changed the base branch from main to codex/chunked-payloads-network July 3, 2026 16:32
@andinux andinux merged commit 3de67c3 into codex/chunked-payloads-network Jul 3, 2026
29 checks passed
@andinux andinux deleted the feature/apply-batch-send branch July 3, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant