State storage

ClawSweeper has three explicit state owners. The Cloudflare Worker is canonical for review records, R2 is canonical for immutable action ledgers and published assets, and the state branch of openclaw/clawsweeper-state retains only the operational paths that have not migrated yet.

Record paths use the repository profile's lowercase slug: the owner/repository separator becomes a hyphen, while dots, underscores, and existing hyphens remain literal. For example, OpenClaw/Some.Repo_Debug uses records/openclaw-some.repo_debug/. Apply selection and cursor reads use the same slug as record publication and hydration; existing records need no migration.

Explicit manual reviews persist publication_policy: record_comment_only in host-owned report front matter and the matching publicationPolicy in existing queue decision JSON. There are no new tables, columns, or stores. The bundle's decision digest and current publication fence bind the report to its producer. Missing policy on pre-existing ordinary records retains existing behavior; unknown values, duplicate fields, and mismatches fail closed. Coalescing cannot widen a manual revision. Background apply skips these reports, and implementation discovery rejects them after queue completion.

Restricted publication also supplies transient authenticated owner metadata: the actual queue lease ID and claimed run/attempt, or the active batch ID, lease owner, runner run/attempt, and exact member generation. These reuse the coordinator's existing owner fields; they add no persisted ownership schema. A retained canonical receipt never authorizes an expired or different owner. After successful publication, the existing publication head advances the next manual request's revision. Direct retries still require the same accepted plan; active reclaimed batches retain their existing receipt convergence behavior. No receipt replacement, counter change, or historical claim adoption is added.

Publication lifecycle projections retain an immutable producer-lineage reference (fence, revision, and claim generation) from the admitted protocol-v2 publication in existing projection_json. Producer and publication revision counters are independent: the reference, never their numerical coincidence, connects the two journeys after queue-item deletion. Audit uses that observational resolution; physical receipts and terminal telemetry remain on the publication fence. A missing or conflicting link cannot complete another producer revision. This approved JSON extension adds no SQL table or column, grants no publication authority, and does not infer links for historical tupleless reports.

On startup the queue removes the retired Bay telemetry tables, command-journey mapping, and pending-telemetry index. Repository projection indexes remain in place to preserve existing query ordering. StatusStore deletes its two retired Bay history keys. The existing bay_telemetry_pending column stays inert: dropping it would rewrite all retained lifecycle rows on replicated storage while blocking startup. Its default still covers inserts; no application reads or updates it. These idempotent schema operations replace request-time pruning of the dead command mapping; no historical JSON backfill, table rewrite, or database vacuum runs. The lifecycle audit still uses the exact_review_lifecycle_projection_bay ordering index. Canonical lifecycle projections continue to own admission, receipts, acknowledgements, and audit history; older projection JSON remains readable without its former Bay and inline-proof markers. Direct and batch publication-outcome telemetry continues to back recent durable publication events.

Report readers share one anchored leading-front-matter parser. A unique header value remains authoritative when report prose or a valid fenced example quotes the same key. Duplicate header keys and competing unfenced metadata blocks (including fragments left by an injected terminator) make the affected fields ambiguous. A missing header field with a body lookalike stays ambiguous rather than enabling legacy close-promotion fallback. Indented nested data cannot override top-level fields. Decision packets reject document-wide structural ambiguity, including duplicate header keys, and remove stale packets using the real report path. Unrelated body-only keys do not create packet ambiguity. The advisory metadata-spoofing inventory remains deliberately broader than runtime authorization. No Bay schema or UI change is needed: the record shape and observer-only projection are unchanged.

setup-state hydrates records from the Worker and, when requested, ledger/assets from R2. Exact and batch review publishers pass the reviewed item numbers to read only their open/closed reports, plans, and decision packets. Those focused reads do not depend on repository-wide snapshots or journal exports.

PR hydration uses current GitHub inputs; only the bounded raw file window is retained in memory for the current review preflight. Review records do not persist pr_hydration_snapshot; later review and apply runs fetch current GitHub inputs. Canonical publication also strips that retired field from older publishers and restored snapshots while retaining the original authenticated retry receipts. Existing canonical reports drain through the finite record-hydration-cache-cleanup:v1 alarm checkpoint: at most 64 raw identities, four tuples and 8 MiB of source bodies per pass. Maintenance preserves report evidence, packets, producer provenance and answered state, and advances tuple and export revisions atomically. Conflicted tuples retry after the first scan; unresolved ownership stays explicitly incomplete. The unresolved count records deferred encounters, not distinct tuples. Logical contents shrink; reported database bytes are measured separately. The signed storage-stats response and maintenance summary expose committed removed-byte/field counters and completion or unresolved status without a separate payload census.

For explicit content attribution, run exact-review-queue-maintenance.yml with mode=storage-stats, execute=false and storage_table set to one user table. Each signed read scans at most 128 row identities and returns per-column TEXT/BLOB byte totals without returning contents. storage_max_pages limits a run to 1–1,000 pages (100 by default), with a four-minute total deadline. Resume an incomplete run by copying nextCursor into storage_cursor; each result reports totals only for that run's scanned range. Failed runs print their last completed page's totals and cursor before failing.

The census fixes the initial upper row identity; inserts above that bound are excluded. Concurrent updates and deletions mean pages are not one atomic snapshot. Numeric values, indexes, row headers and unused pages are excluded; these are exact observed content bytes, not occupied database pages. The ordinary sampled summary and reported database bytes remain separate measurements. Workerd reports occupied pages, including indexes and page overhead, while excluding free pages; this value is not the database file's allocation. Runtime-internal tables, WITHOUT ROWID tables and nonpositive row identities are rejected by this diagnostic. Tables wider than 98 columns exceed the Workerd result-column limit after adding the two pagination fields and are also rejected.

Explicit ledger/assets hydration streams one R2 listing page into its staging tree at a time; descriptor memory does not grow with the entire prefix. It still downloads the complete requested trees. Before replacing the local view, it checks each completed action-ledger import against its existing reservation and fetches any payload or binding missed by listing pagination. A malformed or incomplete committed import fails before replacement. Direct legacy shards with neither transaction marker remain readable.

An import with a valid reservation and an authoritative missing completion is unfinished. Hydration excludes its currently staged payloads, retains its bindings and all remote objects, and reports bounded producer examples and omitted counts. A later hydration can include it after publication completes. blobs and bytes count acquired objects (listed plus closureBlobs); trees counts the initial listings. materializedBlobs and materializedBytes exclude the reported omitted payloads. This is not a retention or pruning operation, and it does not promise a snapshot across R2 listing pages. Local tree replacement remains the existing remove-then-rename operation; custom subtree selection also retains its existing top-level replacement behavior. Jobs that need operational Git state receive a sparse checkout containing only the retained paths above. Canonical-only lanes set hydrate-git-state: "false" and never mint or use a state-repository token.

Repair-result publication consumes the Git-backed operational paths and sets hydrate-records: "false" with hydrate-state-blobs: "false". This avoids unused canonical-record discovery, snapshot caches, and R2 hydration while preserving the normal publication and validation steps. The equivalent CLI flags are --skip-records --skip-state-blobs; other callers retain canonical record hydration by default.

State-blob publication gives each signed Worker request, including multipart operations, a 15-second deadline. Requests retain their existing retry policy of at most four attempts, so a stalled upload eventually fails and releases the review worker.

Remaining Git writers use the Durable Object state-writer coordinator and one ordinary fetch/commit/push. The former Git lease refs, atomic multi-ref pushes, shallow-history deepening, remote-head rebuilds, record reconciliation, and immutable-ledger scratch branches no longer exist.

Target fanout and bounded placeholder recovery read and update /internal/state/cursors/<mode> with the same HMAC authentication as canonical record operations. Each record carries a monotonic revision so concurrent writers cannot silently overwrite one another. Cursor reads and writes are fail-open: an unavailable store emits a prominent warning, but productive work continues and remains safe to retry.

Eligible @clawsweeper re-review comment versions cross the durable boundary before acknowledgement or Actions dispatch. The queue keeps one immutable identity per comment id, update timestamp, and body digest; a newer edit supersedes older pending work, while retries reuse the same receipt. Detailed terminal receipts use the same 30-day horizon as resolved dead letters, and the per-comment watermark remains after receipt compaction.

Direct publication saves its lifecycle plan and receipt outcome on the claimed lease decision. Completion and terminal-run reconciliation use that saved authority for source-drift requeues, even if ingress has replaced the current decision. Recovery preserves the old target, fence, and revision's terminal fact; it does not manufacture router, acknowledgement, or review-result receipts. The queue activity projection shows the requeue disposition without inventing review success or failure.

Every review revision of an item comes from the durable per-item counter that re-review commands already used. A completed queue row is deleted, but its (fence, revision) tuple outlives it: a direct-publication receipt is kept for seven days and rejects different bytes at the same revision, and lifecycle projections are kept indefinitely. Each allocation therefore starts above the counter, the fence's highest retained direct receipt, the target's highest lifecycle projection under both the lowercase and the GitHub-cased key, and the publication head. A re-admitted item and an artifact-refresh recovery never reuse a published revision. Publication fences (<target>@publish:<run>:<attempt>) keep numbering their own revisions. No table or column is added; an item without a counter row is seeded from those floors on its next allocation.

Exact-review publication retries use R2 only as a cache in front of GitHub Artifacts. After a GitHub download passes the normal bundle validator, the publisher stores a deterministic byte-preserving archive at artifacts/exact-review/v1/<sha256> and records a queue receipt binding that digest to producer run id/attempt, artifact name, canonical item key, lease revision, and protocol version. Reuse requires an exact tuple match and a verified object digest; every miss or mismatch falls back to GitHub. Receipts expire after 30 days, and cache traffic incrementally prunes expired receipts plus old unreferenced R2 objects (including upload orphans) in bounded batches. Overlapping receipt requests share one cleanup pass within the queue instance, so they do not repeatedly scan the same R2 page or overwrite its cursor. A completed or failed pass releases that slot for the next request.

GitHub conditional-read entries stay in queue SQLite rather than runner-local files because publication runners are ephemeral. They share the 30-day receipt retention convention, cap the store at 2,048 entries and each JSON body at 512 KiB, and evict least-recently validated entries when full. The durable body is returned only by the post-304 confirmation operation; a lookup alone returns only its ETag and digest.

Ordinary webhook item, comment, and review payloads become eligible for retirement after 30 minutes of family inactivity. Bounded passes preserve active collections, unresolved placeholders, ordering metadata, and real deletion tombstones. Retired reads require authoritative repair before reuse. See the webhook read model for deployment approval, rebuild behavior, and the remaining metadata growth.

Canonical non-commit publication removes its replaced backfill body and chunks inside the same transaction as the canonical record and export pointer. An idle alarm also sweeps the historical backfill rowid range captured on first startup, at most 64 raw candidates per pass. Its durable cursor advances past ineligible rows and stops at that fixed upper bound. Deletion requires an exact record ID and agreement between the canonical record and canonical export pointer on revision, deletion state, and digest. Commits, live backfill exports, and missing or mismatched owners remain. Progress counts logical payload bytes; SQLite's physical database size can stay unchanged while freed pages become reusable.

Git-backed reports, dashboard status, and post-dispatch cursors are best-effort after their productive side effect or canonical publication succeeds. Git publication remains mandatory where it is still the durability fence before a dispatch, notably jobs/** intake and comment-router claims, and in the dedicated cluster-result publisher whose failure must stay visible for retry.

The state materializer and its append-window projection are fully retired. All producers were removed in the canonical-record cutover, the drain workflow was deleted after a week of zero-row runs, and the Durable Object drops the legacy state_append_* tables on upgrade. Canonical record and action-ledger writes go directly to the Worker and R2.

Cluster intake is the one ownership transfer required by that decision. Its workflow directly publishes the still-git jobs/ and results/ paths under the state-writer coordinator, persists the dispatch claim before the Actions side effect, and runs the same pending-claim recovery before accepting new work.

The repository is intentionally not archived or frozen by this migration. Archival is a separate operator action after the remaining Git-backed paths have their own canonical owners and the cutover has remained stable.