improvement(mothership): user_table speed parity — limit bounds, keyset paging, background import/delete jobs

[TheodoreSpeaks]

Context

Mothership's user_table tool lagged the fast/safe paths the tables feature already has. This PR lands three of the four findings from the audit; the fourth (the function_execute large-table CSV mount) is deferred — see note at the bottom.

Changes

C — limit bounds. query_rows took an unbounded limit (whole table into one tool result) and the filter ops took unbounded limits (every matching row's id+data into memory). Now rejected above MAX_QUERY_LIMIT (1000) / MAX_BULK_OPERATION_SIZE (1000) with the contract's message text.

D — query_rows waste + paging. Dropped the always-on executions-metadata load (withExecutions: false), and added keyset pagination: accepts an after cursor, returns nextCursor when a default-order page fills (rejects after+sort). Offset paging was O(n²) walking a big table.

E — async job parity for imports/deletes. import_file / create_from_file (CSV/TSV ≥8MB) and delete_rows_by_filter (>1000 matches, no explicit limit) now dispatch the same trigger.dev jobs the UI routes use, claiming the per-table job slot so they can't race a running background job. Smaller imports stay inline but claim/release the slot. The model polls progress via query_rows (rows appear as an import lands; the delete mask makes query_rows reflect the post-delete view immediately) — no dedicated job-status op. TableImportPayload.deleteSourceFile flag (default true) so Mothership imports of persistent workspace files don't delete the source.

Tests

user-table.test.ts (limit clamps, keyset after/nextCursor, after+sort rejection, slot claim/release, async dispatch payloads) and import-runner.test.ts (source-file cleanup default vs deleteSourceFile: false). bun run lint:check, type-check, and check:api-validation green.

Deferred (not in this PR)

• function_execute large-table mount (finding B): the keyset CSV drain fixed the 100-row truncation but materialized the whole table in the web-container heap before the 50MB check, risking OOM on large/enterprise tables. Backed out here (mount stays at main's 100-row behavior); proper fix — incremental byte-bound + filter/limit/columns selection, or by-reference signed-URL fetch — is a separate effort.

Model-facing docs

Catalog change documenting after / limit maxes / background behavior is in simstudioai/copilot#312.

🤖 Generated with Claude Code

[TheodoreSpeaks]

@greptile review

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jun 16, 2026 2:09am

[cursor]

PR Summary

High Risk
Large change set touching schema-backed job claiming, background delete/export, and filter-scoped mutations; incorrect masking or job lifecycle could cause data visibility issues or stuck table write slots.

Overview
Tables background work moves from per-table import* fields to a unified table_jobs model (jobStatus, jobId, jobType, … on list/detail APIs). Imports claim the same slot via markTableJobRunning; new POST …/delete-async runs paginated deletes from filter + excludeRowIds + created_at cutoff (optional Trigger.dev), and POST …/export-async plus GET …/export/download handle large exports with storage cleanup on cron prune. Job cancel is generalized to POST …/job/cancel; stale-job cron now fails stuck import/delete jobs and prunes old terminal jobs (deleting export files best-effort).

Bulk workflow APIs accept filter / excludeRowIds on column run and cancel-runs, with shared tableFilterError validation. Row GET adds keyset after cursor parsing. Column/row write routes share rootErrorMessage / rowWriteErrorResponse (including friendly row-limit toasts).

Grid UX treats select-all as “all rows matching the filter” with an exclusion set, kicks off async delete instead of loading every id, and routes Play/Refresh/Stop/context-menu actions through filter-scoped dispatch when appropriate. Exports above a threshold (or during a running delete) go async; SSE job events drive tray refresh and auto-download. Sidebars/filters adopt ChipInput / ChipDropdown and shared sidebar field components.

CI: dev db:push surfaces actionable errors when Drizzle hits non-TTY rename/drop prompts.

^{Reviewed by Cursor Bugbot for commit 0e47d6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

HTTP rows missing next cursor

Medium Severity

The GET rows handler now accepts an after keyset cursor and passes it to queryRows, but the JSON response never includes a nextCursor when a full page is returned. REST clients cannot advance past the first default-order page without guessing offsets or reusing Mothership-only tooling.

 

^{Reviewed by Cursor Bugbot for commit 7310c06. Configure here.}

[greptile-apps]

Greptile Summary

This PR migrates Mothership's user_table tool to match the performance and safety guarantees already present in the HTTP table routes: limit caps on query_rows and bulk ops, keyset (cursor) pagination for the default row order, and background-job dispatch for large CSV imports and large filtered deletes.

• Limit enforcement (C): query_rows now rejects requests above MAX_QUERY_LIMIT (1,000) and filter-based bulk ops above MAX_BULK_OPERATION_SIZE (1,000), mirroring the HTTP contract validators.
• Keyset paging (D): queryRows accepts an after: { orderKey, id } cursor and returns nextCursor when a default-order page fills; after+sort is rejected at both the contract and tool layer.
• Async job parity (E): CSV/TSV imports ≥ 8 MB and unbounded filtered deletes > 1,000 matches now dispatch trigger.dev tasks (or detached workers) with the same table_jobs ownership gate that prevents racing; small inline imports also claim/release the slot; deleteSourceFile: false preserves persistent workspace files.

Confidence Score: 5/5

Safe to merge. The ownership gate (partial-unique index + onConflictDoNothing) is atomic; claim/release is correctly handled in both the inline and async dispatch paths. The visibility mask for running delete jobs is logically equivalent to the worker's row-selection predicate, keeping mid-job reads consistent.

The three core changes — limit enforcement, keyset pagination, and async job dispatch — are all narrowly scoped and well-tested. The null-orderKey pagination edge case is the only correctness concern, and it only surfaces for tables that have rows predating migration 0226 in a workspace where fractional ordering is now enabled; any such rows continue to be reachable via offset paging. All DB writes involving the new table_jobs table are transactionally safe.

The keyset cursor logic in user-table.ts (query_rows case, lines 583-599) and the corresponding queryRows path in service.ts are worth a second look for any tables with partially-backfilled order_key values.

Important Files Changed

Filename	Overview
apps/sim/lib/copilot/tools/server/table/user-table.ts	Core Mothership tool handler: adds limit validation, keyset cursor pagination, and async dispatch for large imports/deletes; claim/release logic is correctly guarded via finally/internal error handling.
apps/sim/lib/table/service.ts	Adds pendingDeleteMask (visibility mask for running delete jobs), keyset seek in queryRows, job lifecycle helpers, and export/delete page helpers; contains the noted extra DB query per queryRows call for the mask.
apps/sim/lib/table/delete-runner.ts	New background delete worker: keyset-paginates by id, ownership-gates per page, commits batches in short transactions, and handles cancellation/supersede via JobSupersededError.
apps/sim/lib/table/import-runner.ts	Migrated to unified job-status APIs and adds deleteSourceFile: false guard to preserve persistent workspace files.
packages/db/migrations/0233_table_jobs_and_keyset.sql	Creates table_jobs table with partial-unique index for the one-active-job-per-table gate, migrates existing import state, adds tenant-scoped GIN index for JSONB filter queries, and adds keyset index on (table_id, id) for the delete worker.
apps/sim/lib/api/contracts/tables.ts	Splits tableRowsQuerySchema into base/refined pair, adds after cursor field with after+sort mutual exclusion, and adds new export-async, delete-async, and job-cancel contracts.
apps/sim/app/api/cron/cleanup-stale-executions/route.ts	Updated janitor now marks stale table_jobs (any type) as failed and prunes terminal jobs older than 24 h, with best-effort export-file deletion on prune.
apps/sim/lib/table/validation.ts	Wraps per-column unique-constraint checks in withSeqscanOff to prevent full-relation sequential scans; correctly handles external-transaction callers.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant M as Mothership Model
    participant UT as user-table tool
    participant SVC as table/service
    participant TJ as table_jobs (DB)
    participant W as Background Worker

    Note over M,W: Large CSV import via import_file
    M->>UT: import_file(fileId, tableId)
    UT->>SVC: markTableJobRunning(tableId, importId, 'import')
    SVC->>TJ: INSERT job row (partial-unique index gate)
    TJ-->>SVC: "claimed=true"
    SVC-->>UT: true
    UT->>W: dispatchImportJob(payload)
    UT-->>M: success + jobId
    M->>UT: "query_rows(tableId, after=cursor)"
    UT->>SVC: queryRows + pendingDeleteMask check
    SVC-->>UT: rows + nextCursor
    UT-->>M: rows + nextCursor
    W->>SVC: markJobReady(tableId, importId)

    Note over M,W: Unbounded filter-delete via delete_rows_by_filter
    M->>UT: delete_rows_by_filter(tableId, filter)
    UT->>SVC: "queryRows count — totalCount > 1000"
    UT->>SVC: markTableJobRunning(tableId, jobId, 'delete', payload)
    SVC->>TJ: INSERT job row with filter+cutoff payload
    UT->>W: dispatchDeleteJob(jobId, filter, cutoff)
    UT-->>M: success + jobId, rows hidden immediately
    W->>SVC: selectRowIdPage keyset by id
    W->>SVC: deletePageByIds per-batch transactions
    W->>SVC: markJobReady — mask lifts

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant M as Mothership Model
    participant UT as user-table tool
    participant SVC as table/service
    participant TJ as table_jobs (DB)
    participant W as Background Worker

    Note over M,W: Large CSV import via import_file
    M->>UT: import_file(fileId, tableId)
    UT->>SVC: markTableJobRunning(tableId, importId, 'import')
    SVC->>TJ: INSERT job row (partial-unique index gate)
    TJ-->>SVC: "claimed=true"
    SVC-->>UT: true
    UT->>W: dispatchImportJob(payload)
    UT-->>M: success + jobId
    M->>UT: "query_rows(tableId, after=cursor)"
    UT->>SVC: queryRows + pendingDeleteMask check
    SVC-->>UT: rows + nextCursor
    UT-->>M: rows + nextCursor
    W->>SVC: markJobReady(tableId, importId)

    Note over M,W: Unbounded filter-delete via delete_rows_by_filter
    M->>UT: delete_rows_by_filter(tableId, filter)
    UT->>SVC: "queryRows count — totalCount > 1000"
    UT->>SVC: markTableJobRunning(tableId, jobId, 'delete', payload)
    SVC->>TJ: INSERT job row with filter+cutoff payload
    UT->>W: dispatchDeleteJob(jobId, filter, cutoff)
    UT-->>M: success + jobId, rows hidden immediately
    W->>SVC: selectRowIdPage keyset by id
    W->>SVC: deletePageByIds per-batch transactions
    W->>SVC: markJobReady — mask lifts

Loading

_{Reviews (4): Last reviewed commit: "improvement(mothership): drop get_job op..." | Re-trigger Greptile}

[greptile-apps]

Full table CSVs built before budget enforcement

All input tables are serialized in parallel via Promise.all before any size check runs. For each table the buildTableCsvForMount loop fetches every row page-by-page and accumulates the entire CSV in memory — no size gate inside the loop. The outer budget check (totalSize + mount.content.length > MAX_TOTAL_SIZE) only fires after every table has been fully loaded.

Concrete failure: a user mounts two enterprise-plan tables each at ~40MB of rows. Both CSVs are built (80MB in memory) before the loop sees that the second one exceeds the 50MB cap and throws. File and directory mounts check record.size before fetching; tables lack that pre-flight guard. Under traffic, several concurrent requests doing this can exhaust the web container's heap.

[greptile-apps]

doomedCount can overestimate when rows are inserted between count and dispatch

matchCount comes from queryRows(..., { limit: 1, withExecutions: false }) and reflects the delete-mask-filtered row count. Between that count query and markTableJobRunning, rows inserted after the cutoff timestamp could match the filter, making matchCount stale. The cutoff passed to the background worker correctly caps new-row inclusion, but doomedCount stored in the job payload — and surfaced in the get_job response message — would over-report the number of rows being removed. Not a correctness issue for the worker, but the user-visible estimate is an upper bound, not an exact figure.

[greptile-apps]

Greptile Summary

This PR closes three performance gaps in Mothership's table operations: sandbox table mounts now drain all rows via keyset pagination instead of silently truncating to 100, query_rows gains a 1000-row limit cap and keyset cursor support, and large CSV/TSV imports plus unbounded filter-deletes are handed off to the existing background-job infrastructure (with a new get_job poll operation). The underlying data model also migrates: per-import state columns on user_table_definitions are replaced by a new table_jobs table with a partial-unique index enforcing one running write-job per table.

• function-execute mounts: buildTableCsvForMount pages through selectExportRowPage in 5k-row chunks, remaps column-id keys to display names, and counts toward the 50 MB budget — fixing empty-column mounts on id-keyed tables and the silent 100-row truncation.
• query_rows contract parity: limit now rejected above 1000, withExecutions: false, and keyset after/nextCursor accepted (rejected alongside sort); inline imports/deletes claim the one-write-job slot to block interleaving.
• Background import/delete in Mothership: CSV/TSV ≥ 8 MB dispatches tableImportTask; unbounded filter-deletes above 1000 rows dispatch tableDeleteTask with a pendingDeleteMask that hides doomed rows immediately from all read paths.

Confidence Score: 3/5

Safe to merge for most workloads; enterprise tables with hundreds of thousands of rows could exhaust Vercel function memory during a sandbox mount before the 50 MB budget check fires.

The core job-slot migration, keyset pagination, background import/delete dispatch, and schema change are all well-structured and tested. The one concern worth resolving before a wide enterprise rollout is buildTableCsvForMount: all table CSVs are fully assembled in parallel memory before the budget check, so a user who mounts a 500k-row enterprise table gets a function OOM rather than a clean rejection.

apps/sim/lib/copilot/tools/handlers/function-execute.ts — the buildTableCsvForMount loop and the parallel Promise.all mount assembly need a per-page budget check to avoid OOM on large enterprise tables.

Important Files Changed

Filename	Overview
apps/sim/lib/copilot/tools/handlers/function-execute.ts	Replaces the truncated 100-row queryRows mount with a keyset-draining buildTableCsvForMount that reads all rows and remaps id-keyed cells back to display names. Has a memory safety issue: the full CSV is materialized in memory before the 50 MB budget check runs, and pendingDeleteMask is called once per page (N+1 pattern).
apps/sim/lib/table/service.ts	Major overhaul: migrates import-status fields from user_table_definitions to table_jobs, adds selectExportRowPage (position-keyset export drain), pendingDeleteMask for delete-job visibility, selectRowIdPage/deletePageByIds for async delete workers, and latestJobForTable/latestJobsForTables for the new job fields. pendingDeleteMask now runs on every queryRows call regardless of table state.
apps/sim/lib/copilot/tools/server/table/user-table.ts	Adds query_rows limit clamping, keyset cursor (after/nextCursor), background delete dispatch for unbounded deletes, background import dispatch for large CSV/TSV files, get_job polling operation, and job-slot claim/release for inline imports. Logic is well-structured with proper guards and rollback paths.
apps/sim/lib/table/import-runner.ts	Renames import-specific service calls to generic job variants (markImportFailed→markJobFailed, etc.), adds deleteSourceFile flag (defaults true for UI single-use uploads; pass false for Mothership persistent workspace files), and updates SSE events to the new job event kind.
packages/db/migrations/0233_table_jobs_and_keyset.sql	Creates table_jobs with partial unique index (one running write-job per table), migrates existing import_status rows idempotently via ON CONFLICT DO NOTHING, tunes autovacuum on user_table_rows, and adds two CONCURRENT indexes (table_id+id keyset, tenant-scoped btree_gin) while dropping the old cross-tenant GIN index.
apps/sim/lib/table/delete-runner.ts	New async delete worker: walks rows in keyset pages (selectRowIdPage → deletePageByIds), checks job ownership on each page, applies cutoff/filter/exclusion scope, emits SSE events, and marks the job terminal or failed.
packages/db/schema.ts	Adds tableJobs table with unique partial index, replaces the old cross-tenant GIN index on user_table_rows with a tenant-scoped btree_gin index, adds table_id+id keyset index, and removes import_* columns from userTableDefinitions.
apps/sim/lib/table/dispatcher.ts	Extends DispatchScope with filter/excludeRowIds for select-all-under-filter and select-all-minus-deselections, wraps the dispatcherStep page query in withSeqscanOff for jsonb-filtered scopes, and adds scope-aware markActiveDispatchesCancelled with JSONB scope comparison.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Mothership: import_file / create_from_file] --> B{CSV/TSV 8 MB?}
    B -- No --> C[markTableJobRunning claim slot]
    C --> D[Inline import parseFileRows + batchInsert]
    D --> E[releaseJobClaim]
    B -- Yes --> F[createTable with jobStatus=running or markTableJobRunning]
    F --> G{isTriggerDevEnabled?}
    G -- Yes --> H[tasks.trigger tableImportTask]
    G -- No --> I[runDetached runTableImport]
    H -- dispatch error --> J[releaseJobClaim / deleteTable]
    I --> K[runTableImport streaming keyset import]
    H --> K

    L[Mothership: delete_rows_by_filter] --> M{limit === undefined?}
    M -- Yes --> N[queryRows count with pendingDeleteMask]
    N --> O{matchCount > 1000?}
    O -- No --> P[deleteRowsByFilter inline]
    O -- Yes --> Q[markTableJobRunning claim]
    Q --> R[dispatchDeleteJob]
    R --> S[runTableDelete keyset page walk]
    M -- No --> P

    T[Mothership: query_rows] --> U{limit > 1000?}
    U -- Yes --> V[return error]
    U -- No --> W[queryRows with pendingDeleteMask + after cursor]
    W --> X[return rows + nextCursor]

    Y[Mothership: get_job] --> Z[getTableById latestJobForTable]
    Z --> AA[return job status / rowCount]

Loading

_{Reviews (2): Last reviewed commit: "improvement(mothership): table speed par..." | Re-trigger Greptile}

[greptile-apps]

Full CSV materialized in memory before the 50 MB budget check

buildTableCsvForMount drains every row into a lines[] array and only returns after the full table is assembled. The budget check in the caller happens after all table CSVs are built via Promise.all. For an enterprise table with 100k+ rows, the CSV string can be hundreds of megabytes or more before the check ever fires — the budget guard prevents the file from reaching the sandbox, but the memory damage is already done (and in the parallel case, all tables are materialized simultaneously).

The old queryRows path implicitly bounded this to 100 rows. A straightforward fix is to track a running byte count inside the drain loop and throw early once the caller's budget would be exceeded — or pass maxBytes as a parameter and abort the page walk.

[greptile-apps]

pendingDeleteMask fetched on every page — N+1 DB round-trips during drain

selectExportRowPage calls pendingDeleteMask(table) unconditionally, so a 100k-row table drained in 20 pages makes 20 extra DB round-trips just to check whether a delete job is running. The mask is derived from table.id which is constant, and the delete job status won't change meaningfully mid-drain. Fetching it once before the loop and threading it through would eliminate the repeated queries.

[TheodoreSpeaks]

@greptile review

[cursor]

nextCursor requires truthy orderKey

Low Severity

query_rows only returns nextCursor when the last row has a truthy orderKey. A full page whose last row lacks order_key (e.g. not yet backfilled) omits the cursor even though more rows may exist.

 

^{Reviewed by Cursor Bugbot for commit c1c4160. Configure here.}

[TheodoreSpeaks]

@greptile review

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 4 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0e47d6e. Configure here.}

[cursor]

Shift-click shrinks select-all scope

Medium Severity

After a filter-wide select-all (rowSelection.kind === 'all'), shift-clicking row checkboxes only updates the loaded page: the handler materializes selection to in-memory row ids instead of keeping the exclusion-set model. Unloaded matching rows drop out of delete, run, and stop actions that rely on select-all semantics.

 

^{Reviewed by Cursor Bugbot for commit 0e47d6e. Configure here.}

[cursor]

Stop count ignores select-all scope

Low Severity

For a filter-wide select-all, the context menu stop action calls onStopAllRows with the filter, but runningInContextSelection still sums runningByRowId only over contextMenuRowIds (loaded rows). The Stop item can stay hidden or show the wrong count when running workflows sit on unloaded matching rows.

 

^{Reviewed by Cursor Bugbot for commit 0e47d6e. Configure here.}