v0.7.10: models sorting, compress/decompress file block tools, new enrichment providers, deepseek models, db performance#5106
Open
waleedlatif1 wants to merge 11 commits into
Open
v0.7.10: models sorting, compress/decompress file block tools, new enrichment providers, deepseek models, db performance#5106waleedlatif1 wants to merge 11 commits into
waleedlatif1 wants to merge 11 commits into
Conversation
…n each provider (#5099) * improvement(models): sort model dropdown by latest release date within each provider * fix(models): preserve input provider order and build catalog index once
…5100) * feat(file): add Compress operation to bundle files into a .zip archive * feat(file): add Decompress operation to extract .zip archives Adds the inbound half of the archive pair: extracts a .zip back into the workspace with zip-slip path sanitization, symlink skipping, and entry/ size caps to bound zip-bomb expansion. Extracted files are returned in the files output, ready to chain downstream. * fix(file): align archive ops with v5 output surface and zip mime - Drop the single 'file' output reintroduced for compress/decompress; v5 intentionally exposes only 'files' (plus id/name/size/url scalars), so compress/decompress reuse the existing surface with no new block output - Add zip/gz to EXTENSION_TO_MIME (previously only in the reverse map), so archive extensions resolve to a real mime instead of octet-stream - Update File v5 block test for the two new operations * fix(file): harden compress naming per review - Flatten zip entry names to a safe basename so untrusted fileInput names with .. or / cannot produce zip-slip entry paths (cursor) - Treat archiveName as a flat name landing at the workspace root instead of passing it through splitWorkspaceFilePath, which silently created folders for names with separators (greptile) - Add the upfront empty-input guard before any DB calls, matching the read and content operations (greptile) * fix(file): make decompress extraction atomic and bound per-entry size - Read and validate every entry before writing any file, so hitting a size cap no longer leaves partially-extracted files in the workspace (cursor) - Enforce the per-entry cap on the materialized buffer in addition to the declared size, covering entries that omit an uncompressed size (cursor) - Pre-check declared sizes up front to reject standard zip bombs before materializing, and return 422 when no files could be extracted (cursor) * fix(file): exclude skipped entries from caps and reject multi-archive decompress - Resolve safe (sanitized) zip entries up front so unsafe/skipped entries no longer count toward the per-entry and total uncompressed-size caps (cursor) - Reject decompress input that resolves to more than one archive with a clear error instead of silently extracting only the first (cursor) * fix(file): enforce single-archive decompress at the API boundary The block already rejects multiple archives, but the manage route is the real boundary (callable directly and by the LLM tool) and still took the first of multiple resolved inputs. Add the empty-input and >1-archive guards in the route so extra archives are rejected with a clear error rather than silently ignored (cursor). * docs(file): correct compress description and stale file-output references - Drop the misleading 'under provider upload limits' claim from the compress tool description (models cannot read zip archives) - Fix bestPractices to reference the 'files' output, not a non-existent 'file' - Remove the stale 'file' property from the compress test fixture so it matches the real API response (greptile)
…emoize Anthropic client (#5098) * perf(execution): parallelize preflight gates, cache deployed state, memoize Anthropic client - Memoize Anthropic + Azure-Anthropic SDK clients (new client-cache.ts) keyed by apiKey (+beta header; +baseURL/version/pinnedIP for Azure) so HTTP keep-alive connections are reused instead of a fresh TLS handshake per call. apiKey is the tenant boundary. - Parallelize the read-only preflight gates in preprocessing.ts (ban + subscription, then usage + org-member + rate-limit) while preserving exact error precedence (ban 403 -> usage 402 -> rate 429) and keeping the sole write (admission reservation) last. - Parallelize the independent workflow-state and env-var loads in execution-core. - Cache deployed workflow state by immutable deploymentVersionId with deep-clone-on-read, oldest-first eviction, and a 5-min TTL bounding the credential-mapping edge across ECS tasks. - Parallelize the independent personal-subscription + membership queries in getHighestPrioritySubscription. - BYOK: drop the redundant getWorkspaceById existence check (auth already validates the workspace); read the key list fresh every call for zero cross-instance staleness. Billing/usage/ban/permission reads stay fresh on the primary (no cache, no replica). Adds tests for every new mechanism and fixes a pre-existing vitest class-mock incompatibility that had execution-core.test.ts fully red on staging. * fix(execution): run rate-limit gate only after ban/usage pass The rate-limit gate is not read-only — checkRateLimitWithSubscription consumes a token — so running it in parallel with the read-only gates debited rate-limit quota for requests that the ban (403) or usage (402) gates reject, which the original sequential flow never did. Move the rate-limit gate to run sequentially after the ban and usage gates pass, preserving the read-only gates' parallelism (ban + subscription + usage) and the exact ban -> usage -> rate precedence. Add regression tests asserting the rate limiter is not consumed when an earlier gate rejects, and is consumed once when they pass. Caught by Cursor Bugbot review. * chore(execution): trim redundant preflight comments Tighten the gate overview to match the sequential rate-limit gate and drop inline notes that duplicated it or the runRateLimitGate doc. * refactor(cache): address review — idle TTL for client cache, LRUCache for deployed state - client-cache: add updateAgeOnGet so the TTL is genuinely idle-based (active clients keep their warm keep-alive connections; the JSDoc now matches behavior). - deployed-state: replace the hand-rolled Map + manual FIFO eviction/TTL with LRUCache (real LRU eviction, built-in TTL), matching the effectiveDecryptedEnv and integration-tool-schema caches. TTL stays absolute (not reset on read) so the credential-migration remap still propagates across ECS tasks. Both per review feedback from Greptile. * test(execution): isolate rate-limit gate test from STEP 7 reservation The 'consumes the rate-limit gate once' test reached the STEP 7 admission reservation, which depends on Redis — it passed locally (reserve throws and is swallowed) but failed in CI (reserve returns not-reserved -> 429). Pass skipConcurrencyReservation so the test isolates the rate gate deterministically. * perf(providers): memoize SDK clients where the pool is per-client (bedrock, vllm) Generalize the Anthropic client cache into one shared memoizer (providers/client-cache.ts) and apply it only where each new client owns its own connection pool — so reuse actually keeps connections warm: - bedrock: AWS SDK clients hold a per-client connection pool (reuse is the AWS best practice). Keyed by region + credential identity. - vllm: a pinned endpoint creates its own undici Agent per call; key by the resolved IP so DNS re-validation still runs each request. - anthropic + azure-anthropic: migrated onto the shared memoizer. Deliberately NOT applied to the OpenAI-compatible providers, groq, cerebras, or google: their SDKs share a process-global keep-alive pool (Node openai-sdk module singleton agent; anthropic/global undici), so a fresh client per request already reuses connections and memoization would add complexity with ~no benefit. litellm uses a plain shared-agent client (no pinning) and is likewise skipped. Bounded LRU (max 1000, 30m idle TTL) with no close-on-eviction, avoiding the unbounded-growth and eviction-closes-in-use-client failure modes seen in similar client caches. * chore(perf): trim verbose comments to terse why-notes * chore(perf): drop obvious inline comments, keep nuance as TSDoc * fix(bedrock): key client cache on full credential, not just access key id A corrected secret under the same access key id would otherwise keep serving the stale cached client until TTL/eviction. Caught by Cursor Bugbot. * test(execution,providers): fix preflight mock reset + isolate provider client cache in tests - preprocessing.test: re-establish the checkOrgMemberUsageLimit mock in beforeEach (the only gate mock not re-set). In the full suite its implementation was reset so the success-path test got undefined -> threw -> 500 -> success:false. Mirrors how checkServerSideUsageLimits is handled. - client-cache: add clearProviderClientCacheForTests; call it in the bedrock and vllm test beforeEach so construction assertions always start from a cache miss now that those providers memoize their client. * test(execution): make RateLimiter mock constructable under vitest 4.x The RateLimiter mock used an arrow factory (vi.fn(() => ({...}))). vitest 4.x (CI) rejects `new` on an arrow-implemented mock ("not a constructor"); 3.2.4 allowed it. The new rate-gate test is the first to actually `new RateLimiter()`, so it surfaced the failure only in CI. Switch the mock to a regular function and drop the speculative beforeEach re-establishments that didn't address it.
…crease connector limits + better error propagation (#5089) * fix(execution,connectors): offload large function inputs; harden KB connector size limits Addresses a class of 10 MB limit failures: - executor/variables: offload over-budget function block-output context values to durable large-value refs (lazy `sim.values.read`) so JS function blocks can merge medium files without exceeding the 10 MB inter-block request-body cap. - connectors: stream downloads via `readBodyWithLimit` (memory-safe), and surface oversized files as visible `failed` KB documents instead of silently dropping them — listing-time for github/s3/dropbox/onedrive/sharepoint, fetch-time for gitlab/azure/google-drive via a shared `ConnectorFileTooLargeError`. Raise the per-file cap from a hardcoded 10 MB to the canonical 100 MB KB document limit (`CONNECTOR_MAX_FILE_BYTES`), except Google Drive's export path (Google's hard 10 MB export-API limit). - sync-engine: `classifyExternalDoc` + bulk `skipDocuments` (failed rows with a reason, excluded from retry), byte-bounded batch concurrency to cap peak worker memory at the raised cap, and a `metadata.fileSize ?? size` fallback. * fix zoom * update skill * address comments + fix terminal event in sse stream * fix accounting issue
#5087) * feat(integrations): hosted email-enrichment providers + cascade wiring Add Datagma, Dropcontact, LeadMagic, Icypeas, and Enrow integrations — tools, blocks, brand icons, and BYOK + metered hosted-key support — and register each in the tool/block registries and BYOK provider list. Wire the new finders/verifiers into the enrichment cascades: - work-email: Datagma, LeadMagic, Dropcontact, Icypeas, Enrow - phone-number: LeadMagic, Datagma, Dropcontact - email-verification: Icypeas, Enrow - company-info: Datagma, LeadMagic - company-domain: Datagma Add hosting tests for all five providers and cascade tests covering the new providers (incl. new test files for email-verification, company-info, and company-domain). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(enrichment): address PR review on Icypeas success + Datagma billing - Icypeas find_email/verify_email postProcess return success:true for all terminal statuses (NOT_FOUND/DEBITED_NOT_FOUND included) so the cascade runner calls mapOutput and records invalid/not-found verdicts instead of throwing and inflating the error count - Bill Icypeas verify FOUND (not just DEBITED*) per the documented 0.1-credit charge - Datagma enrich_person only applies the 30-credit phone surcharge when a phone lookup (phoneFull) was requested - Note Datagma's URL-param (apiId) auth in the hosted-key doc comment - Update hosting tests to match Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(enrichment): only bill Enrow verify on a completed verification getCost returned a flat 0.25 credits regardless of output, so a job that fell back to the initial submit response (poll never completed, no qualification) was still metered. Charge 0.25 only when a qualification is present; 0 otherwise. Add a no-qualification test case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(enrichment): peg hosted credit cost to each provider's lowest paid plan Align *_CREDIT_USD to the entry tier Sim will provision: - Datagma: Regular $49/3,000 emails → $0.0163 (was Popular $0.0132) - LeadMagic: Basic $49/2,000 → $0.0245 (was Growth $0.0104) Icypeas (Basic $0.019), Enrow (Starter $0.012), and Dropcontact (Starter ~$0.17) already reflect their lowest plan. Tests derive from the constants, so values stay consistent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(enrichment): address PR review on mononyms + Icypeas verify email map - work-email LeadMagic: pass full_name + domain so single-token (mononym) names are no longer skipped - work-email Icypeas: firstname/lastname are optional on the API, so run a mononym with firstname alone instead of self-skipping - icypeas_verify_email mapItem reads item.email (verify payload shape) with a fallback to the nested results.emails[0].email Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(enrichment): case-insensitive Enrow find billing getCost compared qualification to exactly 'valid' while the cascade normalizes with toLowerCase(), so a differently-cased API qualifier could zero out billing on a valid email. Lowercase before comparing; add a test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(enrichment): drop Dropcontact from the phone-number cascade Dropcontact is an email/company-data enrichment service, not a phone-discovery provider — its phone/mobile_phone fields are unreliable and were surfacing firmographic data (an employee-count range like "5000-20077") as the phone. Keep the two purpose-built phone finders (LeadMagic find_mobile, Datagma find_phone); Dropcontact stays in the work-email and company cascades where its data is reliable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(enrichment): only accept valid-qualified Enrow emails in work-email Enrow's finder qualifies each email valid/invalid. The work-email mapOutput accepted any non-empty email, so an invalid-qualified address could fill the cell while hosted billing (which only charges on valid) charged zero. Gate the cell on qualification === 'valid', consistent with billing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ral context window (#5103)
* fix(input-format): field not editable race condition * remove dead code * simplify
…ot-path write cleanups (#5105) * perf(db): logs-list index, drop redundant indexes, replica routing, hot-path write cleanups * fix(logs): keep /api/v1/logs on primary db — its permissions join is the auth gate, not replica-safe
…eletons (#5104) * fix(sidebar): prefetch chats + workflows so cold loads don't flash skeletons On a cold load (e.g. when the browser discards an idle tab and reloads), the persistent sidebar started with an empty React Query cache and client-fetched its chat + workflow lists, flashing loading skeletons. Prefetch both lists server-side in the workspace layout and hydrate them via HydrationBoundary, under the same query keys and mappers the client hooks use, so the sidebar paints populated on the first render. The prefetch runs concurrently with the existing org-settings fetch and never throws, so it adds no blocking work in the common case and falls back to client fetching on error. * refactor(prefetch): call data layer directly instead of internal HTTP self-fetch The sidebar and settings prefetches fetched their data by making internal HTTP requests to our own API routes. Replace those self-fetches with direct calls to shared server-side data functions, so each route handler and its prefetch read from one source with no extra network hop, serialization, or re-auth. - Extract listWorkflowsForUser (lib/workflows/queries) and listMothershipChats (lib/copilot/chat) from their routes; both routes and the sidebar prefetch now call them. - Extract getUserSettings/getUserProfile (lib/users/queries) shared by the settings/profile routes and their prefetches. - Subscription prefetch calls the existing getSimplifiedBillingSummary + getEffectiveBillingStatus directly. - Sidebar prefetch checks workspace access once via checkWorkspaceAccess and skips silently when denied. * refactor(prefetch): share mothership chat list staleTime constant Export MOTHERSHIP_CHAT_LIST_STALE_TIME from the chats hook and use it in both useMothershipChats and the sidebar prefetch, mirroring WORKFLOW_LIST_STALE_TIME so the prefetch and client hook can't drift. * fix(prefetch): keep subscription prefetch on the wire shape via internal billing API The billing summary returns Date fields (and an untyped metadata blob) that the JSON API serializes to strings. Calling the data layer directly would cache Date objects (App Router preserves them through RSC serialization), mismatching the string wire shape the client useSubscriptionData hook caches. Route the subscription prefetch through the internal billing API so server-hydrated and client-fetched data share the exact same shape. The date-free general-settings and profile prefetches keep calling the data layer directly.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
PR SummaryMedium Risk Overview The File block and KB connectors share a raised per-file cap ( Five new enrichment integrations (Datagma, Dropcontact, Enrow, Icypeas, LeadMagic) add blocks, tools, BYOK entries, and waterfall providers on company-domain / company-info enrichments. Workspace cold load: server prefetch of sidebar workflows and mothership chats into React Query (same keys/mappers as client hooks), with routes delegating to shared Smaller changes: mark read only updates Reviewed by Cursor Bugbot for commit 8353145. Configure here. |
…-UI create gaps (#5107) * fix(locks): enforce workflow/folder locks on the agent + close manual-UI create gaps The copilot/agent workflow & folder mutation tools and the edit_workflow tool bypassed lock enforcement, so the agent could edit a locked workflow and move/create workflows into a locked folder. Add assertWorkflowMutable/ assertFolderMutable guards (from @sim/workflow-authz) to every agent mutation path, mirroring the REST API. Also close two parity gaps on the manual-UI REST side: creating a workflow into a locked folder and creating a subfolder under a locked parent were previously unguarded. The realtime collaborative canvas already enforced workflow-level locks server-side. * fix(locks): normalize optional folderId to null for assertFolderMutable * refactor(locks): hoist constant folder-lock check out of move loop; scope test mocks with Once * refactor(locks): drop redundant ensureWorkflowAccess fetch in rename
…rate integration docs (#5109) * improvement(integrations): validate BigQuery/Forms/PageSpeed + regenerate integration docs - BigQuery: mark null-defaulted outputs optional (get_table type/numRows/numBytes/creationTime/lastModifiedTime/location, list_datasets location, list_tables type, query totalBytesProcessed) - Google Forms: add response pagination (pageToken + filter params, nextPageToken output), fix pageSize visibility, advanced-mode pagination subBlocks + filter wandConfig - PageSpeed: add a 7th BlockMeta template (competitor benchmark) - Regenerate integration docs; add manual intro sections to new datagma/dropcontact/enrow/icypeas/leadmagic pages * fix(docs-gen): preserve apostrophes in tool descriptions when generating docs The doc generator extracted tool descriptions with a character class that excluded both quote types (['"]([^'"]...)['"]), so a double-quoted description containing an apostrophe (e.g. "Find someone's email") was truncated at the apostrophe — the generated docs/catalog showed stubs like "Find someone". Anchor extraction on the actual opening quote (single/double/backtick), matching the existing extractDescription helper, in both buildToolDescriptionMap and extractToolInfo. Regenerated docs restore full descriptions across all affected integrations (Apollo, Ahrefs, LeadMagic, Findymail, OpenAI, Slack, etc.). * fix(docs-gen): resolve tools defined in a sibling file + scope params per tool The doc generator located a tool's definition only by filename convention (decompress.ts / index.ts), so file_decompress — which lives in compress.ts alongside file_compress — fell back to index.ts and rendered an empty Input table. It also read the params block from the first tool in a multi-tool file, so every tool in such a file inherited the first tool's inputs/outputs. - getToolInfo: when no candidate file declares the exact tool ID, scan the whole tool-prefix directory for the file that does. - extractToolInfo: read the params block scoped to the specific tool, falling back to the full file for tools that inherit params via spread. Regenerated docs eliminate ~50 empty/incorrect input tables across integrations (clickhouse, rb2b, reddit, file, etc.); param-less OAuth-only tools correctly keep an empty input table.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.