Skip to content

PDX-514: fix(mcp): base estimatedTokens on content text only#225

Merged
mrdailey99 merged 1 commit into
developfrom
fix/PDX-514-meta-token-estimate
Jun 9, 2026
Merged

PDX-514: fix(mcp): base estimatedTokens on content text only#225
mrdailey99 merged 1 commit into
developfrom
fix/PDX-514-meta-token-estimate

Conversation

@mrdailey99

Copy link
Copy Markdown
Collaborator

Summary

  • Fix _meta.estimatedTokens double-counting. The estimator serialized the entire ToolResult, so the payload was counted twice — once as escaped JSON in content[0].text and again as the structuredContent object — inflating per-call estimates ~2x (e.g. a testrun reported 169,866 vs ~77,873 actual content-text tokens).
  • New estimateResultTokens(result) sums content[].text only (the text an LLM client actually consumes) and is now used by both attachMeta and the depth-guard session accumulator. The generic estimateTokens is retained/exported.
  • Bump version 1.6.1 → 1.6.2 (package.json + server.json kept in sync).

Jira

https://provartesting.atlassian.net/browse/PDX-514

Test plan

  • yarn compile passes
  • unit tests pass — 1491 passing (added an estimateResultTokens describe block + a regression test proving a tiny content with a huge structuredContent yields a tiny estimate)
  • yarn lint passes (lint:script-names + lint)
  • mcp-smoke.cjs: 30 PASS; only the two heaviest tools (provar_automation_testrun, provar_automation_config_load) time out, and they do so even at a 90s per-request timeout with no competing load. Smoke spawns the installed global package (sf provar mcp start, currently 1.6.1), not this worktree build, so this change cannot affect those results — the timeouts are pre-existing/environmental (JVM/automation boot), not a regression from this PR.

Changes

  • src/mcp/utils/tokenMeta.ts: add estimateResultTokens; switch attachMeta and the depth-guard accumulator to it.
  • docs/mcp.md: update the estimatedTokens formula to ceil(len(content[].text) / 4) and expand the implementation note explaining why structuredContent is excluded.
  • test/unit/mcp/tokenMeta.test.ts: rebase the estimate test on content text; add a structuredContent-not-counted regression test; add an estimateResultTokens describe block.
  • package.json, server.json: version 1.6.1 → 1.6.2.

Notes

  • Out of scope (confirmed): the harness diverting oversized results to a file is intentional (avoids context bloat + ENOBUFS) and is untouched here.
  • Public-docs heads-up: this changes the value semantics of the observability field _meta.estimatedTokens (now content-text-based, not whole-result). If docs/provar-mcp-public-docs.md / the University course reference the estimate basis, they should be updated separately by the Provar team.

RCA: estimateTokens serialized the whole ToolResult, counting the payload twice (escaped JSON in content[0].text plus the structuredContent object) and inflating the per-call estimate roughly 2x versus what the model actually reads.
Fix: add estimateResultTokens summing content[].text and use it in attachMeta and the depth-guard accumulator; update docs/mcp.md formula and note plus tokenMeta tests; bump version to 1.6.2 in package.json and server.json.
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown

Quality Orchestrator

🟢 LOW · 12 / 100 · Touches: utils. All changed files have mapped tests.


🧪 Tests to Run · Running 1 of 57 tests

  • unit/mcp/tokenMeta.test.ts
▶ Run command
npx vitest run \
  unit/mcp/tokenMeta.test.ts

⚡ quality-orchestrator  ·  /qo stub <file>  ·  qo analyze-local

@mrdailey99 mrdailey99 merged commit e9508c0 into develop Jun 9, 2026
4 checks passed
@mrdailey99 mrdailey99 deleted the fix/PDX-514-meta-token-estimate branch June 9, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant