FE-1130: Token frame format v2 — packed struct token layout#8944
Open
kube wants to merge 5 commits into
Open
Conversation
Replaces the uniform one-f64-slot-per-dimension token encoding with a
schema-driven packed struct per token, everywhere (interactive engine,
Monte Carlo, readers, playground):
- New engine/token-layout.ts is the single source of truth: real and
integer elements map to f64 (8 B), booleans to u8 (1 B); fields are
ordered by decreasing alignment and the stride is rounded up to 8 B
so f64 fields stay addressable through a shared Float64Array view.
- EngineFrame places are byte-addressed ({ byteOffset, count,
strideBytes }); FRAME_VERSION bumped to 2; whole-token moves are
byte-range copies.
- Euler integration only touches real fields via realFieldF64Offsets;
discrete bytes are copied through untouched.
- Monte Carlo frame buffers use byte capacity with the same x2 growth.
- getPlaceTokenValues / SimulationPlaceTokenValues removed from the
public API (raw f64 access is meaningless under mixed widths); use
getPlaceTokens instead.
- Playground drops its v1 mode and now reuses the core layout module.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Importing Monaco's TypeScript contribution registered the built-in TS service for every `typescript` model in the session, so after opening the playground story the product's LSP-backed editors gained bogus "Cannot find name 'TransitionKernel'" diagnostics on top of the SDCPN LSP's. The mode is now silenced immediately at module load (all feature flags off), enabled only while the playground editor is mounted, and disabled again on unmount with its markers cleared from all models. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
PR SummaryHigh Risk Overview
Transitions and Monte Carlo pack/unpack through the layout helpers ( Public API: Reviewed by Cursor Bugbot for commit 594a02e. Bugbot is set up for automated code reviews on this repo. Configure here. |
github-actions
Bot
added
area/infra
Contributor
There was a problem hiding this comment.
Pull request overview
This PR upgrades Petrinaut’s simulation frame token storage to frame format v2: a packed-struct, byte-addressed token region with a schema-driven per-colour layout (f64 for real/integer, u8 for boolean), and updates the engine, Monte Carlo pipeline, frame readers, and the token encoding playground to use that layout consistently.
Changes:
- Introduces a core
TokenSlotLayoutabstraction and helpers for computing layouts and reading/writing packed token bytes (computeTokenSlotLayout,readTokenRecord,encodeTokenToBytes, etc.). - Bumps engine frames to version 2 and converts all token-region handling from
Float64Arrayvalue slots toUint8Arraybyte regions with sharedu8/f64views. - Updates Monte Carlo buffering/transition effects/dynamics and the dev token encoding playground to reflect the packed layout; removes the now-meaningless raw f64 token-values API.
Reviewed changes
Copilot reviewed 44 out of 45 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/token-memory-view.tsx | Adapts the memory view UI to render TokenSlotLayout fields (byte offsets/sizes and field metadata) from core. |
| libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/token-encoding-playground.tsx | Removes v1/v2 toggle and scopes Monaco’s built-in TS language service to the playground lifecycle. |
| libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/playground-monaco.ts | Adds enable/disable controls for Monaco TS mode configuration and clears TS markers on unmount. |
| libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/physical-layout.ts | Replaces local layout/codec implementation with thin wrappers over core token-layout encode/decode helpers. |
| libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/physical-layout.test.ts | Updates playground layout and encode/decode tests to use core TokenSlotLayout. |
| libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/monaco-typescript.d.ts | Extends the local TS defaults typing to include setModeConfiguration. |
| libs/@hashintel/petrinaut/src/react/playback/provider.test.tsx | Updates playback mocks to match the removal of getPlaceTokenValues. |
| libs/@hashintel/petrinaut/src/main.ts | Removes the exported SimulationPlaceTokenValues type from the Petrinaut package surface. |
| libs/@hashintel/petrinaut-core/src/simulation/runtime/simulation.test.ts | Updates frame construction in tests to use an empty Uint8Array token buffer. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/types.ts | Renames Monte Carlo token capacity/count fields to byte-based equivalents. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/transition-effect.ts | Switches transition effects to read/write packed token bytes via token-layout helpers. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/run-state.ts | Converts frame resizing and run summaries to use token byte counts/capacities. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/monte-carlo-simulator.test.ts | Updates Monte Carlo tests to assert byte-sized token regions and to use typed token access. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/internal-types.ts | Changes transition additions to store Uint8Array token blocks rather than number arrays. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-reader.ts | Updates Monte Carlo frame reader to return typed token records from packed bytes. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-operations.ts | Converts dynamics/removal/addition operations to work on packed byte regions. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-buffer.ts | Updates Monte Carlo frame buffer layout to include u8/f64 views over a byte-addressed token region. |
| libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/advance-run.ts | Updates pending token-additions map type to Uint8Array[]. |
| libs/@hashintel/petrinaut-core/src/simulation/index.ts | Removes SimulationPlaceTokenValues export from the simulation entrypoint. |
| libs/@hashintel/petrinaut-core/src/simulation/frames/internal-frame.ts | Bumps FRAME_VERSION to 2 and migrates internal frame layout/state to byte offsets + packed token bytes. |
| libs/@hashintel/petrinaut-core/src/simulation/frames/frame-reader.ts | Updates the main frame reader to decode typed tokens from packed bytes via readTokenRecord. |
| libs/@hashintel/petrinaut-core/src/simulation/frames/frame-reader.test.ts | Rewrites frame-reader tests to validate copied token records rather than copied f64 buffers. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/types.ts | Updates engine types for byte-addressed dynamics inputs and includes per-place tokenLayout. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.ts | Adds the core packed token layout module (layout computation + encode/decode + shared views). |
| libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.test.ts | Adds unit tests covering layout math, encode/decode round-trips, and alignment checks. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.test-helpers.ts | Adds helpers for constructing/decoding v2 frames and token blocks in tests. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.ts | Migrates token removal/compaction to operate on packed byte ranges. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.test.ts | Rewrites token-removal tests around packed bytes and typed token decoding helpers. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/README.md | Updates engine documentation to describe frame format v2 token storage and access patterns. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/execute-transitions.ts | Updates transition execution to append packed token byte blocks and adjust byte offsets. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/execute-transitions.test.ts | Updates transition execution tests to validate packed-token behavior via decode helpers. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.ts | Updates possible-transition computation to decode/encode packed token byte blocks and preserve determinism. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.test.ts | Updates possible-transition tests to validate packed byte outputs by decoding token blocks. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/compute-place-next-state.ts | Updates dynamics stepping to integrate real fields only while copying discrete bytes through unchanged. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/compute-place-next-state.test.ts | Updates dynamics tests and adds coverage for “discrete fields untouched” behavior. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/compute-next-frame.ts | Updates frame evolution to apply dynamics on packed bytes and write results back by byte offsets. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/compute-next-frame.test.ts | Updates next-frame tests to assert on decoded tokens rather than raw numeric buffers. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/check-transition-enablement.test.ts | Updates enablement tests for byte-offset/stride-based place state and v2 frame buffers. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/build-simulation.ts | Updates initial marking packing and dynamics compilation to use packed token layouts and byte buffers. |
| libs/@hashintel/petrinaut-core/src/simulation/engine/build-simulation.test.ts | Updates build-simulation tests to validate packed layout properties and decoded token records. |
| libs/@hashintel/petrinaut-core/src/simulation/ARCHITECTURE.md | Updates architecture docs to describe frame format v2 and packed-struct token layout. |
| libs/@hashintel/petrinaut-core/src/simulation/api.ts | Removes SimulationPlaceTokenValues and getPlaceTokenValues from the public frame reader API. |
| libs/@hashintel/petrinaut-core/src/index.ts | Exports the new token-layout module API from the package root and removes the old token-values export. |
| libs/@hashintel/petrinaut-core/src/actual-mode/timeline.ts | Removes getPlaceTokenValues implementation from actual-mode timeline frame reader. |
| .changeset/h-6519-discrete-token-attribute-types.md | Bumps changeset impact to minor for both packages and notes the v2 frame format change. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (placeState.strideBytes === 0) { | ||
| if (typeof indices !== "number") { | ||
| throw new Error( | ||
| `For place ${placeId} with zero dimensions, expected number of tokens to remove, got Set.`, |
| (place) => place.strideBytes === 0 && place.arcType === "standard", | ||
| ); | ||
|
|
||
| // TODO: This should acumulate lambda over time, but for now we just consider that lambda is constant per combination. |
Comment on lines
102
to
104
| const standardInputPlacesWithZeroDimensions = inputPlaces.filter( | ||
| (place) => place.dimensions === 0 && place.arcType === "standard", | ||
| (place) => place.strideBytes === 0 && place.arcType === "standard", | ||
| ); |
4 tasks
This was referenced Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🌟 What is the purpose of this PR?
Replaces the simulation engine's uniform one-Float64-slot-per-dimension token encoding with a schema-driven packed struct per token (format v2), everywhere: interactive engine, Monte Carlo, frame readers, and the encoding playground. Booleans now occupy a single byte instead of eight, places are byte-addressed, and the per-colour layout module gives future wider types (128-bit UUIDs, FE-1121) a place to slot in as
u64×2fields.🔗 Related links
🚫 Blocked by
🔍 What does this change?
engine/token-layout.ts— single source of truth for the physical layout:real/integer→ f64 (8 B),boolean→ u8 (1 B); fields ordered by decreasing alignment (stable), stride rounded up to 8 B so every f64 field stays addressable through a sharedFloat64Arrayview (noDataViewin hot paths). Read/write goes throughreadTokenRecord/writeTokenValue/encodeTokenToBytesonly.EngineFramebumped toFRAME_VERSION = 2: per-place state is byte-addressed ({ byteOffset, count, strideBytes }), the header records the token region's byte length, snapshots carryUint8Arrays, and all whole-token moves (kernel-produced insertions, removal/compaction) are byte-range copies.realFieldF64Offsets; discrete bytes are copied through untouched (structurally stronger than zeroing derivatives).tokenByteCount/tokenByteCapacity, same ×2 growth policy, dual u8/f64 views).getPlaceTokenValues/SimulationPlaceTokenValuesremoved — raw f64 access is meaningless under mixed field widths; consumers usegetPlaceTokens(typed records). The monorepo was swept: no usage outside the petrinaut packages.simulation/ARCHITECTURE.mdandengine/README.mdtoken-storage sections rewritten for the packed layout.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
minorfor both packages — public API removal)📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
🐾 Next steps
u64×2field (align 8) withbigintat the JS boundary — the layout machinery here is built for it.🛡 What tests cover this?
engine/token-layout.test.ts(layout computation, encode/decode round-trips incl. u8 booleans and integer rounding) + a discrete-fields-untouched dynamics integration test@hashintel/petrinaut-core, 137 in@hashintel/petrinautlint:tsc(tsgo) andlint:eslint(oxlint) clean for both packages❓ How to test this?
yarn devinlibs/@hashintel/petrinaut.sizeof(token)reflects the packed stride.📹 Demo
See the token-encoding playground story: the memory view renders the exact v2 layout this PR ships (u8 booleans, alignment padding, byte-addressed slots).