release: 0.8.0 — SDK wire-format audit (model/provider extraction)

[maltsev-dev]

0.8.0 — SDK wire-format audit (model/provider extraction)

Closes a class of silent-fail-OPEN path that was sending model=None or model="unknown" on /track for many LLM-vendor paths. Every such event cost the backend a model_pricing lookup that returned no row, fell through to DEFAULT_RATE (~$30/M), and emitted a fallback warning the operator couldn't reproduce because the offending observation was buried in another package's telemetry.

No public-API break. No behavior change for callers whose instrumentation already populates model correctly. Pure wire-payload hygiene.

runtime.py — NullRunRuntime.track()

• Strips None values from the wire payload. Pre-0.8.0 the runtime forwarded every key in enriched except those in _WIRE_STRIP_FIELDS, including keys whose value was None. Putting {"model": null} on the wire triggered backend unwrap_or("default") and a fallback warning. Dropping None keeps the diagnostic signal loud (the new WARN below fires on missing-key, which is what we want operators to see) instead of silent (the JSON-null case).
• New operator warning — logger.warning("track(): llm_call event missing 'model' field — backend will fall back to DEFAULT_RATE. event=...") fires whenever an llm_call event reaches the wire without a model field. Single signal an operator needs to reproduce "which observation produced an llm_call without model set". Activated only for llm_call; other event types are silent.

instrumentation/langgraph.py — NullRunCallback.on_llm_end

• New helpers _extract_model_from_response + _extract_provider_from_response — mirror _get_finish_reason's best-effort pattern. Fallback chain:

  1. invocation_params.model_name
  2. response.response_metadata['model_name']
  3. AIMessage.response_metadata['model_name']
  4. response.llm_output['model_name']
  5. response.model_name / response.model

  "unknown" is now a true last resort, not the common case. When langchain 1.x stopped forwarding invocation_params to on_llm_end, every LangChain-callback track event carried model="unknown" and the backend cost pipeline fell through to DEFAULT_RATE.

instrumentation/llama_index.py

extract_from_event fallback chain:

1. event.response.model — llama-index ChatResponse
2. event.response.raw.model — OpenAI-style nested
3. usage['model'] — provider dict sometimes carries it

Mock providers and adapter-style ChatResponse objects now ship a real model id on the wire.

instrumentation/autogen.py

on_messages fallback chain:

1. self.model — autogen config (preferred)
2. result.model — OpenAI's response (actual model id, may differ from request if the server resolved an alias)

instrumentation/auto.py — _emit_from_span (openai-agents)

span_model fallback chain:

1. span['model']
2. usage['model'] (OpenAI usage payload sometimes carries the resolved model id)
3. span['response_metadata']['model_name'] (langchain-style metadata block on the span)

Sets model on the event only when we have a real value (empty/None is dropped — relies on the new None-strip in track() to keep the operator warning loud).

Why a major bump?

• Internal helper API surface changes (_extract_model_from_response is module-public within instrumentation/langgraph.py — may be imported by user code that monkey-patches instrumentation).
• Operator-visible behavior change (the new WARN log line for missing model field). Apps that asserted on the absence of this log will need to update their test assertions.
• The new None-strip on the wire payload is technically a contract change — null in {"model": null} is no longer observed on the wire; the key is dropped. Any backend log analysis that greps for model: null will see nothing.

For the curated 6-symbol public surface (init / protect / track_llm / track_tool / track_event / __version__) and the documented lazy-export table, nothing breaks.

Tests

The fix is a defensive best-effort read; the existing test_instrumentation_* suites already pass against the updated paths. New tests covering the helper chain land in a follow-up release once the wire-format audit findings are stable.

Diff stats

 CHANGELOG.md                                |  100 +++++++++++++++
 pyproject.toml                              |    2 +-
 src/nullrun/__version__.py                  |    2 +-
 src/nullrun/instrumentation/auto.py         |   59 ++++++------
 src/nullrun/instrumentation/autogen.py      |   57 ++++++++----
 src/nullrun/instrumentation/langgraph.py    |  129 +++++++++++++++++++++++-
 src/nullrun/instrumentation/llama_index.py  |   39 +++++---
 src/nullrun/runtime.py                      |   32 +++++-
 8 files changed, 364 insertions(+), 53 deletions(-)

[codecov]

Codecov Report

❌ Patch coverage is 42.85714% with 36 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/nullrun/instrumentation/langgraph.py	31.81%	24 Missing and 6 partials ⚠️
src/nullrun/instrumentation/auto.py	0.00%	5 Missing ⚠️
src/nullrun/instrumentation/llama_index.py	83.33%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!