Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Open
tejaskash wants to merge 1 commit into
feat/dev-otel-storagefrom
feat/dev-otel-collector
Open

feat(dev): collect local OTEL traces in project dev#1980
tejaskash wants to merge 1 commit into
feat/dev-otel-storagefrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskash tejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review this after #2043

This PR now contains exactly two things: the commits of #2039 (the OTLP trace storage layer it builds on) plus one commit of its own (~1,004 lines) — the part to review here. GitHub's stack tooling wouldn't let the base retarget, so until #2039 merges the diff shows both; after #2039 merges this collapses to just the collector work.

What this does (the ~1,004-line commit)

When you run agentcore project dev, your agent gets local tracing out of the box: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via environment variables, and traces persist through #2039's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf (pinned @opentelemetry/otlp-transformer@0.213.0 — the last version shipping request decoders; don't let dependabot bump it) or JSON.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf+JSON fixtures; handler with fake runners/collector; runner probe fakes) plus the end-to-end proof: built bundle → real project create (uv sync) → project dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun source, the Node bundle, and a compiled binary.

@github-actions github-actions Bot added the size/xl PR size: XL label Aug 12, 2026
@github-actions github-actions Bot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696c Compare August 14, 2026 20:38
@github-actions github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1 Compare August 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (d72e3c9) to head (dd15e9e).

Additional details and impacted files
@@                    Coverage Diff                    @@
##           feat/dev-otel-storage    #1980      +/-   ##
=========================================================
+ Coverage                  97.11%   97.13%   +0.02%     
=========================================================
  Files                        385      387       +2     
  Lines                      22918    23117     +199     
=========================================================
+ Hits                       22256    22455     +199     
  Misses                       662      662              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9 Compare August 17, 2026 17:53
Base automatically changed from feat/project-dev-handler to refactor August 17, 2026 20:46
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch 2 times, most recently from aabb87f to 4548048 Compare August 19, 2026 16:02
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 4548048 to a9eecdf Compare August 19, 2026 18:47
@Hweinstock

Copy link
Copy Markdown
Contributor

based on description, it sounds like #2039 needs to land first so looking at that one now.

@tejaskash
tejaskash changed the base branch from refactor to feat/dev-otel-storage August 19, 2026 19:22
@github-actions github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 19, 2026
Comment thread src/assets/cdk/bin/cdk.ts Outdated
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from a9eecdf to 84429af Compare August 19, 2026 19:42
@tejaskash

Copy link
Copy Markdown
Contributor Author

Review response, all four findings addressed in 84429afb:

  • Unrelated reversions (cdk target guard, config-bundle TUI, error classification) — root-caused: the commit had been rebuilt as a whole-tree snapshot (reset --soft) taken from a branch last rebased onto an older refactor, silently reverting everything merged since. Rebuilt from explicit paths on the current base; the diff now contains only OTEL/dev files (verified: zero files outside io/, core/dev/, handlers/project/, router/, templates, deps).
  • Linux Docker host-gateway — container runs now pass --add-host=host.docker.internal:host-gateway, so the OTLP endpoint rewrite resolves on Docker Engine; harmless on Desktop/Podman/Finch where the name already exists.
  • 413 before close — oversized bodies now receive the 413 (with Connection: close) before the socket is destroyed; covered by a real 51 MiB request test asserting status 413 rather than a reset.

tejaskash added a commit that referenced this pull request Aug 20, 2026
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 84429af to d070faf Compare August 20, 2026 16:09
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.

Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
tejaskash added a commit that referenced this pull request Aug 20, 2026
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from d070faf to dd15e9e Compare August 20, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xl PR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants