feat(tracing): trace scheduled Runtime processing (#1243) by guozhihao-224 · Pull Request #1245 · oceanbase/powercontext

guozhihao-224 · 2026-08-16T15:43:39Z

Which issue or RFC does this PR close?

Closes #1243. Parent: #1214. Aligns scheduled tracing with RFC 0046.

Rationale for this change

Scheduled Source-window processing and Experience incubation reported outcomes only through structured logs (background.operation.completed). With no trace root, a scheduled run was invisible in traces unless it happened to join an unrelated HTTP or MCP context. RFC 0046 requires background activations to start their own trace when tracing is enabled.

What changes are included in this PR?

Extend the existing OTel-free RuntimeTracing port with background() (fresh trace root, unit=background) and RuntimeSpan.set_outcome() so domain code can record success / noop / failure without changing control flow. This port is an internal Runtime hook, not a public extension API (RFC 0046).
Implement ServerTracing.background() with an empty OTel Context() attached for the whole block, including when the root span fails to start, so child stages cannot inherit an inbound HTTP/MCP span.
Wrap each scoped scheduled activation in scheduled.process_source_window / scheduled.incubate_experience_candidates, and add memory.flush / experience.incubation stages on the shared flush() / incubate() boundaries (the same memory.flush stage also appears under HTTP flush_memory).
Outcomes are success, noop, failure, or cancelled, matching the existing logs. failure sets span status ERROR; the others stay UNSET. Attributes are bounded counts only — no scope_id, request ID, or Memory content.
Document the scheduled tree in the Phoenix how-to (EN/ZH) and note unit=background in RFC 0046.

Are there any user-facing changes?

Observability-only. When tracing is enabled, scheduled activations emit independent traces. No breaking API or persisted-format changes. Tracing stays optional; with no tracer injected, scheduler and Runtime control flow are unchanged. Custom RuntimeTracing adapters are not a supported public surface in this release.

How was this change tested?

uv run pytest tests/builtin/runtime/test_scheduler.py tests/e2e/test_observability.py tests/test_server_tracing.py
Unit coverage: scheduled roots, nested memory.flush / experience.incubation, outcomes success / noop / failure / cancelled for both processors, swallowed errors, isolation when the background root fails to start, and no scope_id in exported attributes
E2E: real Server with a short scheduler interval; Source-window and Experience roots have parent is None, do not share a trace_id with HTTP spans, and export no scope ID or captured content

Exported span trees (issue requirement), from the e2e InMemorySpanExporter:

scheduled.process_source_window
  powercontext.operation.name=process_source_window
  powercontext.operation.unit=background
  powercontext.operation.outcome=success
  powercontext.background.source_count=1
└── memory.flush
      powercontext.operation.unit=stage
      powercontext.operation.outcome=success
      powercontext.memory.flush.source_count=1

scheduled.incubate_experience_candidates
  powercontext.operation.name=incubate_experience_candidates
  powercontext.operation.unit=background
  powercontext.operation.outcome=success
  powercontext.background.source_count=1
  powercontext.background.candidate_count=0
└── experience.incubation
      powercontext.operation.unit=stage
      powercontext.operation.outcome=success
      powercontext.experience.incubation.source_count=1
      powercontext.experience.incubation.candidate_count=0

scope.context / scope.lock remain existing setup stages and close before the operation body, so they appear as siblings of memory.flush / experience.incubation. When inference is instrumented, invoke_agent / chat nest under those stages.

AI usage statement

Cursor (Grok 4.6) was used to implement the scheduled tracing port, tests, and docs; to review the change against #1243; and to draft this PR description. The design, issue scope, and final review of the exported trees were directed by the author.

CLAassistant · 2026-08-16T15:43:46Z

All committers have signed the CLA.

Copilot

Pull request overview

This PR adds scheduled-processing trace roots to the built-in Runtime (per RFC 0046 / issue #1243) by introducing a minimal tracing protocol for domain code and adapting the server’s OpenTelemetry tracing to that protocol. The goal is to make scheduled Source-window processing and Experience incubation observable in traces without leaking scope/content data and without coupling Runtime code to OTel.

Changes:

Introduce an OpenTelemetry-free Tracer/Span protocol for domain code and inject it into the built-in Runtime.
Add scheduled trace roots (scheduled.process_source_window, scheduled.incubate_experience_candidates) and boundary spans (memory.flush, experience.incubation) with outcome attributes.
Update tests and Phoenix tracing docs to reflect the new span tree (including memory.flush under HTTP operations and scheduled activations).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/e2e/test_observability.py	Extends e2e assertions for the new `memory.flush` boundary span and adds an e2e scheduled-activation trace-root test.
tests/builtin/runtime/test_scheduler.py	Adds unit tests asserting scheduled trace roots and outcome attributes (noop/success/failure/cancelled) plus data-policy checks.
src/powercontext/tracing.py	Introduces the minimal domain tracing protocol (`Tracer`/`Span`) to decouple Runtime from OTel.
src/powercontext/server/tracing.py	Adds `DomainTracer` adapter bridging the protocol to `ServerTracing` and implements root-span creation for scheduled work.
src/powercontext/server/factory.py	Wires `DomainTracer` into Runtime construction in the server app lifespan.
src/powercontext/builtin/runtime/composition.py	Extends `open_builtin_runtime` to accept and pass through the injected domain tracer.
src/powercontext/builtin/runtime/application.py	Emits scheduled root spans and adds `memory.flush` / `experience.incubation` boundary spans with outcomes and bounded attributes.
docs/zh/docs/how-to/trace-with-phoenix.md	Updates the Phoenix trace walkthrough to include `memory.flush` and scheduled trace roots.
docs/en/docs/how-to/trace-with-phoenix.md	Updates the Phoenix trace walkthrough to include `memory.flush` and scheduled trace roots.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    def start_span(self, name: str, *, attributes: dict[str, object]) -> _ActiveSpan:
+        span_attributes = dict(attributes)
+        request_id = current_request_id()
+        # note(guozhihao-224): only child spans join a request; scheduled roots are fresh traces with no request id.
+        if request_id is not None:
+            span_attributes["powercontext.request.id"] = request_id
+        return self._tracing.start_span(
+            name,
+            kind=SpanKind.INTERNAL,
+            attributes=span_attributes,
+            context=None,
+        )
+
+    def start_root_span(self, name: str, *, attributes: dict[str, object]) -> _ActiveSpan:
+        # note(guozhihao-224): fresh empty context keeps scheduled activations as independent trace roots.
+        return self._tracing.start_span(
+            name,
+            kind=SpanKind.INTERNAL,
+            attributes=dict(attributes),
+            context=Context(),
+        )


…ow and experience incubation - Added support for tracing scheduled activations as independent roots, allowing for better observability of background operations. - Introduced new spans: `scheduled.process_source_window` and `scheduled.incubate_experience_candidates`, with attributes for source and candidate counts. - Enhanced existing spans to include outcome tracking (success, noop, failure) for memory flush and experience incubation operations. - Updated documentation to reflect changes in tracing structure and added examples for scheduled tracing usage.

Kairo-J · 2026-08-29T08:22:03Z

The PR description mentions that the exported Span tree can be regenerated using scripts/trace_scheduled_demo.sh --phoenix, yet the script is missing from the latest commit (HEAD) of the PR.

I noticed that the second commit removed it. Was there a specific reason for this? Could you please clarify, update the PR description, or add the script back?

Ethan-Xingyue · 2026-08-29T08:51:22Z

plz fix ci

…oot failure - Implemented isolation for child spans when the root span fails to start, ensuring that child spans do not join an ambient trace. - Added a new test to verify that child spans are correctly isolated in this scenario. - Updated existing tests to improve coverage and ensure accurate outcome tracking for scheduled experiences.

guozhihao-224 · 2026-08-29T14:39:02Z

plz fix ci

hi @Ethan-Xingyue The Main failure was ruff format --check on tests/test_server_tracing.py. That formatting is fixed in the follow-up commit together with the background isolation and Experience noop coverage.

guozhihao-224 · 2026-08-29T14:39:43Z

The PR description mentions that the exported Span tree can be regenerated using scripts/trace_scheduled_demo.sh --phoenix, yet the script is missing from the latest commit (HEAD) of the PR.

I noticed that the second commit removed it. Was there a specific reason for this? Could you please clarify, update the PR description, or add the script back?

ha @Kairo-J The script was only a local Phoenix helper and was dropped on purpose so it would not ship in the PR. The exported trees in the description come from the e2e InMemorySpanExporter. I will remove the scripts/trace_scheduled_demo.sh --phoenix lines from the PR body so it matches HEAD.

Ethan-Xingyue · 2026-08-29T16:02:50Z

plz fix ci

Kairo-J · 2026-08-30T07:35:03Z

lgtm

AlexStocks requested a lite review from Copilot August 17, 2026 03:40

Copilot started reviewing on behalf of AlexStocks August 17, 2026 03:40 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

guozhihao-224 force-pushed the feat/scheduled-processing-tracing branch from b252bd7 to b672901 Compare August 29, 2026 07:24

guozhihao-224 marked this pull request as ready for review August 29, 2026 07:25

chore: drop local scheduled tracing demo script

66df758

Kairo-J reviewed Aug 29, 2026

View reviewed changes

Comment thread src/powercontext/server/tracing.py Outdated

Kairo-J reviewed Aug 29, 2026

View reviewed changes

Comment thread src/powercontext/builtin/runtime/protocols.py

Kairo-J reviewed Aug 29, 2026

View reviewed changes

Comment thread tests/test_server_tracing.py

Kairo-J reviewed Aug 29, 2026

View reviewed changes

Comment thread tests/builtin/runtime/test_scheduler.py

guozhihao-224 force-pushed the feat/scheduled-processing-tracing branch from 9c7fc98 to 4ecd6ff Compare August 29, 2026 14:35

guozhihao-224 requested a review from Kairo-J August 29, 2026 14:41

fix lint

a1ccd4d

Kairo-J approved these changes Aug 30, 2026

View reviewed changes

AlexStocks merged commit 4213d32 into oceanbase:master Aug 30, 2026
13 checks passed

Navigation Menu

feat(tracing): trace scheduled Runtime processing (#1243) - #1245

feat(tracing): trace scheduled Runtime processing (#1243)#1245
AlexStocks merged 4 commits into
oceanbase:masterfrom
guozhihao-224:feat/scheduled-processing-tracing

guozhihao-224 commented Aug 16, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Aug 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kairo-J commented Aug 29, 2026

Uh oh!

Ethan-Xingyue commented Aug 29, 2026

Uh oh!

guozhihao-224 commented Aug 29, 2026 •

edited

Loading

Uh oh!

guozhihao-224 commented Aug 29, 2026

Uh oh!

Ethan-Xingyue commented Aug 29, 2026

Uh oh!

Kairo-J commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

guozhihao-224 commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue or RFC does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

How was this change tested?

AI usage statement

Uh oh!

CLAassistant commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kairo-J commented Aug 29, 2026

Uh oh!

Ethan-Xingyue commented Aug 29, 2026

Uh oh!

guozhihao-224 commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guozhihao-224 commented Aug 29, 2026

Uh oh!

Ethan-Xingyue commented Aug 29, 2026

Uh oh!

Kairo-J commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

guozhihao-224 commented Aug 16, 2026 •

edited

Loading

CLAassistant commented Aug 16, 2026 •

edited

Loading

guozhihao-224 commented Aug 29, 2026 •

edited

Loading