Skip to content

feat(mecak8s): enable in-chat schedules for remote execution - #2286

Merged
JAORMX merged 8 commits into
mainfrom
feat/remote-schedules-kind-20261013
Oct 14, 2026
Merged

JAORMX merged 8 commits into
mainfrom
feat/remote-schedules-kind-20261013

Conversation

@JAORMX

@JAORMX JAORMX commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

CI status / integration blocker

Head: adb733ff1806f37fae8d0c8edd970daac949889a.

  • Vercel deployment completed successfully; its check watcher finished.
  • GitHub reports CONFLICTING / DIRTY against current main. A non-mutating merge-tree check identified internal/app/scheduler_fire.go, overlapping the scheduler cancellation/ownership lifecycle changes merged in fix(scheduler): keep polling behind stuck fires without releasing ownership #2277 (8e3bea42e).
  • No GitHub Actions run was created for this head: pull-request Actions are blocked by the merge conflict. This is not green CI and does not replace the local validation limitations below.
  • No rebase, force-push, branch merge, or conflict resolution was performed. Updating the branch requires operator approval, followed by integration validation and a fresh CI watch. The PR remains draft.

Summary

Enable in-chat scheduled tasks that borrow a native Kubernetes session's exact remote workspace. Restore the remote Schedule/ScheduleQuery catalog and remove the implicit CLI scheduler disable; explicit --no-scheduler remains honored.

  • Reauthorize the exact owner, origin binding, environment ref/revision and placement scope on each fire, then reserve and publish a distinct successor reference for that fire session.
  • Complete run cleanup with FinishRun, defer attachment reservation closure until publication, release process-local resources after ambiguous Commit responses without deleting durable state, and best-effort abort the exact reservation after a lost ReserveSuccessor response.
  • Add factory, scheduler, authenticated controller-backed and real Kind regression coverage. Update the canonical scheduled-task documentation and native-execution/architecture pages.
  • Fix the Kind fixture's proven API-egress failure: development previously omitted runtime discovery even though kindnet enforces NetworkPolicy. Discover and validate exact service/endpoint IPv4 /32s and API ports for both profiles; keep workload fixture egress production-only. Before/after TCP probes and a passing original-image baseline established the cause; no authorization, fencing or workload isolation was weakened.

Supported scope and intentional limits

  • Workspace-backed remote schedules must be created in chat and borrow the origin workspace. Independent remote workspace allocation is unsupported and fails before allocation; independent no-FS schedules remain supported.
  • A busy workspace fails the fire under the existing trigger policy; this does not introduce durable waiting or broaden retries.
  • The origin session must remain stored with the same binding. Schedule updates cannot retarget it. Retained fire sessions consume the existing shared 64-reference limit; deleting a schedule alone does not free those references.
  • Reservation abort remains best effort, not a distributed orphan-free cleanup guarantee. Ambiguous Commit retains the durable snapshot/reference and does not run the model.

Validation

This is deliberately a draft: the single full local race gate did not complete within 600 seconds, and live qualification is limited to the borrowed-schedule scope below.

Validation Result
Final-source task --concurrency 4 lint PASS
Full task --concurrency 4 test, before the final narrow reservation-response repair PASS; 191 passing package results
Full task --concurrency 4 test:race TIMEOUT at 600 seconds, no reported failure/race before cancellation; not claimed as a pass
Combined bounded race runs before the narrow repair Complete 218/218 unique package outcomes, plus three tagged passes; not a single successful full race invocation
Final-source executionclient, executioncontroller, server and app packages with -race -count=1 PASS
Narrow repair regression Failed before the repair, passed afterward, including race detection
task docs and task site:build after the narrow repair PASS
Offline go run ./cmd/mecademo, before the narrow repair PASS; tool result and permission ask/approval demonstrated
Final-source TestKindExecutionBorrowedSchedule PASS, no skips, 341.03 seconds

The final live test required two completed manual fires and a future durable timer after both provider and agent restarts. Every fire required successful Shell/Read event replay and completed persisted snapshots, distinct published fire references, exact environment/PVC continuity and remote-file verification. It used a scripted mock LLM, not paid model calls. This is not a claim that the entire production-network/renewal suite ran.

An additional explicitly tagged e2e lint invocation reported five findings outside the final narrow repair (context-argument ordering and shadowing, including shadowing in the new e2e helper). Ordinary full lint passed; that extra tagged invocation is not claimed green.

Reproducible committed coverage

Offline regressions:

go test -count=1 ./deploy/mecatl-execution-kind ./cmd/mecak8s
go test -race -count=1 ./internal/adapter/executionclient ./internal/adapter/executioncontroller ./internal/adapter/server ./internal/app

The committed live test is TestKindExecutionBorrowedSchedule in e2e/k8s_execution/borrowed_schedule_test.go (build tag kind_execution_e2e). The fixture command below builds/loads the images and tagged test binary, establishes explicit owned cluster context and fixture identity, and runs the production-profile qualification including this test:

CONTAINER_ENGINE=podman \
MECATL_EXECUTION_DEV_TOOLBOX=dev \
MECATL_EXECUTION_K8S_TOOLBOX=sre \
MECATL_EXECUTION_QUAL_CI=0 \
MECATL_EXECUTION_QUAL_NATURAL_EXPIRY=0 \
task e2e:k8s:execution:production

This is a reproduction entry point, not a claim that this entire command passed during the final focused validation. It provisions a new fixture and retains it locally; do not use it to mutate the retained validation cluster. A focused rerun with the fixture's explicit state/context and authenticated helper environment uses -test.run '^TestKindExecutionBorrowedSchedule$' -test.v -test.count=1 -test.timeout=10m on its tagged binary.

The existing native executor deployment is retained, with all 20 environment/Pod/PVC identities preserved and no deletion or unfencing. Final deployed agent: localhost/mecak8s@sha256:908dad40c1c3cac8a19c722b0ab003417080f8306ea5b8d0d03a67c6fe4a5aa3. Local receipts, fixture state, artifacts and credentials are not committed.

API compatibility

Backward-compatible, additive availability of the existing scheduling API in remote-execution deployments. No protobuf/HTTP schema, wire major, feature identifier, engine public Go API, or persisted schema changes. Existing conforming clients need no migration; deployment-specific availability is not protocol version skew. The explicit scheduler opt-out remains supported. Internal service plumbing changes do not alter the engine's public Go surface. Stored records remain compatible and exact placement authorization stays fail-closed; persisted tool grants are not silently widened. Older sessions without Schedule authority require a new eligible session to expose it. No breaking change or major-bump approval is required.

Publish distinct successor references for borrowed remote schedule fires, preserve exact placement authority, and close run and reservation lifecycles safely. Add offline and Kind coverage and correct fixture API-egress discovery.

Co-authored-by: Mecatl <noreply@mecatl.dev>
Preserve native borrowed-fire placement and ambiguous reservation cleanup while adopting scheduler deadline admission, liveness fencing, and terminal run release.

Co-authored-by: Mecatl <noreply@mecatl.dev>
…failure

Report only call names and result status, not tool output, to distinguish persistence from execution failures in production qualification.

Co-authored-by: Mecatl <noreply@mecatl.dev>
@JAORMX
JAORMX marked this pull request as ready for review October 13, 2026 16:51
JAORMX and others added 5 commits October 13, 2026 20:09
Add non-sensitive conversation shape to the failing production qualification assertion.

Co-authored-by: Mecatl <noreply@mecatl.dev>
Distinguish live execution projection failures from session snapshot persistence failures in Kind qualification.

Co-authored-by: Mecatl <noreply@mecatl.dev>
Started notices are injected into the origin conversation, not LLM turns on the new fire session. The old leading text response ended the fire before its Shell and Read proof, leaving the proof from another actor uncorrelated with the returned session.

Co-authored-by: Mecatl <noreply@mecatl.dev>
The mock provider cursor is deployment-wide; after a completed fire other model work may advance it before the next manual fire. Reload a dedicated script before asserting the second binding and proof.

Co-authored-by: Mecatl <noreply@mecatl.dev>
The second fire now follows an agent rollout, so it must use the same bounded leader-acquisition retry as the first fire.

Co-authored-by: Mecatl <noreply@mecatl.dev>
@JAORMX
JAORMX marked this pull request as draft October 13, 2026 20:33
@JAORMX
JAORMX marked this pull request as ready for review October 13, 2026 20:35
@JAORMX
JAORMX merged commit 84558eb into main Oct 14, 2026
108 of 147 checks passed
@JAORMX
JAORMX deleted the feat/remote-schedules-kind-20261013 branch October 14, 2026 04:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant