Skip to content

test(e2e): Claude per-turn overrides, runtime user, and an SDK-driver Harness - #76

Open
oldsj wants to merge 1 commit into
mainloop-substrate-1-0from
test/claude-kind-dev-proof
Open

oldsj wants to merge 1 commit into
mainloop-substrate-1-0from
test/claude-kind-dev-proof

Conversation

@oldsj

@oldsj oldsj commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Summary

This adds E2E coverage for #69's per-turn override, the harness runtime user, and the Agent SDK driver.

  • TestE2EClaudeTurnOverridesAndRuntimeUser sends three turns through the public A2A API against a recorded mock Anthropic API, then checks what reached the model:

    1. With kagent.dev/model=claude-opus-5-5 and kagent.dev/effort=high, every agent-loop request carries that model and effort.
    2. With neither key, requests use the configured claude-sonnet-5-5 with its default effort. They never carry the previous turn's value.
    3. With kagent.dev/effort=low only, requests keep the configured model and use low.

    On the first turn, the harness's Bash tool also reports its uid and gid, and the owner and mode of the durable directories. It then writes a file to the workspace. The test requires a non-root user that owns what it writes. Substrate 1.0 creates /data root-owned with mode 0777.

  • claude-sdk-e2e Harness fixture (KAGENT_CLAUDE_DRIVER=sdk) in lifecycle.yaml.tmpl, plus KAGENT_E2E_CLAUDE_HARNESS. With that set to claude-sdk-e2e, every Claude case runs on the SDK driver. The default stays claude-e2e, so CI is unchanged. The upgrade test's fixture filter ignores the new Harness.

  • With Claude 5 family models, the CLI ends each request with a system-role context message (token budget and date). mockllm matches only the last message, and only on role. The new test's proxy therefore strips trailing system messages before matching, while the recorder keeps the original request. The other Claude E2E cases use claude-sonnet-4-5 and aren't affected.

Testing

I ran this on kind-substrate-1-0 against the ported controller 0.0.0-7945f19 and harness image 0.0.0-c3cb83a-arm64, with no real model turns. It passes on both drivers:

Driver turn 1 (model, effort) turn 2 turn 3
process opus-5-5, high sonnet-5-5, medium sonnet-5-5, low
SDK opus-5-5, high sonnet-5-5, medium sonnet-5-5, low

Probe output was identical on both drivers:

65532
65532
0:0 777 /data
65532:65532 700 /data/workspace
65532:65532 700 /data/claude
65532:65532 700 /data/adapter
probe 65532:65532 644
PROBE_WRITE_OK
  • go vet passes, and lint reports 0 issues for core/test/e2e.
  • To run against an install that uses service-account auth, I added a bearer-token interceptor to the E2E client. I kept it out of this PR, because the suite assumes x-user-id auth.

🤖 Generated with Claude Code

🤖 Generated with Claude Code

… Harness

TestE2EClaudeTurnOverridesAndRuntimeUser sends three turns through the
public A2A API against a recorded mock Anthropic API. A turn with
kagent.dev/model and kagent.dev/effort uses them. The next turn, with
neither key, uses the configured model and its default effort. An
effort-only turn keeps the configured model. The first turn's Bash probe
checks that the harness runs non-root and owns what it writes under
Substrate's root-owned, mode 0777 durable directory.

The claude-sdk-e2e Harness fixture sets KAGENT_CLAUDE_DRIVER=sdk.
KAGENT_E2E_CLAUDE_HARNESS selects it, so every Claude case can run on
the Agent SDK driver.

Claude 5 family requests end with a system-role context message, which
mockllm's last-message matching can't see past. The test strips that
message before matching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: James Olds <12104969+oldsj@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址