Skip to content

Stream apply file output, and keep the script alive if the session drops - #1151

Merged
cdalar merged 2 commits into
mainfrom
apply-stream-output
Sep 29, 2026
Merged

cdalar merged 2 commits into
mainfrom
apply-stream-output

Conversation

@cdalar

@cdalar cdalar commented Sep 29, 2026

Copy link
Copy Markdown
Owner

Problem

onctl up/create/ssh -a <file> ran the script as ./file > output-<file>.log 2>&1 and showed only a spinner until it finished. For rancher/rancher-local.sh that meant minutes of silence.

Why not just | tee

A plain | tee would stream the output, but it would also make the script depend on the SSH session. On Ctrl+C or a dropped connection, tee dies of SIGPIPE and takes the script with it partway through the install. I checked this on a real box: the script was killed right after its first line, with exit code 141.

What this does

In applyCommand (internal/tools/remote-run.go):

  • The apply file runs in a background subshell, with stdin, stdout and stderr all redirected. That subshell doesn't depend on the SSH session.
  • Output goes to the same output-<file>.log as before.
  • The apply dir also gets exit-code-<file> when the script finishes, and pid-<file> when it starts.
  • The session only follows the log (tail -f --pid) until the script exits, then exits with the script's exit code. Errors are reported the same way as before.

If the session goes away, only the output stops. The script finishes on the box, and its log, exit code and PID stay in the apply dir. A later session can reconnect with:

tail -n +1 -f --pid=$(cat pid-<file>) output-<file>.log

A logs command that does this is a natural follow-up, but it isn't in this PR.

While an apply file runs, the spinner is replaced by a plain Running <file> on Remote... line, because the spinner would garble the streamed output. This applies to both create and ssh.

Testing

  • New regression test: TestApplyCommandSurvivesLostSession runs the generated script, drops the reader after the first line, and checks that the apply file still finishes. It checks the full log, exit code 3, and a PID in the PID file. It needs GNU tail --pid, so it's skipped on macOS. It passes on Linux (Debian trixie box).
  • Full run on a real box: CopyAndRunRemoteFile against the box's own sshd printed output live, one line per second. Exit code 3 came back as an error.
  • Killed client: I sent onctl SIGTERM partway through, which is what its Ctrl+C handler amounts to (os.Exit(1)). The script kept running and finished with a complete log and exit-code 3. No stray tail processes were left.
  • Reconnecting: the first session dropped after 2 lines. A second session attached using the PID file, printed the rest as the lines were written, and ended when the script finished.
  • Existing failures: some tests in cmd fail both with and without this change: TestReadConfig_NoConfigDirectory and TestCreateFlagsBindToViper on macOS, and TestPopulateOnctlEnv_ErrorWritingFile as root on Linux.

Side effects

  • onctl's stdout now includes the apply file's output. boxctl-vms saves onctl's output and puts it in its error messages, so those will be longer.
  • Streaming needs GNU tail --pid, which Debian, Ubuntu and RHEL-family systems have, but BusyBox/Alpine may not.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NqKMF7otLzf5b9Vk8Tov6F

cdalar and others added 2 commits September 29, 2026 06:30
-a ran the script as `./script > output-<script>.log 2>&1` and showed
nothing but a spinner until it finished -- minutes of silence for
something like Rancher.

A plain `| tee` would stream it, but would also tie the script's life to
the SSH session: on Ctrl+C or a dropped connection tee dies of SIGPIPE
and takes the script with it mid-install. So the script now runs in a
background subshell with every stream redirected to the same log file,
independent of the session, and writes exit-code-<script> when done; the
session only tails the log (tail --pid) and exits with the script's
code. Losing the session now just stops the output -- the apply finishes
on the box, and its log and exit code stay in the apply dir.

The spinner is replaced by a plain "Running ... on Remote..." line while
an apply file runs, since it would garble the streamed output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NqKMF7otLzf5b9Vk8Tov6F
pid-<file> in the apply dir holds the background subshell's PID, so a
session started after the original one went away (Ctrl+C, a dropped
connection) can follow the same log the same way:
tail -n +1 -f --pid=$(cat pid-<file>) output-<file>.log

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NqKMF7otLzf5b9Vk8Tov6F
@cdalar
cdalar enabled auto-merge September 29, 2026 05:01
@cdalar
cdalar merged commit 3ed7c63 into main Sep 29, 2026
12 checks passed
@codecov

codecov Bot commented Sep 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 31.81818% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.89%. Comparing base (32f963a) to head (fc1a185).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
internal/tools/remote-run.go 38.88% 11 Missing ⚠️
cmd/create.go 0.00% 2 Missing ⚠️
cmd/ssh.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1151      +/-   ##
==========================================
+ Coverage   36.85%   36.89%   +0.03%     
==========================================
  Files          51       51              
  Lines        5643     5656      +13     
==========================================
+ Hits         2080     2087       +7     
- Misses       3563     3569       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址