Skip to content

Cancellable reads; EOF on "read" sets var to "" - #1066

Merged
mvdan merged 1 commit into
mvdan:masterfrom
theclapp:ctxreader2
Apr 7, 2024
Merged

mvdan merged 1 commit into
mvdan:masterfrom
theclapp:ctxreader2

Conversation

@theclapp

@theclapp theclapp commented Mar 8, 2024

Copy link
Copy Markdown
Contributor
  • Make reading cancellable. Not all input from stdin, just that done directly by the shell (i.e. the "read" builtin). Exec'ed programs still read directly from stdin's os.File and are not cancellable.
  • If you press ^D (EOF) when reading into a shell variable, set the variable to "". This is consistent with bash & zsh.

Fixes #856.
See also #857 for my first try at this.

@mvdan mvdan left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is one of those that is hard to test properly, but I'd still like us to try, because the logic is non-trivial and feels fragile.

For example, with https://pkg.go.dev/os#Pipe to create a stdin file, and cancelling the runner context to mimic ^D, we could test a few edge cases with the read builtin:

  • only newline (empty input)
  • only ^D (empty input)
  • some string and newline (non-empty input)
  • some string and ^D (non-empty input)

Comment thread interp/builtin.go Resolved
@mvdan

mvdan commented Mar 11, 2024

Copy link
Copy Markdown
Owner

Overall nothing against merging this feature though - it is one added dependency, but it seems like a light one, and I don't think we can possibly implement cancelling reads any way other than OS-specific code.

@theclapp

Copy link
Copy Markdown
Contributor Author

I assume this is one of those that is hard to test properly, but I'd still like us to try, because the logic is non-trivial and feels fragile.

Sure, no worries.

@theclapp

Copy link
Copy Markdown
Contributor Author
  • only newline (empty input)
  • only ^D (empty input)
  • some string and newline (non-empty input)
  • some string and ^D (non-empty input)

It turns out the current code already works with the first, third, and fourth. I've added a test case with the second.

None of these cases actually test having a read in-progress and cancelling its context, so I wrote a separate test for that. It turns out it wasn't that difficult.

@mvdan mvdan left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI CI appears broken

Comment thread interp/interp_test.go Resolved
Comment thread interp/interp_test.go Resolved
Comment thread interp/builtin.go Resolved
@theclapp

Copy link
Copy Markdown
Contributor Author

As noted in the commit: TestCancelreader failing intermittently under Linux (on a Debian VM), even though the read context is definitely cancelled. Not sure what's going on. Committing what I've got, for posterity.

@theclapp

Copy link
Copy Markdown
Contributor Author

TestCancelreader failing intermittently under Linux

At a guess, this is because of, or at least related to, the data race reported in the CI tests. Will look more tomorrow.

@theclapp
theclapp requested a review from mvdan March 26, 2024 14:23
@theclapp

Copy link
Copy Markdown
Contributor Author

@ghostsquad Wanna take a look?

@mvdan mvdan left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM, thanks! If anyone else has thoughts or a follow-up review, I'm happy to get more PRs after this lands.

@mvdan

mvdan commented Mar 30, 2024

Copy link
Copy Markdown
Owner

@theclapp do you want to squash your commits into one and write a single commit message? I could squash myself, only keeping the original commit message, but I suspect you might want to keep some of the text from the other commit messages, e.g. about the test.

- Make reading cancellable. Not all input from stdin, just that done
  directly by the shell (i.e. the "read" builtin). Exec'ed programs
  still read directly from stdin's os.File and are not cancellable.
- If you press ^D (EOF) when reading into a shell variable, set the
  variable to "". This is consistent with bash & zsh.
@theclapp

theclapp commented Apr 2, 2024

Copy link
Copy Markdown
Contributor Author

Squashed 'em all.

apricote pushed a commit to hetznercloud/fleeting-plugin-hetzner that referenced this pull request Jan 9, 2026
…ercloud/fleeting-plugin-hetzner!333)

This MR contains the following updates:

| Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| [mvdan.cc/sh/v3/cmd/shfmt](https://github.com/mvdan/sh) | `v3.8.0` → `v3.12.0` | ![age](https://developer.mend.io/api/mc/badges/age/go/mvdan.cc%2fsh%2fv3%2fcmd%2fshfmt/v3.12.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/mvdan.cc%2fsh%2fv3%2fcmd%2fshfmt/v3.8.0/v3.12.0?slim=true) |

Note: The `pre-commit` manager in Renovate is not supported by the `pre-commit` maintainers or community. Please do not report any problems there, instead [create a Discussion in the Renovate repository](https://github.com/renovatebot/renovate/discussions/new) if you have any questions.

---

### Release Notes

<details>
<summary>mvdan/sh (mvdan.cc/sh/v3/cmd/shfmt)</summary>

### [`v3.12.0`](https://github.com/mvdan/sh/releases/tag/v3.12.0)

[Compare Source](mvdan/sh@v3.11.0...v3.12.0)

- The `mvdan-sh` JS package is discontinued in favor of `sh-syntax` - [#&#8203;1145](mvdan/sh#1145)
- **cmd/shfmt**
  - Support the "simplify" and "minify" flags via EditorConfig - [#&#8203;819](mvdan/sh#819)
  - `--simplify` and `--minify` are now formatting options, disabling the use of EditorConfig
  - Do not allow `--write` to replace non-regular files - [#&#8203;843](mvdan/sh#843)
- **interp**
  - Add `IsBuiltin` to check if a command name is a shell built-in - [#&#8203;1164](mvdan/sh#1164)
  - Add `HandlerContext.Builtin` to allow `ExecHandlerFunc` to call built-ins
  - Initial support for `$!` and `wait PID` - [#&#8203;221](mvdan/sh#221)
  - Return non-fatal `ExecHandlerFunc` errors via the `Runner.Run` API
  - Add `HandlerContext.Pos` to provide handlers with source positions
  - Deprecate `NewExitStatus` and `IsExitStatus` in favor of `ExitStatus`
  - Fix `wait` to always return the status of the last given job
  - Copy all env vars for background subshells to avoid data races
  - Support reading random numbers via `$RANDOM` and `$SRANDOM`
  - Set `$BASH_REMATCH` when matching regular expressions via `=~`
  - Support modifying local vars from the parent calling function
- **expand**
  - Adjust which backslash sequences are expanded in here-docs - [#&#8203;1138](mvdan/sh#1138)
  - Tweak tilde expansions to match Bash semantics
- **pattern**
  - Remove the flawed and broken `Braces` mode; use `syntax.SplitBraces` instead
  - Tweak `**` to only act as "globstar" when alone as a path element - [#&#8203;1149](mvdan/sh#1149)
  - Tweak `*` and `**` to not match leading dots in basenames
  - Add a `NoGlobStar` mode to match the POSIX semantics
- **fileutil**
  - Treat all non-regular files as definitely not shell scripts - [#&#8203;1089](mvdan/sh#1089)

Consider [becoming a sponsor](https://github.com/sponsors/mvdan) if you benefit from the work that went into this release!

Binaries built on `go version go1.24.4 linux/amd64` with:

```
CGO_ENABLED=0 go build -trimpath -ldflags="-w -s -X=main.version=v3.12.0"
```

### [`v3.11.0`](https://github.com/mvdan/sh/releases/tag/v3.11.0)

[Compare Source](mvdan/sh@v3.10.0...v3.11.0)

This release drops support for Go 1.22 and includes many enhancements.

- **cmd/shfmt**
  - Support `-l=0` and `-f=0` to split filenames with null bytes - [#&#8203;1096](mvdan/sh#1096)
- **syntax**
  - New iterator API: `Parser.WordsSeq`
  - Fix `Parser.Incomplete` and `IsIncomplete` to work well with `Parser.Words` - [#&#8203;937](mvdan/sh#937)
  - Initial support for parsing incomplete shell via `RecoverErrors`
  - Expand `LangError` to include which language was used when parsing
- **interp**
  - Refactor setting variables to fix array declaration edge cases - [#&#8203;1108](mvdan/sh#1108)
  - Fix `test` read/write/exec operators to work correctly on directories - [#&#8203;1116](mvdan/sh#1116)
  - Replace the `cancelreader` dependency with `os.File.SetReadDeadline`
  - Avoid waiting for process substitutions, matching Bash
  - Skip `OpenHandler` when opening named pipes for process substitutions - [#&#8203;1120](mvdan/sh#1120)
  - Use `TMPDIR` if set via `Env` to create temporary files such as named pipes
- **expand**
  - New iterator API: `FieldsSeq`
  - Correctly handle repeated backslashes in double quotes - [#&#8203;1106](mvdan/sh#1106)
  - Don't expand backslashes inside here-documents - [#&#8203;1070](mvdan/sh#1070)
  - Replace the `Unset` kind with a new `Variable.Set` boolean field

Consider [becoming a sponsor](https://github.com/sponsors/mvdan) if you benefit from the work that went into this release!

Binaries built on `go version go1.24.0 linux/amd64` with:

```
CGO_ENABLED=0 go build -trimpath -ldflags="-w -s -X=main.version=v3.11.0"
```

### [`v3.10.0`](https://github.com/mvdan/sh/releases/tag/v3.10.0)

[Compare Source](mvdan/sh@v3.9.0...v3.10.0)

- **cmd/shfmt**
  - Report the correct language variant in parser error messages - [#&#8203;1102](mvdan/sh#1102)
  - Move `--filename` out of the parser options category - [#&#8203;1079](mvdan/sh#1079)
- **syntax**
  - Parse all CRLF line endings as LF, including inside heredocs - [#&#8203;1088](mvdan/sh#1088)
  - Count skipped backslashes inside backticks in position column numbers - [#&#8203;1098](mvdan/sh#1098)
  - Count skipped null bytes in position column numbers for consistency
- **interp**
  - Fix a regression in `v3.9.0` which broke redirecting files to stdin - [#&#8203;1099](mvdan/sh#1099)
  - Fix a regression in `v3.9.0` where `HandlerContext.Stdin` was never nil
  - Add an `Interactive` option to be used by interactive shells - [#&#8203;1100](mvdan/sh#1100)
  - Support closing stdin, stdout, and stderr via redirections like `<&-`

Consider [becoming a sponsor](https://github.com/sponsors/mvdan) if you benefit from the work that went into this release!

Binaries built on `go version go1.23.2 linux/amd64` with:

```
CGO_ENABLED=0 go build -trimpath -ldflags="-w -s -X=main.version=v3.10.0"
```

### [`v3.9.0`](https://github.com/mvdan/sh/releases/tag/v3.9.0)

[Compare Source](mvdan/sh@v3.8.0...v3.9.0)

This release drops support for Go 1.21 and includes many fixes.

- **cmd/shfmt**
  - Switch the diff implementation to remove one dependency
- **syntax**
  - Protect against overflows in position offset integers
- **interp**
  - Use `os.Pipe` for stdin to prevent draining by subprocesses - [#&#8203;1085](mvdan/sh#1085)
  - Support cancelling reads in builtins when stdin is a file - [#&#8203;1066](mvdan/sh#1066)
  - Support the `nocaseglob` bash option - [#&#8203;1073](mvdan/sh#1073)
  - Support the Bash 5.2 `@k` parameter expansion operator
  - Support the `test -O` and `test -G` operators on non-Windows - [#&#8203;1080](mvdan/sh#1080)
  - Support the `read -s` builtin flag - [#&#8203;1063](mvdan/sh#1063)
- **expand**
  - Add support for case insensitive globbing - [#&#8203;1073](mvdan/sh#1073)
  - Don't panic when pattern words are nil - [#&#8203;1076](mvdan/sh#1076)

A special thanks to [@&#8203;theclapp](https://github.com/theclapp) for their contributors to this release!

Consider [becoming a sponsor](https://github.com/sponsors/mvdan) if you benefit from the work that went into this release!

Binaries built on `go version go1.23.0 linux/amd64` with:

```
CGO_ENABLED=0 go build -trimpath -ldflags="-w -s -X=main.version=v3.9.0"
```

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever MR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi43Ni4wIiwidXBkYXRlZEluVmVyIjoiNDIuNzYuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

interp: Cancelling Runner.Run's context doesn't abort read builtin

2 participants

⌂ 换网址