Repository navigation
v9.4.0.0p1 and later enforce permissions on the logs folder, leading to undiagnosable crashes of the service after Windows Update #2282
Description
Prerequisites
- Write a descriptive title.
- Make sure you are able to repro it on the latest version
- Search the existing issues.
Steps to reproduce
- Have OpenSSH working as a service
- Update OpenSSH via Windows Updates from something older than v9.4.0.0p1 to something newer
- Have the service fail to start because you had somehow created a
logsfolder atC:\ProgramData\ssh\logswithout extremely locked down permissions (I'm unable to confirm this right now, but it could have even been created by runningsshd.exeas a user, not as a service)
Expected behavior
- Service fails to start with a log message anywhere at all saying why
- Or service starts anyways
- Or this requirement for the
logsfolder to also have locked down permissions is documented somewhere, at least at https://github.com/PowerShell/Win32-OpenSSH/wiki/Security-protection-of-various-files-in-Win32-OpenSSH- To be fair I figured this out because of the patch notes for
v9.4.0.0p1saythe sshd service will check the
$env:ProgramData\sshfolder permissions upon startup to ensure only SYSTEM and Administrator accounts have write access to the folder; similar to the existing check upon install incontrib\win32\openssh\install-sshd.ps1. - And because of the PR add check for prog data folder permissions during sshd service startup openssh-portable#686
- To be fair I figured this out because of the patch notes for
Actual behavior
- Service fails to start with error 1067: The process terminated unexpectedly
- No logs show up in Event Viewer indicating why
- Troubleshooting as per https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps does not show any issues,
sshdruns fine withpsexec -s sshd.exe -d - It only gets fixed by either running the full
install-ssh.ps1or by doingrm C:\ProgramData\ssh\logs(and thus depending on the parent folder to have proper permissions)
Solutions
- Validate that the issue is undiagnosable by running
psexec -s sshd.exe -dsuccessfully - Run the full
install-ssh.ps1
or - Run
rm C:\ProgramData\ssh\logs(and thus depending on the parent folder to have proper permissions)
Error details
No response
Environment data
$PSVersionTable
Name Value
---- -----
PSVersion 7.3.5
PSEdition Core
GitCommitId 7.3.5
OS Microsoft Windows 10.0.20348
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0Version
OpenSSH_for_Windows_9.5p1
Visuals
No response
Reacted by ipcjs, Dan Nelson and Konrad
Reacted by serpentes80, Gabriel Ortiz and ms-mayya
Activity
seansaleh commentedon Oct 8, 2024
To be clear why I believe this is a notable issue:
I installed OpenSSH from the Windows Feature on a number of machines.
I was able to SSH to my servers just fine.
I updated Windows Server on Patch Tuesday.
OpenSSH got updated.
I was no longer to access or manage my servers, because the OpenSSH service would no longer start.
I was not able to diagnose the issue for a long time because there were no errors in the event viewer, and running sshd.exe as per the docs for troubleshooting didn't identify the issue, it worked just fine!
I had to physically connect to each machine individually to fix the OpenSSH folder permissions settings on the
logfolder.Generally speaking I'd expect that using Windows Update would be safe to update my servers with, and I'd still maintain access to them after running Windows Update.
Miraclys commentedon Oct 9, 2024
Thank you, I've wasted the morning looking for mistakes!
fmiermont commentedon Oct 9, 2024
Same issue here, after the patch tuesday OpenSSH server won't start, and nothing in the log even if set in DEBUG level.
The Log folder was previously created by OpenSSH... Just a to delete it, start the server : working ok.
crondrift commentedon Oct 9, 2024
Doesn't work for me. Installed it via msi, stopped the service, deleted the folder
C:\ProgramData\ssh\logs, started the service. The Log-Folder then gets recreated, but the service keeps crashing. Could someone who got it working post the exact folder permissions please?Superberti commentedon Oct 9, 2024
I also tried 9.8.0.0preview and I used the install-ssh.ps1 script.
Unfortunately the sshd still doesn't want to start.
Firewall rule created for ssh with
"New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH SSH Server' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22 -Program "C:\OpenSSH-Win64\sshd.exe""
and even tried to give sshd an firewall exception for everything.
From an ELEVATED power shell with ./sshd -ddd I got:
debug1: get_passwd: lookup_sid() failed: 1332.
debug1: rexec_argv[1]='-ddd'
debug3: using c:\openssh-win64/sshd-session.exe for re-exec
debug2: fd 7 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 7 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
debug3: socketio_bind - ERROR:10013
Bind to port 22 on :: failed: Permission denied.
debug2: fd 7 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
debug3: socketio_bind - ERROR:10013
Bind to port 22 on 0.0.0.0 failed: Permission denied.
Cannot bind any address.
Windows server 2022 21H2, Developer mode ON
Any idea where to look else? This doesn't look too bad, but I wonder why port 22 is still not accessable...
seansaleh commentedon Oct 9, 2024
crondrift and Superberti I think your issues are unrelated (see #2281 for issues with 9.8). This issue is specifically about existing working installs when the service was working before and update, but failing afterwards and when the troubleshooting steps (specifically running
psexec -s sshd.exe -d) don't show any issues. I'm also going to update the issue to make that clearer.(crondrift make sure you've tried both https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps and
powershell.exe -ExecutionPolicy Bypass -File install-sshd.ps1)[-]v9.4.0.0p1 and later enforce permissions more strictly, leading to undiagnosable crashes of the service[/-][+]v9.4.0.0p1 and later enforce permissions on the `logs` folder, leading to undiagnosable crashes of the service after Windows Update[/+]giacomoscoccia commentedon Oct 9, 2024
Same Issue here, same morning wasted.
Deleting
logsfolder seems solve the issue, but...DO NOT NAVIGATE the
logsfolder for any reason. It seems that Windows adds additional permissions that will prevent the service from starting again.In fact, Windows notices that the folder cannot be accessed and prompts for authorization to permanently gain access to the folder.
Evidently, pressing the Continue button adds additional permissions, which will then cause the sshd.exe service to fail to start.
twhug commentedon Oct 10, 2024
Thanks for reporting this and for providing the solution. Pretty appalled that this was released without a proper heads-up. Lost a good hour or two this morning chasing a fix until I saw yours.
crondrift commentedon Oct 10, 2024
Sean Saleh (@seansaleh)
Sorry, my fault. I forgot to mention, that I've updated from v9.5.0.0p1-Beta and everything was working fine until the update to v9.8.0.0p1-Preview. Already tried the troubleshooting and the
install-sshd.ps1.tgauth commentedon Oct 11, 2024
Sorry for all the headaches around this, and thank you for raising the issue here. To be clear, read access to the logs by any user is still permitted, but using the prompt for access grants full control. If you press 'continue` on the prompt, you'll just need to manually restrict the permissions in the folder properties to read/read & execute/list folder contents.
giacomoscoccia commentedon Oct 12, 2024
Thanks por clarification.
Why not consider creating the
logsfolder with at least read permissions for the administrators group?user8446 commentedon Oct 12, 2024
Is anyone else seeing connections and disconnects not logged anymore in v9.8?
EDIT: Opened separate issue: #2292
tgauth commentedon Oct 14, 2024
Full control to the
logsfolder is permitted for the administrators group and the SYSTEMbennoleslie commentedon Oct 15, 2024
I also ran into this, however the fix for me was correctly adjusting the permission on both the
C:\ProgramData\ssh\logsdirectory and also theC:\ProgramData\sshdirectory. Sharing in case anyone else hits this issue and the above fixes don't help.Trispast commentedon Oct 15, 2024
I had the same symptoms but none of the listed fixes ended up working. In my case I had double quotes in my sshd_config file around the path for the sftp-server.exe default path
Subsystem sftp sftp-server.exe -d "C:\SFTP". Once I removed thoseSubsystem sftp sftp-server.exe -d C:\SFTPI was able to get past the error.Chao216 commentedon Oct 16, 2024
Got 1607 error, not being able to start sshd. found my account have full control for %porgramdata%\ssh\log folder, issue fixed after removing my own account full control for this folder
used GitHub release Zip to deploy
Windows OS IOT LTSC 24h2
System
[ Name] Service Control Manager
[ Guid] {555908d1-a6d7-4695-8e1e-26931d2012f4}
[ EventSourceName] Service Control Manager
[ Qualifiers] 49152
Version 0
Level 2
Task 0
Opcode 0
Keywords 0x8080000000000000
[ SystemTime] 2024-10-11T11:02:29.2755749Z
EventRecordID 1576
Correlation
[ ProcessID] 680
[ ThreadID] 12840
Channel System
Computer Lavazza
Security
EventData
param1 OpenSSH SSH Server
param2 1
73007300680064000000
Deployment Image Servicing and Management tool
Version: 10.0.26100.1150
Image Version: 10.0.26100.2033
Packages listing:
------------------------------------------------------------------------------------------------------- | ---------- | --------------- | -------------------
31bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.131bf3856ad364e35amd64~~10.0.26100.131bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.131bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64de-DE11.0.26100.131bf3856ad364e35amd64en-US11.0.26100.174231bf3856ad364e35amd64en-US11.0.26100.203331bf3856ad364e35amd64zh-CN11.0.26100.131bf3856ad364e35amd64~~11.0.26100.174231bf3856ad364e35amd64~~11.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35wow64de-DE10.0.26100.131bf3856ad364e35wow64en-US10.0.26100.131bf3856ad364e35wow64zh-CN10.0.26100.131bf3856ad364e35wow64~~10.0.26100.174231bf3856ad364e35wow64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35wow64de-DE10.0.26100.131bf3856ad364e35wow64en-US10.0.26100.131bf3856ad364e35wow64zh-CN10.0.26100.131bf3856ad364e35wow64~~10.0.26100.174231bf3856ad364e35wow64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35wow64de-DE10.0.26100.131bf3856ad364e35wow64en-US10.0.26100.131bf3856ad364e35wow64zh-CN10.0.26100.131bf3856ad364e35wow64~~10.0.26100.174231bf3856ad364e35wow64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35wow64de-DE10.0.26100.131bf3856ad364e35wow64en-US10.0.26100.131bf3856ad364e35wow64zh-CN10.0.26100.131bf3856ad364e35wow64~~10.0.26100.174231bf3856ad364e35wow64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.131bf3856ad364e35amd64en-US10.0.26100.131bf3856ad364e35amd64zh-CN10.0.26100.131bf3856ad364e35amd64~~10.0.26100.131bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35wow64de-DE10.0.26100.131bf3856ad364e35wow64en-US10.0.26100.131bf3856ad364e35wow64zh-CN10.0.26100.131bf3856ad364e35wow64~~10.0.26100.174231bf3856ad364e35wow64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64de-DE10.0.26100.174231bf3856ad364e35amd64de-DE10.0.26100.203331bf3856ad364e35amd64en-US10.0.26100.174231bf3856ad364e35amd64en-US10.0.26100.203331bf3856ad364e35amd64zh-CN10.0.26100.174231bf3856ad364e35amd64zh-CN10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35wow64de-DE10.0.26100.131bf3856ad364e35wow64en-US10.0.26100.131bf3856ad364e35wow64zh-CN10.0.26100.131bf3856ad364e35wow64~~10.0.26100.174231bf3856ad364e35wow64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.26100.174231bf3856ad364e35amd64~~10.0.26100.203331bf3856ad364e35amd64~~10.0.9266.131bf3856ad364e35amd64~~10.0.9277.231bf3856ad364e35amd64~~26100.1742.1.1031bf3856ad364e35amd64~~26100.2033.1.2031bf3856ad364e35amd64~~26100.1738.1.331bf3856ad364e35amd64~~26100.2032.1.5The operation completed successfully.