Worker thread hangs when it attempts to print an error message

[asmichi]

I happened to notice this core problem of #1330 that makes ssh.exe hang.

"OpenSSH for Windows" version
7.7.2.0 (v7.7.2.0p1-Beta, x64)

Server OperatingSystem
Unknown (github.com)

Client OperatingSystem
Windows 10 Pro Version 1903 Build 18362.239

What is failing
A worker thread hangs when it attempts to print an error message.

Expected Behavior
ssh.exe doesn't hang.

Actual Behavior
ssh.exe hangs.

[asmichi]

While investigating git-for-windows/git#2143 (same as #1330), I have noticed that this issue is a much more fundamental issue.

In short, fileio_write on a blocking fd in a worker thread will never complete.

• fileio_write performs syncio_initiate_write that spawns WriteThread
  • WriteThread queues APC to the queue of the main thread on completion.
• fileio_write then performs wait_for_any_event(NULL, 0, INFINITE) that ultimately calls SleepEx(INFINITE, TRUE)
  • A worker thread will never be alerted, because APC goes to the main thread,

Looking through the overall implementation of fd emulation (complete lack of synchronization), it doesn't look like fileio_write is expected to be called outside the main thread.

git-for-windows/git#2143 (comment)

The worker thread was stuck here:
https://github.com/PowerShell/openssh-portable/blob/v7.7.2.0/contrib/win32/win32compat/signal_wait.c#L96

 	ntdll.dll!NtDelayExecution�()	Unknown
 	KernelBase.dll!SleepEx()	Unknown
>	ssh.exe!wait_for_multiple_objects_enhanced(unsigned long nCount, void * const * lpHandles, unsigned long dwMilliseconds, int bAlertable) Line 96	C
 	ssh.exe!wait_for_any_event(void * * events, int num_events, unsigned long milli_seconds) Line 289	C
 	ssh.exe!fileio_write(w32_io * pio, const void * buf, unsigned __int64 max_bytes) Line 746	C
 	ssh.exe!w32_write(int fd, const void * buf, unsigned __int64 max) Line 531	C
 	ssh.exe!do_log(LogLevel level, const char * fmt, char * args) Line 462	C
 	ssh.exe!error(const char * fmt, ...) Line 164	C
 	ssh.exe!ReadThread(void * lpParameter) Line 96	C
 	[Inline Frame] ssh.exe!invoke_thread_procedure(unsigned int(*)(void *) context, void * const) Line 91	C++
 	ssh.exe!thread_start<unsigned int (__cdecl*)(void * __ptr64)>(void * const parameter) Line 115	C++
 	kernel32.dll!00007ff8e15d7bd4()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

The main thread was waiting the worker thread:
https://github.com/PowerShell/openssh-portable/blob/v7.7.2.0/contrib/win32/win32compat/termio.c#L264

 	ntdll.dll!NtWaitForSingleObject�()	Unknown
 	KernelBase.dll!WaitForSingleObjectEx�()	Unknown
>	ssh.exe!syncio_close(w32_io * pio) Line 266	C
 	ssh.exe!fileio_close(w32_io * pio) Line 972	C
 	ssh.exe!w32_close(int fd) Line 611	C
 	ssh.exe!channel_close_fd(ssh * ssh, int * fdp) Line 428	C
 	ssh.exe!chan_shutdown_read(ssh * ssh, Channel * c) Line 409	C
 	ssh.exe!chan_rcvd_oclose(ssh * ssh, Channel * c) Line 292	C
 	ssh.exe!channel_input_oclose(int type, unsigned int seq, ssh * ssh) Line 3040	C
 	ssh.exe!ssh_dispatch_run(ssh * ssh, int mode, volatile int * done) Line 114	C
 	[Inline Frame] ssh.exe!ssh_dispatch_run_fatal(ssh * ssh, int) Line 133	C
 	[Inline Frame] ssh.exe!client_process_buffered_input_packets() Line 1157	C
 	ssh.exe!client_loop(ssh * ssh, int) Line 1299	C
 	ssh.exe!main(int ac, char * * av) Line 1551	C
 	ssh.exe!wmain(int argc, wchar_t * * wargv) Line 61	C
 	[Inline Frame] ssh.exe!invoke_main() Line 79	C++
 	ssh.exe!__scrt_common_main_seh() Line 253	C++
 	kernel32.dll!00007ff8e15d7bd4()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

[asmichi]

In other words, error (and the like) cannot be called inside a worker thread (ReadThread here).

[asmichi]

Forgot to paste the repro...

C:\>C:\tmp\OpenSSH-Win64-v7.7.2.0p1-Beta\ssh.exe -- [email protected] < NUL
warning: agent returned different signature type ssh-rsa (expected rsa-sha2-512)
GetConsoleMode on STD_INPUT_HANDLE failed with 6

GetConsoleMode on STD_INPUT_HANDLE failed with 6

PTY allocation request failed on channel 0
(hangs here)