core.sshCommand set to Windows integrated SSH freezes on LFS authentication if SSH remote is used #2143
Closed
1 task done
Labels
Comments
|
So, that was the initial issue, now I also worked to get a MCVE with raw CMD: This should be sufficient: And the log for that one: |
|
Another update: no difference if key has been added to agent or not. In scenario of no key added to agent, upon verifying passphrase, the checkout freezes. |
|
Looking at our SSH code, we invoke SSH with Normally this isn't a problem, since the user will have authenticated with a key by cloning with SSH using Git, and any prompting will have occurred there, but if the user normally receives a prompt on SSH usage, or is for some reason getting one here, that might explain it. If that's what it looks like it is, I've pushed up a commit to the |
|
@bk2204 built your version and tried with it, the problem still persists. |
|
I also hit this issue, so I decided to investigate it. Let me share my current progress. ssh.exe is stuck while it is trying to write to stderr " There seems to be two separate (but strongly related) problems:
Details
I got the stack traces below. The worker thread was stuck here: The main thread was waiting the worker thread: |
|
I had seen something similar in the past.
A child process was hanging (waiting for the parent to read from the pipe
it tries to write to, but the buffer is full) because the parent process
was not clearing the child's output buffer to which it connected itself
using pipes. Not sure if that's the case here, I just wanted to give a hint
where to look further.
…On Sat, Jul 27, 2019, 6:38 AM asmichi ***@***.***> wrote:
I also hit this issue, so I decided to investigate it. Let me share my
current progress.
ssh.exe is stuck while it is trying to write to stderr "GetConsoleMode on
STD_INPUT_HANDLE failed with 6".
There seems to be two separate (but strongly related) problems:
- GetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), &dwAttributes)
<https://github.com/PowerShell/openssh-portable/blob/v7.7.2.0/contrib/win32/win32compat/termio.c#L93>
fails.
- This write to stderr doesn't complete, even after the ancestors
(git.exe - git.exe - sh.exe - git-lfs.exe) exit.
Details
1. Download and extract OpenSSH for Windows v7.7.2.0p1-Beta
<https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v7.7.2.0p1-Beta>
(the executables and the symbols)
2. Configure git to use it (GIT_SSH or whatever)
3. Reproduce the issue.
4. Attach to the stuck ssh process.
I got the stack traces below.
The worker thread was stuck here:
https://github.com/PowerShell/openssh-portable/blob/v7.7.2.0/contrib/win32/win32compat/signal_wait.c#L96
ntdll.dll!NtDelayExecution�() Unknown
KernelBase.dll!SleepEx() Unknown
> ssh.exe!wait_for_multiple_objects_enhanced(unsigned long nCount, void * const * lpHandles, unsigned long dwMilliseconds, int bAlertable) Line 96 C
ssh.exe!wait_for_any_event(void * * events, int num_events, unsigned long milli_seconds) Line 289 C
ssh.exe!fileio_write(w32_io * pio, const void * buf, unsigned __int64 max_bytes) Line 746 C
ssh.exe!w32_write(int fd, const void * buf, unsigned __int64 max) Line 531 C
ssh.exe!do_log(LogLevel level, const char * fmt, char * args) Line 462 C
ssh.exe!error(const char * fmt, ...) Line 164 C
ssh.exe!ReadThread(void * lpParameter) Line 96 C
[Inline Frame] ssh.exe!invoke_thread_procedure(unsigned int(*)(void *) context, void * const) Line 91 C++
ssh.exe!thread_start<unsigned int (__cdecl*)(void * __ptr64)>(void * const parameter) Line 115 C++
kernel32.dll!00007ff8e15d7bd4() Unknown
ntdll.dll!RtlUserThreadStart�() Unknown
The main thread was waiting the worker thread:
https://github.com/PowerShell/openssh-portable/blob/v7.7.2.0/contrib/win32/win32compat/termio.c#L264
ntdll.dll!NtWaitForSingleObject�() Unknown
KernelBase.dll!WaitForSingleObjectEx�() Unknown
> ssh.exe!syncio_close(w32_io * pio) Line 266 C
ssh.exe!fileio_close(w32_io * pio) Line 972 C
ssh.exe!w32_close(int fd) Line 611 C
ssh.exe!channel_close_fd(ssh * ssh, int * fdp) Line 428 C
ssh.exe!chan_shutdown_read(ssh * ssh, Channel * c) Line 409 C
ssh.exe!chan_rcvd_oclose(ssh * ssh, Channel * c) Line 292 C
ssh.exe!channel_input_oclose(int type, unsigned int seq, ssh * ssh) Line 3040 C
ssh.exe!ssh_dispatch_run(ssh * ssh, int mode, volatile int * done) Line 114 C
[Inline Frame] ssh.exe!ssh_dispatch_run_fatal(ssh * ssh, int) Line 133 C
[Inline Frame] ssh.exe!client_process_buffered_input_packets() Line 1157 C
ssh.exe!client_loop(ssh * ssh, int) Line 1299 C
ssh.exe!main(int ac, char * * av) Line 1551 C
ssh.exe!wmain(int argc, wchar_t * * wargv) Line 61 C
[Inline Frame] ssh.exe!invoke_main() Line 79 C++
ssh.exe!__scrt_common_main_seh() Line 253 C++
kernel32.dll!00007ff8e15d7bd4() Unknown
ntdll.dll!RtlUserThreadStart�() Unknown
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#2143?email_source=notifications&email_token=ABZH5SBRBFGBRE73D4AVDN3QBQQTLA5CNFSM4HCIQB72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26JB4Y#issuecomment-515674355>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABZH5SE5FW536BEGA7NYGBLQBQQTLANCNFSM4HCIQB7Q>
.
|
|
Hey, this does not reproduce when I use v8.0.0.0p1-Beta. Well, so, this is an issue in OpenSSH for Windows and... fixed in PowerShell/Win32-OpenSSH#1330 |
I had hoped for some confirmation by @joltmode , but hey, at least I have one vote for closing this ticket and none for leaving it open. |
|
@dscho ah, haven't got around to testing it with different versions of OpenSSH. Though, it makes perfect sense to me that the problem relied there and has been fixed. Hence, I'm also cool with this getting closed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Setup
defaults?
Installed with chocolatey's default options.
to the issue you're seeing?
Besides the fact that I'm trying to use Windows integrated Win32-OpenSSH - none that I can think of.
Details
Cmd through Cmder (ConEmu)
Minimal, Complete, and Verifiable example
this will help us understand the issue.
Successful clone, checkout and LFS retrieval.
Checkout happens, but when LFS object retrieval should start, it freezes.
The last ran process:
stays open with no activity:
And it will stay open for hours and nothing changes.
URL to that repository to help us with testing?
It's happening with any repository and any provider (tested with GitLab and GitHub for both clones and pushes) if using SSH remote and the repo has LFS.
I thought that maybe the problem is due to wrapped
sh -ccall.I tested by running raw
git-lfs-authenticatecommands from CMD:C:/Windows/System32/OpenSSH/ssh.exe -- [email protected] 'git-lfs-authenticate mindboiler/rollhill/rollhill.git download'- this returnedGitLab: Disallowed commandC:/Windows/System32/OpenSSH/ssh.exe -- [email protected] "git-lfs-authenticate mindboiler/rollhill/rollhill.git download"- returned the expected token response."C:\Program Files\Git\usr\bin\sh.exe" -c "C:/Windows/System32/OpenSSH/ssh.exe -- [email protected] 'git-lfs-authenticate mindboiler/rollhill/rollhill.git download'"- also returned the expected token response.So the
sh -ccall works as expected, at least when run raw with nogitcontext.There is no difference whether I run this through Cmder (ConEemu) or native CMD, both pose the same issue.
As of finishing this issue post, I have three open and inactive and stalled console windows on
"C:\Program Files\Git\usr\bin\sh.exe" -c "C:/Windows/System32/OpenSSH/ssh.exe -- git@<provider> 'git-lfs-authenticate <repo> download'":The text was updated successfully, but these errors were encountered: