Frequent crashes loading Hasktorch #1465

tscholak · 2021-03-01T01:06:17Z

This problem is new for me. I just finished updating my Hasktorch dev environment by bumping the version of ghc to 8.10.4. Before I was using hls 0.7 with Hasktorch, now I've tried both 0.9 and 1.0, but both are suddenly very unstable.

Your environment

Output of haskell-language-server --probe-tools or haskell-language-server-wrapper --probe-tools:

haskell-language-server version: 1.0.0.0 (GHC: 8.10.4) (PATH: /nix/store/wsrhqfn07rif6n27qglc3p68kjaflyax-haskell-language-server-exe-haskell-language-server-1.0.0.0/bin/haskell-language-server)
Tool versions found on the $PATH
cabal:		3.2.0.0
stack:		2.5.1
ghc:		8.10.4

Which lsp-client do you use:
vs code

Describe your project (alternative: link to the project):
https://github.com/hasktorch/hasktorch/blob/master/cabal.project

Contents of hie.yaml:
https://github.com/hasktorch/hasktorch/blob/master/hie.yaml

Steps to reproduce

start vscode from nix shell, work for a few seconds on random things, crash occurs

Expected behaviour

no crash

Actual behaviour

crash

Include debug information

Execute in the root of your project the command haskell-language-server --debug . and paste the logs here:

Debug output:

the full output is too long to post here. it finishes after a while with:

Completed (36 files worked, 298 files failed)

Paste the logs from the lsp-client, e.g. for VS Code

LSP logs:

likewise, this log is too long. I'll upload a file...

The text was updated successfully, but these errors were encountered:

konn · 2021-03-01T01:51:09Z

I encountered this issue several times in a closed in-house codebase (without deps to Hasktorch), although I couldn't make minimal repro.
Telling from situational evidence in our project, it seems that modules with type-checker plugins can make HLS 1.0 crash, but not always. There is no concrete evidence though.

EDIT: We use ghc-typelits-natnormalise, ghc-typelits-presburger-knownnat and ghc-typelits-presburger in our code base, where the first two are common with hasktorch. Our crashing modules use -natnormalise and -presburger simultaneously.

tscholak · 2021-03-01T13:08:46Z

Thanks for these pointers.
It would be a surprise to me, though. Have hls 0.9 and 1.0 become less stable under the presence of these plugins? I'm a longtime user of hls with hasktorch, and with 0.7 and ghc 8.10.3 I did not have such severe stability problems.

wz1000 · 2021-03-01T13:16:06Z

GHC 8.10.4 had some changes to the linker which might be responsible. Maybe you could try going back to 8.10.3 with HLS 1.0?

tscholak · 2021-03-01T13:53:46Z

let me try that @wz1000!

wz1000 · 2021-03-01T14:42:06Z

It seems like its segfaulting with GHC 8.10.4, but I can't manage to get a proper backtrace.

tscholak · 2021-03-01T14:43:33Z

at least I know now that I'm not crazy and alone with this issue. did you use the nix environment? stack? cabal?

wz1000 · 2021-03-01T14:48:54Z

I used cabal.

The segfault happens non-deterministically after editing for a few minutes. I can't figure out how to consistently reproduce it. My guess it that some code is not exception safe, probably in a plugin. ghcide uses async exceptions to cancel old compiles when the file is modified.

tscholak · 2021-03-01T14:59:35Z

type checker plugin or hls plugin?

wz1000 · 2021-03-01T15:05:58Z

I meant type checker plugin, but I'm doubting this hypothesis since an uninterruptibleMask_ around the typechecking code didn't seem to help.

The problem is not with a HLS plugin because I can reproduce it with plain ghcide.

On my last attempt, ghcide printed out ghcide: munmap: Invalid argument before dying.

wz1000 · 2021-03-01T15:47:59Z

I think the root cause is this ghc issue: https://gitlab.haskell.org/ghc/ghc/-/issues/19417

wz1000 · 2021-03-01T16:03:26Z

With the debug RTS, ghcide crashes with ghcide: internal error: ASSERTION FAILED: file rts/CheckUnload.c, line 457

wz1000 · 2021-03-01T16:04:36Z

Workaround for now would be to stick GHC 8.10.2 or earlier. (8.10.3 may or may not work)

tscholak · 2021-03-01T18:23:48Z

no luck with 8.10.3, it still crashes over and over again :(

tscholak · 2021-03-01T22:07:19Z

hls 1.0 seems to be stable with 8.10.2 so far

konn · 2021-03-02T02:25:51Z

Sorry for the late reply, and thank you for the detailed survey @wz1000!

The segfault happens non-deterministically after editing for a few minutes.

Hmm. In my environment, it occurs WITHOUT any editing, but just by opening the module alone.

Workaround for now would be to stick GHC 8.10.2 or earlier. (8.10.3 may or may not work)

I tried with GHC 8.10.2, but unfortunately, it still continues to crash... Perhaps, mine issue deserves another issue?

We are using stack.

konn · 2021-03-02T02:30:48Z

How can we use the debug RTS to debug on our codebase?

wz1000 · 2021-03-02T23:52:31Z

@konn you just need to compile your executable (ghcide or HLS) with ghc-options: -debug

konn · 2021-03-03T02:21:05Z

Hmm, I compiled ghcide with -debug and just run without RTS, but no trace is given before SEGV.
It seems there are several debug flags of form -D<x>, which flag was used to get ghcide: internal error: ASSERTION FAILED: file rts/CheckUnload.c?

konn · 2021-03-03T06:19:23Z

I tried with ghcide typecheck path/to/module +RTS -Da by guess, and also get the same ghcide: internal error: ASSERTION FAILED: file rts/CheckUnload.c, line 457 with 8.10.4. I'm also trying with 8.10.2 now.

konn · 2021-03-03T06:55:38Z

Hmm... It seems that on Linux, ghcide fails with 8.10.4 with error ghcide: internal error: ASSERTION FAILED: file rts/CheckUnload.c, line 457 but it succeeds with 8.10.2.

On the other hand, on macOS, it SEGVs also with 8.10.2. Perhaps I encountered the same issue as this issue AND another platform-dependent issue.

konn · 2021-03-03T06:56:39Z

I'll also try with Hasktorch code base on my macOS laptop.

konn · 2021-03-03T08:07:00Z

Tested against our codebase, it seems HLS + 8.10.2 suddenly SEGVs on macOS without any debug trace even with +RTS -Da.

konn · 2021-03-03T08:54:59Z

Agh, it seems Hasktorch won't compile on my environment, due to the C-errors in inline-c-cpp. I will try to use HLS as a testcase instead, as indicated in GitLab issue.

tscholak · 2021-03-03T13:00:35Z

@konn I can help you getting through these pesky C errors.
Alternatively, try the nix shell environment Hasktorch ships with. To try ghc 8.10.2 with hls 1.0, set the compiler version to "ghc8102" here (https://github.com/hasktorch/hasktorch/blob/master/nix/haskell.nix#L8) and launch vscode from within the nix-shell:

$ nix-shell
[nix-shell:~/hasktorch] code .

vs code will then use the hls that the nix shell provides (which is 1.0)
(edit: to speed up compilation, use hasktorch's cachix: cachix use hasktorch)

tscholak · 2021-03-03T13:06:47Z

update: still no stability issues with hls 1.0 and ghc 8.10.2 on hasktorch and macOS.
Of course I'd prefer to use 8.10.4, but at least I'm productive again.

tscholak · 2021-03-03T13:25:54Z

@wz1000 do you know for sure now that this is what needs to be fixed? #1465 (comment)

there is a wip fix here, https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5128/diffs, but I'm not sure when/if this will be released. is this going into 8.10.5?

wz1000 · 2021-03-03T13:31:40Z

Yes, the fix should go in 8.10.5

konn · 2021-03-08T12:26:32Z

@tscholak Thanks, and sorry for the late reply.
I followed macos+stack+cpu instruction, but I added compiler: ghc-8.10.2 to stack.yaml to force stack to use GHC 8.10.2.

Then stack build inline-c-cpp fails with the following C-error:

$ stack build inline-c-cpp
inline-c-cpp> configure
inline-c-cpp> Configuring inline-c-cpp-0.4.0.3...
inline-c-cpp> build
inline-c-cpp> Preprocessing library for inline-c-cpp-0.4.0.3..
inline-c-cpp> Building library for inline-c-cpp-0.4.0.3..
inline-c-cpp> [1 of 2] Compiling Language.C.Inline.Cpp
inline-c-cpp> 
inline-c-cpp> /private/var/folders/pv/mtbzyjyj229g928n710c9d_40000gn/T/stack-d1fe17bdc8914168/inline-c-cpp-0.4.0.3/src/Language/C/Inline/Cpp.hs:11:1: warning: [-Wunused-imports]
inline-c-cpp>     The import of ‘Data.Monoid’ is redundant
inline-c-cpp>       except perhaps to import instances from ‘Data.Monoid’
inline-c-cpp>     To import instances alone, use: import Data.Monoid()
inline-c-cpp>    |
inline-c-cpp> 11 | import           Data.Monoid ((<>), mempty)
inline-c-cpp>    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
inline-c-cpp> [2 of 2] Compiling Language.C.Inline.Cpp.Exceptions
inline-c-cpp> 
inline-c-cpp> /private/var/folders/pv/mtbzyjyj229g928n710c9d_40000gn/T/stack-d1fe17bdc8914168/inline-c-cpp-0.4.0.3/In file included from cxx-src/HaskellException.cxx:1:0: error: 
inline-c-cpp> 
inline-c-cpp> /private/var/folders/pv/mtbzyjyj229g928n710c9d_40000gn/T/stack-d1fe17bdc8914168/inline-c-cpp-0.4.0.3/include/HaskellException.hxx:26:23: error:
inline-c-cpp>      error: exception specification of overriding function is more lax than base version
inline-c-cpp>       virtual const char* what() const noexcept override;
inline-c-cpp>                           ^
inline-c-cpp>    |
inline-c-cpp> 26 |   virtual const char* what() const noexcept override;
inline-c-cpp>    |                       ^
inline-c-cpp> 
inline-c-cpp> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/exception:102:25: error:
inline-c-cpp>      note: overridden virtual function is here
inline-c-cpp>         virtual const char* what() const _NOEXCEPT;
inline-c-cpp>                             ^
inline-c-cpp>     |
inline-c-cpp> 102 |     virtual const char* what() const _NOEXCEPT;
inline-c-cpp>     |                         ^
inline-c-cpp> 
inline-c-cpp> /private/var/folders/pv/mtbzyjyj229g928n710c9d_40000gn/T/stack-d1fe17bdc8914168/inline-c-cpp-0.4.0.3/In file included from cxx-src/HaskellException.cxx:1:0: error: 
inline-c-cpp> 
inline-c-cpp> /private/var/folders/pv/mtbzyjyj229g928n710c9d_40000gn/T/stack-d1fe17bdc8914168/inline-c-cpp-0.4.0.3/include/HaskellException.hxx:26:35: error:
inline-c-cpp>      error: expected ';' at end of declaration list
inline-c-cpp>       virtual const char* what() const noexcept override;
inline-c-cpp>                                       ^
inline-c-cpp>                                       ;
inline-c-cpp>    |
inline-c-cpp> 26 |   virtual const char* what() const noexcept override;
inline-c-cpp>    |                                   ^
inline-c-cpp> 
inline-c-cpp> /private/var/folders/pv/mtbzyjyj229g928n710c9d_40000gn/T/stack-d1fe17bdc8914168/inline-c-cpp-0.4.0.3/include/HaskellException.hxx:19:7: error:
inline-c-cpp>      error: exception specification of overriding function is more lax than base version
inline-c-cpp>    |
inline-c-cpp> 19 | class HaskellException : public std::exception {
inline-c-cpp>    |       ^
inline-c-cpp> class HaskellException : public std::exception {
inline-c-cpp>       ^
inline-c-cpp> 
inline-c-cpp> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/exception:101:13: error:
inline-c-cpp>      note: overridden virtual function is here
inline-c-cpp>         virtual ~exception() _NOEXCEPT;
inline-c-cpp>                 ^
inline-c-cpp>     |
inline-c-cpp> 101 |     virtual ~exception() _NOEXCEPT;
inline-c-cpp>     |             ^
inline-c-cpp> 
inline-c-cpp> /private/var/folders/pv/mtbzyjyj229g928n710c9d_40000gn/T/stack-d1fe17bdc8914168/inline-c-cpp-0.4.0.3/cxx-src/HaskellException.cxx:31:44: error:
inline-c-cpp>      error: expected function body after function declarator
inline-c-cpp>    |
inline-c-cpp> 31 | const char* HaskellException::what() const noexcept {
inline-c-cpp>    |                                            ^
inline-c-cpp> const char* HaskellException::what() const noexcept {
inline-c-cpp>                                            ^
inline-c-cpp> 4 errors generated.

Environment: macOS Catalina 10.15.7

tscholak · 2021-03-24T12:19:25Z

@konn sorry I never followed up with you. It's not as easy to diagnose as I thought. Stack pulls its dependencies from Stackage independently of nix. It's possible that the version Stack uses is not working with ghc 8.10.2.

For those using Hasktorch with Cabal and nix, I can confirm that input-output-hk/haskell.nix@1abbd16 has resolved the issue originally described in this issue. I can use ghc 8.10.4 with hls again without problem. The necessary updates to hasktorch are in this branch, https://github.com/hasktorch/hasktorch/tree/mcwitt-update-deps, and I hope we can merge them soon as part of hasktorch/hasktorch#533.

jneira · 2021-03-24T15:28:29Z

thanks for update the issue with the workaround for 8.10.4 until ghc-8.10.5 is released
I would keep open until that

Anton-Latukha · 2021-12-25T00:56:44Z

All prerequisites in this thread seem to be resolved (GHC reports, PRs (even backports are done) & releases made).

So, seems no longer depends on upstream.

It is still applies, or can be closed?

tscholak · 2021-12-25T14:24:56Z

Hi @Anton-Latukha, thanks 🙏 this can be closed 😀

konn added the type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc.. label Mar 1, 2021

jneira added the status: needs repro label Mar 1, 2021

jneira changed the title ~~frequent crashes~~ Frequent crashes loading Hasktorch Mar 1, 2021

jneira added the type: setup label Mar 1, 2021

wz1000 removed the status: needs repro label Mar 1, 2021

tscholak mentioned this issue Mar 3, 2021

request for a stability bug hotfix in ghc 8.10.{3,4} input-output-hk/haskell.nix#1062

Closed

jneira added status: blocked Not actionable, because blocked by upstream/GHC etc. can-workaround labels Mar 24, 2021

konn mentioned this issue May 24, 2021

Remove support for ghc-8.8.2 #1863

Closed

Ailrun mentioned this issue Jun 6, 2021

Add GHC 8.10.5 support #1899

Merged

tscholak closed this as completed Dec 25, 2021

Uh oh!

Frequent crashes loading Hasktorch #1465

Frequent crashes loading Hasktorch #1465

Comments

tscholak commented Mar 1, 2021

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Include debug information

konn commented Mar 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tscholak commented Mar 1, 2021

Uh oh!

wz1000 commented Mar 1, 2021

Uh oh!

tscholak commented Mar 1, 2021

Uh oh!

wz1000 commented Mar 1, 2021

Uh oh!

tscholak commented Mar 1, 2021

Uh oh!

wz1000 commented Mar 1, 2021

Uh oh!

tscholak commented Mar 1, 2021

Uh oh!

wz1000 commented Mar 1, 2021

Uh oh!

wz1000 commented Mar 1, 2021

Uh oh!

wz1000 commented Mar 1, 2021

Uh oh!

wz1000 commented Mar 1, 2021

Uh oh!

tscholak commented Mar 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tscholak commented Mar 1, 2021

Uh oh!

konn commented Mar 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

konn commented Mar 2, 2021

Uh oh!

wz1000 commented Mar 2, 2021

Uh oh!

konn commented Mar 3, 2021

Uh oh!

konn commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

konn commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

konn commented Mar 3, 2021

Uh oh!

konn commented Mar 3, 2021

Uh oh!

konn commented Mar 3, 2021

Uh oh!

tscholak commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tscholak commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tscholak commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wz1000 commented Mar 3, 2021

Uh oh!

konn commented Mar 8, 2021

Uh oh!

tscholak commented Mar 24, 2021

Uh oh!

jneira commented Mar 24, 2021

Uh oh!

Anton-Latukha commented Dec 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

konn commented Mar 1, 2021 •

edited

Loading

tscholak commented Mar 1, 2021 •

edited

Loading

konn commented Mar 2, 2021 •

edited

Loading

konn commented Mar 3, 2021 •

edited

Loading

konn commented Mar 3, 2021 •

edited

Loading

tscholak commented Mar 3, 2021 •

edited

Loading

tscholak commented Mar 3, 2021 •

edited

Loading

tscholak commented Mar 3, 2021 •

edited

Loading

Anton-Latukha commented Dec 25, 2021 •

edited

Loading