Analysis server hangs with Linux 5.5 and IntelliJ based IDEs

[simolus3]

After upgrading to Linux 5.5 (I'm on Linux 5.5.2-1-MANJARO), analysis in IntelliJ stops working after a while (analysis errors view doesn't update anymore, formatting times out, no autocomplete, ...).
The analyzer diagnostics page is still reachable after that happens, but it appears that the analysis server just stops serving regular requests. Interestingly, I can't reproduce this with VS Code. I tried deleting ~/.dartServer/, but that didn't fix the problem.

I first thought this was an IntelliJ problem, but flutter/flutter#49185 (comment) makes me think that this is analyzer related. I wanted to open another issue here because for me that also happens on the non-Flutter Dart SDK (both stable & latest dev).

I can reproduce this consistently and on different projects, so I'd be glad to provide more information if that's necessary.

[marcusaram]

Dart also hangs while compiling Flutter project. Dart process was stuck while generating kernel snapshot. Running on 5.4 LTS kernel everything works again....

[simolus3]

Yeah this is probably more general than just the analyzer. pub lish on 5.5 never shows the "Looks great! Are you ready to upload your package" message - on 5.4 it happens right away.

[mraleph]

Based on the symptoms this is Dart VM related - potentially a dead lock in the VM or a bug in dart:io implementation.

If you could attach GDB and provide stack traces for all threads that would help before we get access to a machine with new Kernel which might take some time.

/cc @rmacnak-google @mkustermann @a-siva

[simolus3]

I'm on Dart VM version: 2.7.1 (Thu Jan 23 13:02:26 2020 +0100) on "linux_x64".

Backtrace when the analysis server hangs

Thread 4 (Thread 0x7f2994e7e700 (LWP 4240)):
#0  0x00007f29980c2c45 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055f0e9b4f428 in dart::Monitor::WaitMicros(long) ()
#2  0x000055f0e9d2c4ca in dart::BackgroundCompiler::Run() ()
#3  0x000055f0e9bce157 in dart::ThreadPool::Worker::Loop() ()
#4  0x000055f0e9bcdfde in dart::ThreadPool::Worker::Main(unsigned long) ()
#5  0x000055f0e9b4ed19 in ?? ()
#6  0x00007f29980bc4cf in start_thread () from /usr/lib/libpthread.so.0
#7  0x00007f299831a2d3 in clone () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7f29968ff700 (LWP 4235)):
#0  0x00007f29980c2c45 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055f0e9b4f428 in dart::Monitor::WaitMicros(long) ()
#2  0x000055f0e9d2c4ca in dart::BackgroundCompiler::Run() ()
#3  0x000055f0e9bce157 in dart::ThreadPool::Worker::Loop() ()
#4  0x000055f0e9bcdfde in dart::ThreadPool::Worker::Main(unsigned long) ()
#5  0x000055f0e9b4ed19 in ?? ()
#6  0x00007f29980bc4cf in start_thread () from /usr/lib/libpthread.so.0
#7  0x00007f299831a2d3 in clone () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7f2996e68700 (LWP 4232)):
#0  0x00007f299831a60e in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055f0e986ecd5 in dart::bin::EventHandlerImplementation::Poll(unsigned long) ()
#2  0x000055f0e98b8245 in ?? ()
#3  0x00007f29980bc4cf in start_thread () from /usr/lib/libpthread.so.0
#4  0x00007f299831a2d3 in clone () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7f29980acc00 (LWP 4229)):
#0  0x00007f29980c2c45 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055f0e9b4f428 in dart::Monitor::WaitMicros(long) ()
#2  0x000055f0e9fb6d8a in Dart_RunLoop ()
#3  0x000055f0e986b48d in dart::bin::RunMainIsolate(char const*, dart::bin::CommandLineOptions*) ()
#4  0x000055f0e986c1d2 in dart::bin::main(int, char**) ()
#5  0x000055f0e986ccf9 in main ()

Backtrace when pub lish hangs:

Thread 4 (Thread 0x7ff74117f700 (LWP 2981)):
#0  0x00007ff74449c6f7 in wait () from /usr/lib/libc.so.6
#1  0x00005633955a8358 in ?? ()
#2  0x00005633955eb245 in ?? ()
#3  0x00007ff7442714cf in start_thread () from /usr/lib/libpthread.so.0
#4  0x00007ff7444cf2d3 in clone () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7ff741aff700 (LWP 2976)):
#0  0x00007ff744277c45 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x0000563395882428 in dart::Monitor::WaitMicros(long) ()
#2  0x0000563395a5f4ca in dart::BackgroundCompiler::Run() ()
#3  0x0000563395901157 in dart::ThreadPool::Worker::Loop() ()
#4  0x0000563395900fde in dart::ThreadPool::Worker::Main(unsigned long) ()
#5  0x0000563395881d19 in ?? ()
#6  0x00007ff7442714cf in start_thread () from /usr/lib/libpthread.so.0
#7  0x00007ff7444cf2d3 in clone () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7ff743557700 (LWP 2974)):
#0  0x00007ff7444cf60e in epoll_wait () from /usr/lib/libc.so.6
#1  0x00005633955a1cd5 in dart::bin::EventHandlerImplementation::Poll(unsigned long) ()
#2  0x00005633955eb245 in ?? ()
#3  0x00007ff7442714cf in start_thread () from /usr/lib/libpthread.so.0
#4  0x00007ff7444cf2d3 in clone () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7ff744261c00 (LWP 2971)):
#0  0x00007ff744277c45 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x0000563395882428 in dart::Monitor::WaitMicros(long) ()
#2  0x0000563395ce9d8a in Dart_RunLoop ()
#3  0x000056339559e48d in dart::bin::RunMainIsolate(char const*, dart::bin::CommandLineOptions*) ()
#4  0x000056339559f1d2 in dart::bin::main(int, char**) ()
#5  0x000056339559fcf9 in main ()

[aam]

@simolus3 The fact that it only affects IJ, not VSCode perhaps points to the issue with the plugin itself?

@DanTup @devoncarew might suggest how to troubleshoot the plugin, enable logging of the interaction between IJ plugin and analysis server

[marcusaram]

@simolus3 The fact that it only affects IJ, not VSCode perhaps points to the issue with the plugin itself?

@DanTup @devoncarew might suggest how to troubleshoot the plugin, enable logging of the interaction between IJ plugin and analysis server

Mine was stuck at gradle build, nothing to do with IntelliJ. This was on a relative big Flutter project. Small empty flutter project compiles just fine. But when switching kernel to 5.4 there are no issues...

Will see if I can add GDB thread dump soon

[a-siva]

The stack trace posted above does not seem to indicate any kind of locking deadlock in the VM, it is possible some IO event was missed and the event handler is not waking up indicating a potential issue with dart:io.
It is going to be hard to debug using the stack traces above, I guess we would have to look for access to a machine with Linux 5.5 to investigate this issue.

[mraleph]

I spinned up a cloud instance with 5.5.3 kernel. Going to run tests to see if it reproduces the issue.

[rmacnak-google]

@zanderso

[zanderso]

This appears to be affecting Flutter CLI users: flutter/flutter#49185