LSP enhancement

[itome]

@DanTup

Hello, I'm using dart lsp for emacs, and writing flutter code.
Dart analysis server returns so large json response for textDocument/completion that emacs freezes while parsing json file for 10~ seconds
Do you have some idea to reduce it?

I checked the json response and noticed that

1. documentation is included to response.

  {
    "textEdit": {
      "newText": "widget${1:}",
      "range": {
        "end": {
          "character": 15,
          "line": 45
        },
        "start": {
          "character": 12,
          "line": 45
        }
      }
    },
    "insertTextFormat": 2,
    "insertText": "widget${1:}",
    "sortText": "999000",
    "documentation": {
      "value": "The current configuration.\n\nA [State] object's configuration is the corresponding [StatefulWidget]\ninstance. This property is initialized by the framework before calling\n[initState]. If the parent updates this location in the tree to a new\nwidget with the same [runtimeType] and [Widget.key] as the current\nconfiguration, the framework will update this property to refer to the new\nwidget and then call [didUpdateWidget], passing the old configuration as\nan argument.",
      "kind": "markdown"
    },
    "detail": "HomePage",
    "kind": 10,
    "label": "widget"
  },

Maybe it must be excluded from response according to this issue -> #37060

2. Set suggestFromUnimportedLibraries = false has no effects for flutter user, beacause import 'package:flutter/material.dart imports almost all source of flutter sdk.

3. Irrelevant completion item seems to be returned. When we request in position below, All classes in flutter sdk is returned. I don't know it is due to lsp server or client.

Align(
  
  ↑ send completion request here.
)

[DanTup]

@itome what version of the Dart/Flutter SDK are you using? The docs are expected to be added in the resolve call. There are some tests to check docs are not included in the original request:

sdk/pkg/analysis_server/test/lsp/completion_test.dart

Lines 326 to 328 in dd5fa9c

	// Expect no docs or text edit, since these are added during resolve.
	expect(completion.documentation, isNull);
	expect(completion.textEdit, isNull);

It's possible I've missed somewhere though. That change should've shipped in Dart v2.4.

Set suggestFromUnimportedLibraries = false has no effects for flutter user, beacause import 'package:flutter/material.dart' imports almost all source of flutter sdk.

It'll have less of an impact, but it should still be significant - there's a lot of unimported code that should still be excluded. I don't think there's much we can do there since they're all valid to use though.

If you can confirm which version of the SDK you're using (and that you're not using a custom version/snapshot of the server from previous testing), I'll do some measuring here.

When we made previous changes, it sounded like this was solved (emacs-lsp/lsp-mode#851 (comment)) - does it seem like it's gotten worse, or was the improvement just not enough?

[itome]

I'm using dart 2.8.0

Dart VM version: 2.8.0-edge.08bd1d6be7ecad43e5921df432efca4d18b6ba41 (Sat Jan 4 19:43:06 2020 +0000) on "linux_x64"

When I checked dart lsp after the previous change, It seems work well, so I thought there were some regression.
Can you check it in the dart 2.8?

[itome]

@DanTup I think split completion response and mark it as isIncomplete works in this case. Does dart analysis lsp already support response splitting or plan to implement it?

[DanTup]

Thanks, I was able to reproduce this - I'm seeing documentation included in the response to textDocument/completion with the latest nightly build.

I don't think isComplete would help unless we truncated the results - it would probably make things worse as the client would request a new completion list on every key stroke, and you'd end up deserialising JSON on each keystroke (whereas normally you'd only do it once per completion).

Hopefully we won't need it if we fix this, I'll figure out what's gone wrong.

[DanTup]

Oh, I see what's going on... The server only has a resolve-like API that we can use to get documentation for the completion items that come via Suggestion Sets. That means some items (which the server returns in-line) will still have documentation attached. I think this should usually only be a small portion though.

Can you confirm what size JSON you're seeing in the project you're testing with? (If you can repro with a public project and attach the file, that would help too). In emacs-lsp/lsp-mode#851 (comment) it says emacs was taking around 200ms to parse 2MB JSON, but it sounds like you're seeing significantly slower than that - is the JSON significantly larger?

[ericdallo]

Any news on that?

[DanTup]

@ericdallo nothing has changed here recently. It's still not clear how much JSON is being transferred when this is occurring. The comment linked above suggests it takes 200ms to parse 2MB of JSON, so it's not clear if it's much slower than that, or if there is considerably more JSON. Are you able to capture the JSON that's taking a long time to parse?

[ericdallo]

Thanks, @DanTup, no I didn't capture, but I updated my Emacs from 26 to 28, and it's much faster. I think It's related to the Emacs 27 feature of native JSON parse.

[itome]

@DanTup Sorry for leaving issue aside. This file is log of lsp. Can you check why the response is so big? (13mb~ response for just one textDocument/completion call)

lsp-log.txt

[DanTup]

Looking at the log, I think it's big because:

1. There are a lot of completions (when including the unimported symbols)
2. There's a lot of unnecessary whitespace/formatting (assuming the attached log is as it came)
3. The "data" that's round-tripped in order to handle the resolve call is quite large (probably over half of the JSON)

I don't think there's much to do about 1, though 2 could be an easy win (I don't know if it'll change the deserialise time much though) and we may need to make changes that could affect 3 anyway based on discussions in Dart-Code/Dart-Code#2290.

[itome]

I think the log is pretty printed. So I don't know whether the whitespace and newline is included in the response or not.

And I noticed that after I send textDocument/completion in the below positon, the response returned from lsp includes many item that doesn't include Si. Is this standard behavior?

      body: ListView(
        children: <Widget>[
          Si
            ^ <- Cursor here.
          const SizedBox(height: 24),

[DanTup]

the response returned from lsp includes many item that doesn't include Si. Is this standard behavior?

I believe that's expected - the full list is sent to the client so that it can be filtered client-side as the user is typing. There's a flag in LSP (isIncomplete=true) that tells the client the list is not complete, so it must go back to the server as the user is typing.

That said, the LSP spec is not explicit about whether the client should call back to the server if IsIncomplete=false and the user presses backspace.. If the intention is that it should, then perhaps these could be filtered on the server. I've opened microsoft/language-server-protocol#954 seeking clarification in the spec.