Fetch full documentation in code completion

[a7medev]

Depends on swiftlang/swift#82464

---

When resolving documentation for code completion items, we fetch full documentation through the newly added swiftide_completion_item_get_doc_full_copy SourceKitD function, if not found we fallback to brief documentation as before using swiftide_completion_item_get_doc_brief.

Note

Unlike brief documentation, SourceKitD doesn't cache full documentation for completion results to avoid bloating memory with a lot of large strings.

As of now, SourceKit-LSP doesn't cache completion item documentation either, should we introduce a new full documentation cache (e.g. using LRUCache)?

[ahoppen]

Very cool stuff. Excited to see this. I left a couple comments inline.

[ahoppen]

Thanks, looks good to me, the only open question is whether we want to return both the brief and the full documentation from the plugin.

Also: Could you format your changes using swift-format as described in https://github.com/swiftlang/sourcekit-lsp/blob/main/CONTRIBUTING.md#formatting?

[a7medev]

I have added one more functionality change.
In the initial implementation, I converted XML documentation to markdown using the same existing flow which includes the declaration as part of the documentation.

I found that existing LSP implementations (mainly Clang and TypeScript) don't show the declaration itself as part of the full documentation in completion (contrary to the hover request!) and I think the reason for this is that the completion item already describes the declaration and typically the popup showing the documentation is small so it makes sense to show the documentation comment right away.

@ahoppen @hamishknight Can you please recheck? 🙏🏼

[ahoppen]

Looks great to me. Thank you 🚀

[ahoppen]

@swift-ci Please test

[ahoppen]

CI failed because Swift CI unconditionally enables all tests that are guarded by skipUnlessSupportedByToolchain (we don’t want to accidentally leave tests disabled in CI due to configuration issues). I suggest that we wait until swiftlang/swift#82464 is merged before merging this. Is that alright with you?

[a7medev]

@ahoppen Yes, I think that makes sense.

I wonder where in the console output did you find this though 😅

Edit: I found it by downloading the plain text output and using grep to filter it out. Let me know if there's a more efficient way to do that though 😅

[ahoppen]

No, I always download the log files as well.

[bnbarham]

Woo! This is awesome, thanks @a7medev!

As of now, SourceKit-LSP doesn't cache completion item documentation either, should we introduce a new full documentation cache (e.g. using LRUCache)?

It's probably worth merging this, seeing how it performs, and only adding the cache if we need it. If we do then it'll be a little more than just caching them as they're retrieved, since if that's too slow then getting them in the first place is as well 😅 (in which case we might need to pre-fetch a bunch).

[a7medev]

We've found that the XML to markdown implementation has some problems.

Consider this example:

/// Adds two integers together
/// A really interesting function, no?
///
/// Usage:
/// ```swift
/// add(x: 14, y: 22)
/// ```
///
/// - Parameters:
///   - x: First arg
///   - y: Second arg
/// - Returns: Sum
func add(x: Int, y: Int) -> Int {
    return x + y
}

The full documentation for add is shown like this in VS Code:

this has some issues:

1. Code blocks have line numbers and extra spacing before each line (prepended as part of the code to each line which can also cause issues with syntax highlighting) and there’s always an empty numbered line at the end which makes it even weirder. These line numbers are being added since the full XML output adds code between <zCodeLineNumbered> tags. When I checked Xcode’s documentation output, it didn’t actually include any line numbers at all, which I believe looks a lot nicer.
2. Section headings are a bit large considering the area reserved for documentation output.
3. The current XML to markdown implementation is incomplete (see Bullets in doc comments rendered as "Discussion," labels are dropped. #1014) which led to switching to the raw comment for the hover LSP request in Emit raw documentation comments instead of parsing XML #1091.

If we just output the raw comment instead, it looks like this. Which looks a lot nicer.

The downside of course is losing the structured format that the XML-based comment has so we should ultimately switch back to using the XML to markdown approach after improving it.

I agreed with @hamishknight on sticking to the raw comment for now to follow what hover is doing and to not block this on improving XML to markdown output.

@hamishknight @ahoppen @bnbarham Can you please re-review the PR after these changes? 🙏🏼

[ahoppen]

I agree that we should render the raw Markdown instead of converting Markdown to XML only to convert it back to Markdown for the same reason as I made the change in #1091.

[ahoppen]

Looks great. Excited to use this 🚀

[ahoppen]

Shouldn’t this case pass even if sourcekitd doesn’t support full documentation in completion? You didn’t actually change the behavior or am I missing something.

[a7medev]

It would pass since the brief documentation is equal to the full documentation in this test case but I don't think this is something we need to maintain since we can change the formatting for full/brief documentation or update the doc comment in the test for some reason which would in turn break the test if we didn't have full documentation support in sourcekitd.

[ahoppen]

Oh, I see. Makes sense. Thanks for explaining. 👍🏽