fixes to support clangd: LSP Extension Deserialization & Document Symbol Handling

[Shamazo]

Hi,

First of all great project. I'm integrating it with clangd for C++ development and have encountered a couple of issues related to LSP extensions and document handling. I've investigated these and have potential solutions

Issue 1: Deserialization Failure with clangd LSP Extensions

Clangd uses some lsp extensions. These are incompatible with the LSP types/parsers generated by cmd/generate because the parsers use DisallowUnknownFields().

What broke for me, is the clangd additionally returns a "score" for symbols. See clangd discussion here and clangd commit here

This results in a deserialization issue when handling workspace/symbol responses.

logs

``` { "error": "Server stderr: 2025/05/07 13:46:01.653864 [DEBUG][core] Executing definition for symbol: prepare34_adaptive_shared_ht\n2025/05/07 13:46:01.653893 [DEBUG][lsp] Making call: method=workspace/symbol id=3\n2025/05/07 13:46:01.653987 [DEBUG][lsp] Sending message: method=workspace/symbol id=3\n2025/05/07 13:46:01.653997 [DEBUG][wire] -> Sending: {\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"workspace/symbol\",\"params\":{\"query\":\"prepare34_adaptive_shared_ht\",\"workDoneToken\":null}}\n2025/05/07 13:46:01.654039 [DEBUG][lsp] Waiting for response to request ID: 3\n2025/05/07 13:46:01.654289 [INFO][lsp-process] I[13:46:01.654] <-- workspace/symbol(3)\n2025/05/07 13:46:01.655923 [DEBUG][wire] <- Header: Content-Length: 324\n2025/05/07 13:46:01.655969 [DEBUG][wire] <- Received: {\"id\":3,\"jsonrpc\":\"2.0\",\"result\":[{\"containerName\":\"\",\"kind\":12,\"location\":{\"range\":{\"end\":{\"character\":46,\"line\":214},\"start\":{\"character\":18,\"line\":214}},\"uri\":\"file:///tmp/tmp.YD2SgUVlV5/apps/standalones/adm/adm-play/prepared_queries/q3_4_adaptive.cpp\"},\"name\":\"prepare34_adaptive_shared_ht\",\"score\":1.1000000238418579}]}\n2025/05/07 13:46:01.655978 [INFO][lsp-process] I[13:46:01.655] --> reply:workspace/symbol(3) 1 ms\n2025/05/07 13:46:01.656021 [DEBUG][lsp] Received response for ID: 3\n2025/05/07 13:46:01.656035 [DEBUG][lsp] Sending response for ID 3 to handler\n2025/05/07 13:46:01.656059 [DEBUG][lsp] Received response for request ID: 3\n2025/05/07 13:46:01.656298 [ERROR][lsp] Failed to unmarshal result: unmarshal failed to match one of [[]SymbolInformation []WorkspaceSymbol]\n2025/05/07 13:46:01.656325 [ERROR][core] Failed to get definition: failed to fetch symbol: failed to unmarshal result: unmarshal failed to match one of [[]SymbolInformation []WorkspaceSymbol]", "timestamp": "2025-05-07T11:46:01.705Z", "sessionId": "48160c35-0327-47da-bc96-c78e7a708e7b", "cwd": "/tmp/tmp.YD2SgUVlV5" }, { "error": "Error calling tool definition: undefined", "timestamp": "2025-05-07T11:46:01.719Z", "sessionId": "48160c35-0327-47da-bc96-c78e7a708e7b", "cwd": "/tmp/tmp.YD2SgUVlV5" } ```

I first tried to just remove the DisallowUnknownFields() on struct parsers, but that broke the Rust integration tests. A better approach is to add an optional field to BaseSymbolInformation:

type BaseSymbolInformation struct {
	// The name of this symbol.
	Name string `json:"name"`
	// The kind of this symbol.
	Kind SymbolKind `json:"kind"`
	// Tags for this symbol.
	//
	// @since 3.16.0
	Tags []SymbolTag `json:"tags,omitempty"`
	// The name of the symbol containing this symbol. This information is for
	// user interface purposes (e.g. to render a qualifier in the user interface
	// if necessary). It can't be used to re-infer a hierarchy for the document
	// symbols.
	ContainerName string `json:"containerName,omitempty"`
	// The score that clangd calculates to rank the returned symbols.
        // This is a clangd extension, set only for workspace/symbol responses.
	Score float64 `json:"score,omitempty"`
}

For now I have hard coded this in the generated go files. I tried for quite a while to modify the generating code, but could not figure out how to generate this optional field. If anyone could suggest how to modify the generating code in cmd/generate, I would be grateful.

Issue 2: textDocument/documentSymbol Called on Non-Open File

The definition tool sequence is:

1. Call workspace/symbol to find the file containing a symbol.
2. Immediately call textDocument/documentSymbol on the located file URI.

This fails if the document is not already open in the LSP server, as clangd (and likely other servers) expects a document to be explicitly opened via textDocument/didOpen before operations like documentSymbol can be performed on it.

logs

``` { "error": "Server stderr: 2025/05/07 15:55:14.682726 [DEBUG][core] Executing definition for symbol: prepare34_adaptive_shared_ht\n2025/05/07 15:55:14.682752 [DEBUG][lsp] Making call: method=workspace/symbol id=3\n2025/05/07 15:55:14.682791 [DEBUG][lsp] Sending message: method=workspace/symbol id=3\n2025/05/07 15:55:14.682814 [DEBUG][wire] -> Sending: {\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"workspace/symbol\",\"params\":{\"query\":\"prepare34_adaptive_shared_ht\",\"workDoneToken\":null}}\n2025/05/07 15:55:14.682845 [DEBUG][lsp] Waiting for response to request ID: 3\n2025/05/07 15:55:14.683126 [INFO][lsp-process] I[15:55:14.682] <-- workspace/symbol(3)\n2025/05/07 15:55:14.683780 [DEBUG][wire] <- Header: Content-Length: 324\n2025/05/07 15:55:14.683817 [INFO][lsp-process] I[15:55:14.683] --> reply:workspace/symbol(3) 0 ms\n2025/05/07 15:55:14.683820 [DEBUG][wire] <- Received: {\"id\":3,\"jsonrpc\":\"2.0\",\"result\":[{\"containerName\":\"\",\"kind\":12,\"location\":{\"range\":{\"end\":{\"character\":46,\"line\":214},\"start\":{\"character\":18,\"line\":214}},\"uri\":\"file:///tmp/tmp.YD2SgUVlV5/apps/standalones/adm/adm-play/prepared_queries/q3_4_adaptive.cpp\"},\"name\":\"prepare34_adaptive_shared_ht\",\"score\":1.1000000238418579}]}\n2025/05/07 15:55:14.683941 [DEBUG][lsp] Received response for ID: 3\n2025/05/07 15:55:14.683953 [DEBUG][lsp] Sending response for ID 3 to handler\n2025/05/07 15:55:14.683975 [DEBUG][lsp] Received response for request ID: 3\n2025/05/07 15:55:14.684123 [DEBUG][tools] Found symbol: prepare34_adaptive_shared_ht\n2025/05/07 15:55:14.684134 [DEBUG][lsp] Making call: method=textDocument/documentSymbol id=4\n2025/05/07 15:55:14.684196 [DEBUG][lsp] Sending message: method=textDocument/documentSymbol id=4\n2025/05/07 15:55:14.684205 [DEBUG][wire] -> Sending: {\"jsonrpc\":\"2.0\",\"id\":4,\"method\":\"textDocument/documentSymbol\",\"params\":{\"textDocument\":{\"uri\":\"file:///tmp/tmp.YD2SgUVlV5/apps/standalones/adm/adm-play/prepared_queries/q3_4_adaptive.cpp\"},\"workDoneToken\":null}}\n2025/05/07 15:55:14.684230 [DEBUG][lsp] Waiting for response to request ID: 4\n2025/05/07 15:55:14.684452 [INFO][lsp-process] I[15:55:14.684] <-- textDocument/documentSymbol(4)\n2025/05/07 15:55:14.684472 [INFO][lsp-process] I[15:55:14.684] --> reply:textDocument/documentSymbol(4) 0 ms, error: -32602: trying to get AST for non-added document\n2025/05/07 15:55:14.684490 [DEBUG][wire] <- Header: Content-Length: 101\n2025/05/07 15:55:14.684505 [DEBUG][wire] <- Received: {\"error\":{\"code\":-32602,\"message\":\"trying to get AST for non-added document\"},\"id\":4,\"jsonrpc\":\"2.0\"}\n2025/05/07 15:55:14.684590 [DEBUG][lsp] Received response for ID: 4\n2025/05/07 15:55:14.684599 [DEBUG][lsp] Sending response for ID 4 to handler\n2025/05/07 15:55:14.684617 [DEBUG][lsp] Received response for request ID: 4\n2025/05/07 15:55:14.684625 [ERROR][lsp] Request failed: trying to get AST for non-added document (code: -32602)\n2025/05/07 15:55:14.684642 [ERROR][tools] Error getting definition: failed to get document symbols: request failed: trying to get AST for non-added document (code: -32602)", "timestamp": "2025-05-07T13:55:14.735Z", "sessionId": "b1d91696-3b78-4c18-9e35-f4735a534600", "cwd": "/tmp/tmp.YD2SgUVlV5" } ```

This is an easy fix, just a call to client.OpenFile.

Next Steps

I haven't yet tested other tools beyond definition and diagnostics.

I'd like to prepare a Pull Request with these fixes and the new clangd integration tests (modelled on the Rust tests). However, the primary blocker is correctly generating the optional score field for Issue 1. I have a dev fork with my changes so far here

Any help with the cmd/generate modification would be fantastic!

Thanks!

[isaacphi]

@Shamazo Thanks for this! Can you take a look at #28?

I think removing DisallowUnknownFields() might be a better fix but I can't figure out why it's breaking rust. The document symbols come back without location information when you do this... weird. Anyway, I put in a way to toss extra properties in during code generation.

I'd really appreciate any feedback and testing for clangd because I don't use C++ so feel free to open a PR!

[Shamazo]

Thanks for opening the PR with the fixes. I tested it and it works for me locally. I also couldn't figure out how/why rust broke.

I'll work on a PR for some clangd integration tests

[Shamazo]

@isaacphi Opened #29 with initial clangd integration tests