[Track] KCL IDE/LSP workspace design

[He1pa]

KCL IDE/LSP workspace

Main Problem

KCL's CLI specifies the compilation entry by inputting a file, files list or a configuration file by argument -Y
But IDE and LSP dont know what to compile and analyze

Cases

Now we are facing several cases

1. Without configuration file

-- pkg
  -- a.k
  -- b.k

2. kcl.yaml

This pattern appears in the internal, closed-source konfig repository

-- kcl.mod (kcl v0.4.3, it is not the `kcl.mod` in current kpm)
-- base
  -- pkg1
    -- a.k
  -- pkg2
-- prog1
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- project.yaml
-- prog2
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
   -- project.yaml

kcl.yaml

kcl_cli_configs:
  file:
    - base/pkg1.a.k
    - ../base/base.k
    - main.k

Compile with kcl run -Y kcl.yaml

3.kcl.mod

-- pkg
  -- a.k
-- main.k
-- kcl.mod

kcl.mod

[package]
name = "xxx"
version = "0.1.0"

[dependencies]
xx = { .. }
[profile]
entries = ["main.k"]

Current processing logic

• Search for kcl.mod or kcl.yaml in the current directory. If it exists, compile according to this configuration
• If not, use the current directory as the entry(KCL requires that definitions within a package be visible to each other)

Problem

For case 2 and 3, IDE/LSP will only get the correct project structure when processing main.k. But when processing pkg/a.k, it will be treated as case 1, and pkg will be used as the entry for compile. At this time, the lack of information about kcl.mod and main.k will lead to incomplete semantic information, and features such as find_ref and rename will be unavailable.

-- pkg
  -- a.k   -> GlobalState 1
-- main.k  
-- kcl.mod -> GlobalState 2

In addition, when we compile with kcl.mod as the entry, and then open pkg/a.k.Aactually, it has been compiled and analyzed. But according to the current logic, it will be recompiled, which will also cause performance loss in IDE.

In a workspace, there are multiple entries at the same time. In order to save the status of multiple entries at the same time, LSP will also have performance loss.
#1237

## Solution
LSP set two modes: SingleEntry and MultiEntry

• SingleEntry: For a standard kpm package(case 3), define kcl.mod in the root directory of the package and use it as the compilation entry and config
• MultiEntry:

Try to find kcl.mod in the IDE workspace

• There is kcl.mod, does kcl.mod have an entries field?
  • There is an entries field, use the entries field as the compilation entry
  • There is no entries field, does the workspace have main.k?
    • There is main.k as the compilation entry
    • No main.k, does the workspace have other kcl files?
      • Yes, use the workspace as the compilation entry
      • No, the same logic as before
• if not, does the workspace have main.k?
  • There is main.k as the compilation entry
  • No main.k, does the workspace have other kcl files?
    • Yes, use the workspace as the compilation entry
    • No, the same logic as before

The logic is consistent with previous to be compatible with the konfig repository. New KCL projects are recommended to use kcl.mod

### LSP Changes

Previously, opened/changed a file -> found compile entry -> compile -> update GlobalState

After the change, Start LSP -> found compile entry -> compile -> update GlobalState -> update the GlobalState cache according to the changed file

### Impact on lsp cache/incremental compilation

In the previous design, no matter what the case, the changed files are in the main pkg, and the incremental compilation of resolver/advanced_resolver is based on the premise that the changed files are in the main pkg.

After changing the entry point, it is necessary to reconsider the processing of editing files that are not in the main pkg.

For the Parse stage:
The cache is a map of filename -> AST. There is no pkg, so no need to update

For the Resolver stage:
Previously, the resolver needed to clear the cache of main_pkg and the affected pkgs. After the change, it is necessary to first query which pkg the file belongs to, analyze the affected pkgs, and clear the cache of these pkgs.

For Namer and Advanced Resolver:
Update the pkg that was updated in the resolver stage, no changes are required

KCL/KPM run

KCL/KPM run also has similar logic when executing, and needs to find the compilation entry and config from the input. The relevant implementation is in
https://github.com/kcl-lang/kpm/blob/2c16041e969cd2f7cdaf3af4e4d0cf7d96c67003/pkg/client/run.go#L409

// getPkgSource returns the package source.
// Compiling multiple packages at the same time will cause an error.
func (o *RunOptions) getPkgSource() (*downloader.Source, error) {
	workDir := o.WorkDir

	var pkgSource *downloader.Source
	if len(o.Sources) == 0 {
		workDir, err := filepath.Abs(workDir)
		if err != nil {
			return nil, err
		}
		// If no sources set by options, return a localSource to facilitate searching for
		// compilation entries from configurations such as kcl.yaml and kcl.mod files.
		return &downloader.Source{
			Local: &downloader.Local{
				Path: workDir,
			},
		}, nil
	} else {
		var rootPath string
		var err error
		for _, source := range o.Sources {
			if pkgSource == nil {
				pkgSource = source
				rootPath, err = source.FindRootPath()
				if err != nil {
					return nil, err
				}
			} else {
				rootSourceStr, err := pkgSource.ToString()
				if err != nil {
					return nil, err
				}

				sourceStr, err := source.ToString()
				if err != nil {
					return nil, err
				}

				if pkgSource.IsPackaged() || source.IsPackaged() {
					return nil, reporter.NewErrorEvent(
						reporter.CompileFailed,
						fmt.Errorf("cannot compile multiple packages %s at the same time", []string{rootSourceStr, sourceStr}),
						"only allows one package to be compiled at a time",
					)
				}

				if !pkgSource.IsPackaged() && !source.IsPackaged() {
					tmpRootPath, err := source.FindRootPath()
					if err != nil {
						return nil, err
					}
					if tmpRootPath != rootPath {
						return nil, reporter.NewErrorEvent(
							reporter.CompileFailed,
							fmt.Errorf("cannot compile multiple packages %s at the same time", []string{tmpRootPath, rootPath}),
							"only allows one package to be compiled at a time",
						)
					}
				}
			}
		}

		// A local k file or folder
		if pkgSource.IsLocalKPath() || pkgSource.IsDir() {
			rootPath, err = pkgSource.FindRootPath()
			if err != nil {
				return nil, err
			}

			pkgSource, err = downloader.NewSourceFromStr(rootPath)
			if err != nil {
				return nil, err
			}
		}
	}

	if pkgSource == nil {
		return nil, errors.New("no source provided")
	}

	return pkgSource, nil
}

and

• // if local.Path is a directory, judge if it has kcl.mod file
• // if local.Path is a *.k file, find the kcl.mod file in the same directory and in the parent directory

func (local *Local) FindRootPath() (string, error) {
	if local == nil {
		return "", fmt.Errorf("local source is nil")
	}

	// if local.Path is a directory, judge if it has kcl.mod file
	if utils.DirExists(filepath.Join(local.Path, constants.KCL_MOD)) {
		abspath, err := filepath.Abs(local.Path)
		if err != nil {
			return "", err
		}
		return abspath, nil
	}

	// if local.Path is a *.k file, find the kcl.mod file in the same directory and in the parent directory

	dir := filepath.Dir(local.Path)
	for {
		kclModPath := filepath.Join(dir, constants.KCL_MOD)
		if utils.DirExists(kclModPath) {
			abspath, err := filepath.Abs(kclModPath)
			if err != nil {
				return "", err
			}
			return filepath.Dir(abspath), nil
		}

		parentDir := filepath.Dir(dir)
		if parentDir == dir {
			break
		}
		dir = parentDir
	}

	// If no kcl.mod file is found, return the directory of the original file
	var abspath string
	var err error
	if local.IsLocalKPath() {
		abspath, err = filepath.Abs(filepath.Dir(local.Path))
		if err != nil {
			return "", err
		}
	} else {
		abspath, err = filepath.Abs(local.Path)
		if err != nil {
			return "", err
		}
	}

	return abspath, nil
}

There are two kinds of cases related to IDE

1. workDir and IDE workspace. If there is no input, like

# Run the current package
kcl run

	if len(o.Sources) == 0 {
		workDir, err := filepath.Abs(workDir)
		if err != nil {
			return nil, err
		}
		// If no sources set by options, return a localSource to facilitate searching for
		// compilation entries from configurations such as kcl.yaml and kcl.mod files.
		return &downloader.Source{
			Local: &downloader.Local{
				Path: workDir,
			},
		}, nil
	} 

This is similar to opening an IDE WorkSpace (but without opening any files), using the current directory(workspace in IDE) as the entry

2. a signle input & edit a kcl file in IDE

for _, source := range o.Sources {
	if pkgSource == nil {
		pkgSource = source
		rootPath, err = source.FindRootPath()
		if err != nil {
			return nil, err
		}
	} else {}

Ref

Rust Analyzer

First, we need to figure out what to analyze. To do this, we run cargo metadata to learn about Cargo packages for current workspace and dependencies, and we run rustc --print sysroot and scan the "sysroot" (the directory containing the current Rust toolchain's files) to learn about crates like std. This happens in the [GlobalState::fetch_workspaces] method. We load this configuration at the start of the server in [GlobalState::new], but it's also triggered by workspace change events and requests to reload the workspace from the client.

• Cargo.toml and Cargo.lock are stored in the root of your package (package root).
• Source code goes in the src directory.
• The default library file is src/lib.rs.
• The default executable file is src/main.rs.
  • Other executables can be placed in src/bin/.
• Benchmarks go in the benches directory.
• Examples go in the examples directory.
• Integration tests go in the tests directory.

Go:

The VS Code Go extension supports both GOPATH and Go modules modes.

Go modules are used to manage dependencies in recent versions of Go. Modules replace the GOPATH-based approach to specifying which source files are used in a given build, and they are the default build mode in go1.16+. We highly recommend Go development in module mode. If you are working on existing projects, please consider migrating to modules.

Unlike the traditional GOPATH mode, module mode does not require the workspace to be located under GOPATH nor to use a specific structure. A module is defined by a directory tree of Go source files with a go.mod file in the tree's root directory.

Your project may involve one or more modules. If you are working with multiple modules or uncommon project layouts, you will need to configure your workspace by using Workspace Folders. See the Supported workspace layouts documentation for more information.

[He1pa]

ref: https://dpe.org/how-your-monorepo-breaks-the-ide/

In Intellij IDEA we have long-standing assumption that one code file belongs to exactly one module and this module is compiled with a specific dependencies and has a specific compiler version and scpecific libiaries
SO you can only really edit it in one context at once even though you would want to do something in multiple context

[He1pa]

https://github.com/golang/tools/blob/master/gopls/doc/workspace.md
https://go.dev/blog/gopls-scalability

[He1pa]

Design

kcl.work

Reference from gopls's go work and rust's workspace in cargo.toml, use an explicit configuration file kcl.work to define the LSP workspace.

The kcl.work file is line oriented. Each line holds a single directive, made up of a keyword(there is only one keyword workspace) followed by arguments. For example:

workspace ./a
workspace ./b
workspace ./c/d 

For User

In single workspace

Most kcl projects may not need kcl.work. They only work in one workspace and manage the directory structure through kpm.

kcl mod init

- kcl.mod
- kcl.mod.lock
- main.k

LSP workspace is equal to kcl run in current path

In addition to main.k, users can also define the compilation entry of the current pkg through the etries field in kcl.mod

In mult-workspace(Konfig)

It is almost impossible to maintain the semantic information of hundreds or thousands of workspaces in memory at the same time. Also, in the Konfig repository, users may not care about the code in other projects. Therefore, we need to explicitly use kcl.work to define the active workspace.

In the following project structure:

-- kcl.mod (kcl v0.4.3, it is not the `kcl.mod` in current kpm)
-- base
  -- pkg1
    -- a.k
  -- pkg2
-- prog1
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- project.yaml
-- prog2
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
   -- project.yaml

Assuming that user A maintains prog1, he/she can create a kcl.work in the root directory and add the following code to kcl.work

workspace ./prog1/prod
workspace ./prog1/test

Functions like rename and find_ref will only work in stack prod and test.

It is generally inadvisable to commit go.work files into version control systems.

Compatibility Solutions

We recommend to set kcl.work or kcl.mod in the root directory to set up the workspace. But if not, use the directory of edit file. Search for compilation configuration and compilation entry in the current directory according to the following priority

• kcl.mod
• kcl.yaml
• main.k
• whole directory as entry

For dev

Change on server

• analysis edited file belongs to which pkg and affected pkgs, clear cache in resolver, namer, and global state feat: Clear the cache according to the changed module. #1581
• analysis edited file belongs to which workspaces, re-complie each workspace according to step 1: feat: Update lsp workspace #1594
• try to find workspaces setting(kcl.work or kcl.mod) when init language server: feat: init lsp workspace when start language server #1584
• When processing find ref, rename and other requests that may cross workspaces need to analyze them in multiple workspaces.