Discovering Python interpreters takes a long time

[minsa110]

Type: Bug

Behaviour

Expected vs. Actual

1. Launch VS Code insiders after an update
2. The "Discovering Python Interpreters" step takes a very long time (I didn't time it), so I don't see my Python environments in the Jupyter kernel picker for a while
3. Close VS Code insiders
4. Launch VS Code insiders again (no update this time)
5. The "Discovering Python Interpreters" step also takes a very long time (~40 seconds for that operation alone), but since Python environments were already loaded from previous session, they still do show up in the Jupyter kernel picker

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.8.5
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

Extension version: 2022.13.12212101
VS Code version: Code - Insiders 1.71.0-insider (dced70b, 2022-08-10T05:17:53.560Z)
OS version: Windows_NT x64 10.0.22000
Modes:
Sandboxed: Yes

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz (8 x 1498)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	undefined
Memory (System)	15.60GB (1.34GB free)
Process Argv	--crash-reporter-id 7c55b80b-a2c8-4229-8934-3646319b9311
Screen Reader	no
VM	0%

A/B Experiments

vsliv695:30137379
vsins829:30139715
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30422396
pythontb:30258533
pythonvspyt551cf:30291413
pythonptprofiler:30281269
vshan820:30294714
pythondataviewer:30285072
vscod805:30301674
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30404738
wslgetstarted:30449409
cppdebug:30492333
pylanb8912:30522163
vsclangdf:30492506
c4g48928:30535728
dsvsc012:30540213

[fireman0865]

I think you have to change version

[karrtikr]

Discovering some of these envs may require activating them which can take more time. Please provide the logs as mentioned in the issue template which has this information. Set python.logging.level to debug in User settings, reload window, then perform the steps to reproduce before providing the logs.

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

[karrtikr]

Once Jupyter starts using proposed discovery APIs, discovering interpreters won't be auto-triggered at startup and only when the kernel/interpreter list is opened, so a lot of this problem goes away.

[DonJayamanne]

discovering interpreters won't be auto-triggered at startup and only when the kernel/interpreter list is opened, so a lot of this problem goes away.

Won't this mean listing interpreters will be slower? See below why i ask this.

they still do show up in the Jupyter kernel picker

Based on her comments, the issue is the python envs are not showing up in the list for a while. And now if we trigger the query only when the notebook is opened, then the list will be fetched even later in the process hencing slowing the discovery even further.

[karrtikr]

Yes, but the fact that discovering interpreters is slow shouldn't matter as much as it would not block the startup of the extension. (language servers etc.)

List is loaded immediately with envs founds in the previous session, and for any new envs, it can be refreshed accordingly within those 40 seconds. Loading the whole list of interpreters would be slower, agreed.

[DonJayamanne]

matter as much as it would not block the startup of the extension. (language servers etc.)

If it will not block the extnesion, then i don't see why we should even wait like you have suggested. I.e. i think its best to tigger the search as soon as the exension loads

discovering interpreters won't be auto-triggered at startup and only when the kernel/interpreter list is opened, so a lot of this problem goes away.

I.e. i don't see why we need to wait till later, that would be be slower.
Anyways, just a thought. I guess Ian can look into that and figure out the pros/cons and the optimal solution.

FYI - @IanMatthewHuff will be working on integrating this API into the Jupyter extension.

[karrtikr]

If it will not block the extnesion

Thing is running discovery at startup does block starting other features in the extension, because of the single threaded nature of Node. Even though it's run in background, practically it seems to occupy so much clock that other features are not started up. That is the reason for waiting until later.

Particularly for the Python extension, refreshing the list when the picker is opened isn't so bad as it updates the list in place. But each extension is free to decide what's best for them.

[DonJayamanne]

Thing is running discovery at startup does block starting other features in the extension, because of the single threaded nature of Node. Even though it's run in background,

Thanks for the explanation, however I am aware of this (my comment was a question based on some assumptions, hence prefixed with if).
I've suggested using workers in the past to over come such blocking scenarios, I thought the plan was to implement such a technique, at least thats what I was informed when I demoed some perf improvements in the past #15813

[DonJayamanne]

I think we can continue the discussion of the API in the Jupyter repo thread (the issue you created for the adoption of the new API)

[karrtikr]

using workers in the past to over come such blocking scenarios

Interesting, I think the last comment #15813 (comment) indicated it wasn't a priority so we missed to do it. cc/ @karthiknadig

[karrtikr]

Closing in favor of #15813 where Jupyter adopts the Python API.