FileDecorationProvider intermittently fails

[mattseddon]

Issue Type: Bug

I am currently developing a data science extension for iterative.ai which will integrate VS Code with the DVC cli (which is an open source Python project for versioning machine learning projects). Part of the extension is to do with large file management. In each repository we will have a list of Uris that are gitignored but tracked by DVC. We want to deliver as native an experience as possible to users so we decorate these Uris with colors that match up to the native git extension statuses (gitDecoration.addedResourceForeground, etc).

We register our FileDecorationProvider as close as possible to the instantiation of the extension however, recently the decorations have (intermittently) started getting stomped by the gitignore decoration whenever the ms-python.python extension is installed.

If I disable the Python extension then the decorations appear as expected but we would really like to be able to reliably run the extension in parallel with other extensions, especially ms-python.python (as we want to integrate with as much of the existing data science toolkit as possible).

I am wondering if something has changed recently in your extension that would be consuming a high amount of resources at startup and also how our extension can be effected by this one in this way.

The reason that I am raising a bug is because this worked up to a few days ago, I cannot find anything in the docs and I thought it was a solved problem.

Steps to Reproduce:

1. Run extension in parallel with ms-python.python.
2. FileDecorations will intermittently be overwritten (static per session, i.e they work or they don't).

Note: When starting up our extension having the ms-python.python installed vs not installed takes instantiation time from < 20ms to ~300ms.

Thanks for taking the time to read this 👍🏻 . Please let me know if need any further details.

Extension version: 2021.9.1230869389
VS Code version: Code 1.60.1 (83bd43bc519d15e50c4272c6cf5c1479df196a4d, 2021-09-10T17:09:14.403Z)
OS version: Darwin x64 20.6.0
Restricted Mode: No

System Info

Item	Value
CPUs	Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz (16 x 2300)
GPU Status	2d_canvas: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled webgl: enabled webgl2: enabled
Load (avg)	5, 3, 3
Memory (System)	16.00GB (0.20GB free)
Process Argv	--crash-reporter-id 343c20dd-d8ac-482e-bcc6-2244d6ab7bb5
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30300192
pythontb:30283811
vshan820:30294714
vstes263cf:30335440
pythonvsuse255:30340121
vscod805cf:30301675
vscextlangct:30333562
binariesv615:30325510
vsccppwt:30364497
bridge0708:30335490
pygetstartedt2:30360495
bridge0723:30353136
javagetstartedt:30364666
pydsgst2:30361792
vssid140cf:30363604
vssur157:30365996

[karthiknadig]

Strangely we don't even register a file decoration provider. Can you provide a link to VSIX for your extension that we can test this with?

[mattseddon]

@karthiknadig thanks for coming back to me. We've invited you to the project. You can download the VSIX from the CI. Can I ask that you don't share it externally as it is not published yet. The repo will be open source once we've done an alpha release and ironed out a lot of the problems we're facing.

Let me know if you need anything else, thanks again.

[karthiknadig]

@mattseddon Can you share logs from Output > Python? I want to check if the experiments are really running for you. Do you by chance have "telemetry.enableTelemetry": false, in your user settings? If possible can you share your user settings?

[mattseddon]

Python logs

User belongs to experiment group 'pythonaa'
User belongs to experiment group 'pythonJoinMailingListVar3'
User belongs to experiment group 'pythonJediLSP'
User belongs to experiment group 'pythonDiscoveryModuleWithoutWatcher'
User belongs to experiment group 'pythonTensorboardExperiment'
User belongs to experiment group 'PythonPyTorchProfiler'
User belongs to experiment group 'pythonDeprecatePythonPath'
User belongs to experiment group 'pythonSortEnvs'
User belongs to experiment group 'pythonRunFailedTestsButtonDisplayed'
User belongs to experiment group 'pythonRefreshTestsButtonDisplayedcf'
User belongs to experiment group 'pythonRememberDebugConfigcf'
Error 2021-09-16 17:25:54: Custom .env is likely not pointing to a valid file [EntryIsADirectory (FileSystemError): Unable to read file './example-get-started/.env' (Error: Unable to read file './example-get-started/.env' that is actually a directory)
	at _handleError (/Applications/Visual Studio 
...
Python interpreter path: ./.env/bin/python
Starting Pylance language server.
Error 2021-09-16 17:25:57: Custom .env is likely not pointing to a valid file [EntryIsADirectory (FileSystemError): 
...
> conda --version
Error 2021-09-16 17:25:57: Custom .env is likely not pointing to a valid file [EntryIsADirectory (FileSystemError): Unable to ...
Error 2021-09-16 17:25:57: Custom .env is likely not pointing to a valid file [EntryIsADirectory (FileSystemError): Unable to ...

User settings

{
    "editor.renderWhitespace": "all",
    "editor.wordSeparators": "`~!#^&*()-=+[{]}\\|;:'\",.<>/?",
    "breadcrumbs.enabled": true,
    "git.enableSmartCommit": true,
    "workbench.editor.enablePreviewFromQuickOpen": false,
    "workbench.activityBar.visible": true,
    "editor.detectIndentation": false,
    "editor.insertSpaces": false,
    "[python]": {
        "editor.insertSpaces": true
    },
    "git.ignoreLegacyWarning": true,
		"explorer.confirmDelete": false,
    "workbench.settings.openDefaultKeybindings": true,
    "typescript.updateImportsOnFileMove.enabled": "always",
    "python.formatting.provider": "black",
    "editor.largeFileOptimizations": false,
    "js/ts.implicitProjectConfig.experimentalDecorators": true,
    "files.trimFinalNewlines": true,
    "terminal.integrated.automationShell.osx": "/bin/zsh",
    "terminal.integrated.fontFamily": "MesloLGS NF",
    "bracket-pair-colorizer-2.colors": ["#0adb8b", "#00eaff", "#f60e67"],
    "editor.matchBrackets": "never",
    "bracket-pair-colorizer-2.highlightActiveScope": true,
    "bracket-pair-colorizer-2.scopeLineRelativePosition": false,
    "diffEditor.ignoreTrimWhitespace": false,
    "importCost.smallPackageColor": "#0adb8b",
    "importCost.mediumPackageColor": "#00eaff",
    "importCost.largePackageColor": "#f60e67",
    "prettier.tabWidth": 4,
    "python.showStartPage": true,
    "python.languageServer": "Pylance",
    "terminal.external.osxExec": "/Applications/iTerm.app/Contents/MacOS/iTerm2",
    "workbench.editorAssociations": {
        "*.ipynb": "jupyter-notebook"
    },
    "javascript.updateImportsOnFileMove.enabled": "always",
    "terminal.integrated.tabs.enabled": true,
    "terminal.integrated.defaultProfile.osx": "/bin/zsh (migrated)",
    "coverage-gutters.gutterIconPathDark": "/Users/mattseddon/.vscode/extensions/gutter-icon-dark.svg",
    "coverage-gutters.noGutterIconPathDark": "/Users/mattseddon/.vscode/extensions/no-gutter-icon-dark.svg",
    "coverage-gutters.partialGutterIconPathDark": "/Users/mattseddon/.vscode/extensions/partial-gutter-icon-dark.svg",
    "editor.formatOnSave": true,
    "terminal.integrated.profiles.osx": {
        "/bin/zsh (migrated)": {
            "path": "/bin/zsh",
            "args": ["-l"]
        }
    },
    "files.insertFinalNewline": true,
    "window.zoomLevel": 1,
    "notebook.cellToolbarLocation": {
        "default": "right",
        "jupyter-notebook": "left"
    },
    "git-graph.graph.colours": [
        "#FF428E",
        "#A8FFEF",
        "#0ADB8B",
        "#00EAFF",
        "#F834BB",
        "#2BF5E9",
        "#D831D7",
        "#FF594C"
	],
	"workbench.colorTheme": "Dive Bar"
}

Telemetry is enabled:

[mattseddon]

Closing this as the issue appears to have been caused by a massive amount of (extension generated) erroneous data underneath the ms-python.python key in my local globalState sqlite database (~/Library/Application Support/Code/User/globalStorage/state.vscdb). The only theory that I have is that the extension host was behaving in unexpected ways when the python extension attempted to load the memento.

For now steps to resolve the issue where:

1. Uninstall ms-python.python.
2. rm ms-python.python-2021.9.1246542782, ms-python.vscode-pylance-2021.9.2, ms-toolsai.jupyter-2021.8.2041215044 & ms-toolsai.jupyter-keymap-1.0.0 underneath ~/L/Application Su/Code/CachedExtensionVSIXs
3. delete from ItemTable where key = 'ms-python.python'; in ~/Library/Application\ Support/Code/User/globalStorage/state.vscdb
4. Completely close VS Code
5. Reopen VS Code
6. Install ms-python.python
7. Check globalState db (still empty)
8. Start debugger
9. Check globalState entry seemed reasonable - decorations appear as expected
10. Reload window - decorations appear as expected
11. Check db - PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS has grown
12. Reload window - decorations appear as expected
13. Check db - PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS has grown again

After completing the above steps I now have significantly less environments in my globalState. I tried the above without removing the extensions and the same ~6mb of data would be generated underneath the ms-python.python key each time I reloaded the window. My first guess would be something unexpected happening in your CacheableLocatorService. I am slightly concerned that as I continue to load the window that PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS will again get out of control and I'll have to go through the above again.

Please let me know if you would like me to raise a new bug.

Thank you