Extension API `getActiveInterperter` and `getInterpreters` return `undefined` and `[]`

[DonJayamanne]

Environment data

• VS Code version: 1.6.0
• Extension version (available under the Extensions sidebar): Latest Marketplace
• OS and version: Ubuntu 20 (CI)
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.11
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: Global
• Relevant/affected Python-related VS Code extensions and their versions: N/A
• Value of the python.languageServer setting: n/a

[NOTE: If you suspect that your issue is related to the Microsoft Python Language Server (python.languageServer: 'Microsoft'), please download our new language server Pylance from the VS Code marketplace to see if that fixes your issue]

Expected behaviour

getActiveInterperter and getInterpreters should not return empty values

Actual behaviour

On CI, the above two methods return empty values.
Our tests have been failing since thursday. I believe after the new Python extension was released.
We didn't chage any of the code in jupyter extension.

I can see from our logs the first calls to getActiveInterpreter & getInterpreters always returns undefined and []
this is causing our CI tests to fail & I expect this will will cause issues on end user machines as well.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Very difficult to repro on the client side (i guess its a timing thing)
This fails very consistently on CI, See last few runs of the PR microsoft/vscode-jupyter#7410 (last few runs)
The first calls to the two methods always return

Logs

See last few runs of the PR microsoft/vscode-jupyter#7410 (last few runs)

See here https://pipelines.actions.githubusercontent.com/KyBlh4sBTj8eoTzBTvt0CAApvyyj3PXuc4Tbg2yHyMGda4Kk7D/_apis/pipelines/1/runs/17104/signedlogcontent/7?urlExpires=2021-09-07T18%3A05%3A55.8149811Z&urlSigningMethod=HMACV1&urlSignature=%2F2YZdnccCib3FsH92LNpDZid6Mr7dPi9RQy7n%2BamSu0%3D

All outputs from the PR microsoft/vscode-jupyter#7410