Description
Test plan
Environment
- OS: XXX (Windows, macOS, latest Ubuntu LTS)
- Shell: XXX (Command Prompt, PowerShell, bash, fish)
- Python
- Distribution: XXX (CPython, miniconda)
- Version: XXX (2.7, latest 3.x)
- VS Code: XXX (Insiders)
Tests
ALWAYS:
- Check the
Output
window underPython
for logged errors - Have
Developer Tools
open to detect any errors - Consider running the tests in a multi-folder workspace
- Focus on in-development features (i.e. experimental debugger and language server)
Environment
Interpreters
- Interpreter is shown in the status bar
- An interpreter can be manually specified using the
Select Interpreter
command - Detected system-installed interpreters
- Detected an Anaconda installation
- (Linux/macOS) Detected all interpreters installed w/ pyenv detected
-
"python.pythonPath"
triggers an update in the status bar -
Run Python File in Terminal
-
Run Selection/Line in Python Terminal
- Right-click
- Command
-
Shift+Enter
Virtual environments
ALWAYS:
-
Use the latest version of Anaconda
-
Realize that
conda
is slow -
Create an environment with a space in their path somewhere as well as upper and lowercase characters
-
Make sure that you do not have
python.pythonPath
specified in yoursettings.json
when testing automatic detection -
Do note that the
Select Interpreter
drop-down window scrolls -
Detected a single virtual environment at the top-level of the workspace folder on Mac when when
python
command points to default Mac Python installation orpython
command fails in the terminal.- Appropriate suffix label specified in status bar (e.g.
(venv)
)
- Appropriate suffix label specified in status bar (e.g.
-
Detected a single virtual environment at the top-level of the workspace folder on Windows when
python
fails in the terminal.- Appropriate suffix label specified in status bar (e.g.
(venv)
)
- Appropriate suffix label specified in status bar (e.g.
-
Detected a single virtual environment at the top-level of the workspace folder
- Appropriate suffix label specified in status bar (e.g.
(venv)
) -
Create Terminal
works- Steals focus
-
"python.terminal.activateEnvironment": false
deactivates automatically running the activation script in the terminal
- After the language server downloads it is able to complete its analysis of the environment w/o requiring a restart
- Appropriate suffix label specified in status bar (e.g.
-
Detect multiple virtual environments contained in the directory specified in
"python.venvPath"
-
Detected all conda environments created with an interpreter
- Appropriate suffix label specified in status bar (e.g.
(condaenv)
) - Prompted to install Pylint
- Asked whether to install using conda or pip
- Installs into environment
-
Create Terminal
works-
"python.terminal.activateEnvironment": false
deactivates automatically running the activation script in the terminal
-
- After the language server downloads it is able to complete its analysis of the environment w/o requiring a restart
- Appropriate suffix label specified in status bar (e.g.
-
(Linux/macOS until
-m
is supported) Detected the virtual environment created by pipenv- Appropriate suffix label specified in status bar (e.g.
(pipenv)
) - Prompt to install Pylint uses
pipenv install --dev
-
Create Terminal
works-
"python.terminal.activateEnvironment": false
deactivates automatically running the activation script in the terminal
-
- After the language server downloads it is able to complete its analysis of the environment w/o requiring a restart
- Appropriate suffix label specified in status bar (e.g.
-
(Linux/macOS) Virtual environments created under
{workspaceFolder}/.direnv/python-{python_version}
are detected (for direnv and itslayout python3
support)- Appropriate suffix label specified in status bar (e.g.
(venv)
)
- Appropriate suffix label specified in status bar (e.g.
Environment files
Sample files:
# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')
# .env
WHO=world
PYTHONPATH=some/path/somewhere
ALWAYS:
-
Make sure to use
Reload Window
between tests to reset your environment -
Note that environment files only apply under the debugger and Jedi
-
Environment variables in a
.env
file are exposed when running under the debugger -
"python.envFile"
allows for specifying an environment file manually (e.g. Jedi picks upPYTHONPATH
changes) -
envFile
in alaunch.json
configuration works
Debugging
-
pythonPath
setting in yourlaunch.json
overrides yourpython.pythonPath
default setting
Linting
ALWAYS:
- Check under the
Problems
tab to see e.g. if a linter is raising errors
Pylint/default linting
[Prompting to install Pylint is covered under Environments
above]
For testing the disablement of the default linting rules for Pylint:
# pylintrc
[MESSAGES CONTROL]
enable=bad-names
# example.py
foo = 42 # Marked as a blacklisted name.
- Installation via the prompt installs Pylint as appropriate
- Uses
--user
for system-install of Python - Installs into a virtual environment environment directly
- Uses
- Pylint works
-
"python.linting.pylintUseMinimalCheckers": false
turns off the default rules w/ nopylintrc
file present - The existence of a
pylintrc
file turns off the default rules
Other linters
Note:
-
You can use the
Run Linting
command to run a newly installed linter -
When the extension installs a new linter, it turns off all other linters
-
flake8 works
-
Select linter
lists the linter and installs it if necessary
-
-
mypy works
-
Select linter
lists the linter and installs it if necessary
-
-
pep8 works
-
Select linter
lists the linter and installs it if necessary
-
-
prospector works
-
Select linter
lists the linter and installs it if necessary
-
-
pydocstyle works
-
Select linter
lists the linter and installs it if necessary
-
-
pylama works
-
Select linter
lists the linter and installs it if necessary
-
-
3 or more linters work simultaneously (make sure you have turned on the linters in your
settings.json
)-
Run Linting
runs all activated linters -
"python.linting.enabled": false
disables all linters - The
Enable Linting
command changes"python.linting.enabled"
-
-
"python.linting.lintOnSave
works
Editing
IntelliSense
Please also test for general accuracy on the most "interesting" code you can find.
-
"python.autoComplete.extraPaths"
works -
"python.autocomplete.addBrackets": true
causes auto-completion of functions to append()
Formatting
Sample file:
# There should be _some_ change after running `Format Document`.
import os,sys;
def foo():pass
- Prompted to install a formatter if none installed and
Format Document
is run- Installing
autopep8
works - Installing
black
works - Installing
yapf
works
- Installing
- Formatters work with default settings (i.e.
"python.formatting.provider"
is specified but not matching*Path
or*Args
settings)- autopep8
- black
- yapf
- Formatters work when appropriate
*Path
and*Args
settings are specified (use absolute paths; use~
if possible)- autopep8
- black
- yapf
-
"editor.formatOnType": true
works and has expected results
Refactoring
-
Extract Variable
works- You are prompted to install
rope
if it is not already available
- You are prompted to install
-
Extract method
works- You are prompted to install
rope
if it is not already available
- You are prompted to install
-
Sort Imports
works
Debugging
ALWAYS:
-
Test the current debugger
-
Test the experimental debugger (and note whether it is at least as fast as the old debugger)
-
Configurations work (see
package.json
and the"configurationSnippets"
section for all of the possible configurations) -
Running code from start to finish w/ no special debugging options (e.g. no breakpoints)
-
Breakpoint-like things
- Breakpoint
- Set
- Hit
- Conditional breakpoint
- Expression
- Set
- Hit
- Hit count
- Set
- Hit
- Expression
- Logpoint
- Set
- Hit
- Breakpoint
-
Stepping
- Over
- Into
- Out
-
Can inspect variables
- Through hovering over variable in code
-
Variables
section of debugger sidebar
-
Remote debugging works
- ... over SSH
Unit testing
unittest
import unittest
MODULE_SETUP = False
def setUpModule():
global MODULE_SETUP
MODULE_SETUP = True
class PassingSetupTests(unittest.TestCase):
CLASS_SETUP = False
METHOD_SETUP = False
@classmethod
def setUpClass(cls):
cls.CLASS_SETUP = True
def setUp(self):
self.METHOD_SETUP = True
def test_setup(self):
self.assertTrue(MODULE_SETUP)
self.assertTrue(self.CLASS_SETUP)
self.assertTrue(self.METHOD_SETUP)
class PassingTests(unittest.TestCase):
def test_passing(self):
self.assertEqual(42, 42)
def test_passing_still(self):
self.assertEqual("silly walk", "silly walk")
class FailingTests(unittest.TestCase):
def test_failure(self):
self.assertEqual(42, -13)
def test_failure_still(self):
self.assertEqual("I'm right!", "no, I am!")
-
Run All Unit Tests
triggers the prompt to configure the test runner - Tests are discovered (as shown by code lenses on each test)
- Code lens for a class runs all tests for that class
- Code lens for a method runs just that test
-
Run Test
works -
Debug Test
works - Module/suite setup methods are also run (run the
test_setup
method to verify)
-
pytest
def test_passing():
assert 42 == 42
def test_failure():
assert 42 == -13
-
Run All Unit Tests
triggers the prompt to configure the test runner-
pytest
gets installed
-
- Tests are discovered (as shown by code lenses on each test)
-
Run Test
works -
Debug Test
works
-
- A
Diagnostic
is shown in the problems pane for each failed/skipped test- The
Diagnostic
s are organized according to the file the test was executed from (not neccesarily the file it was defined in) - The appropriate
DiagnosticRelatedInformation
is shown for eachDiagnostic
- The
DiagnosticRelatedInformation
reflects the traceback for the test
- The
nose
def test_passing():
assert 42 == 42
def test_failure():
assert 42 == -13
-
Run All Unit Tests
triggers the prompt to configure the test runner- Nose gets installed
- Tests are discovered (as shown by code lenses on each test)
-
Run Test
works -
Debug Test
works
-
General
- Code lenses appears
-
Run Test
lens works (and status bar updates as appropriate) -
Debug Test
lens works - Appropriate ✔/❌ shown for each test
-
- Status bar is functioning
- Appropriate test results displayed
-
Run All Unit Tests
works -
Discover Unit Tests
works (resets tests result display in status bar) -
Run Unit Test Method ...
works -
View Unit Test Output
works - After having at least one failure,
Run Failed Tests
works