When getting the repr of a pandas dataframe in the repl, don't do any customization to avoid getting a big output.

[zljubisic]

Type: Bug

Please look at the picture:

If I am debugging a pytest test, when I reach the breakpoint, debug console behaves as I have described in the picture above.

I have few questions:

1. why you are not showing (why "..."?) all columns because there is enough width in my terminal?
2. why 3rd row is swallowed after setting max_columns = None
3. why 3rd row is swalowed after expand_frame_repr = False

Extension version: 2022.14.0
VS Code version: Code 1.71.2 (74b1f979648cc44d385a2286793c226e611f59e7, 2022-09-14T21:03:37.738Z)
OS version: Windows_NT x64 10.0.19042
Modes:
Sandboxed: No
Remote OS version: Linux x64 3.10.0-1127.el7.x86_64

In case you need it:

df = pd.DataFrame(
    [
        ['A', 'short text', 'very very very very very very very very very very long text', 1],
        ['A', '1 very 2 very 3 very 4 very 5 very 6 very 7 very 8 very 9 very 10 very text', 'short text', 2],
        ['A', 'not so long text', 'also short text', 3],
    ], columns=['first', 'description', 'middle', 'last']
)
df
  first  ... last
0     A  ...    1
1     A  ...    2
2     A  ...    3

[3 rows x 4 columns]
len(df)
3
pd.options.display.max_colwidth = None
df
  first  ... last
0     A  ...    1
1     A  ...    2
2     A  ...    3

[3 rows x 4 columns]
pd.options.display.max_columns = None
df
  first                                        description  \
0     A                                         short text   
1     A  1 very 2 very 3 very 4 very 5 very 6 very 7 ve...   
2     A                                   not so long text   

                                              middle  last  
0  very very very very very very very very very v...     1  
1                                         short text     2  
2                                    also short text     3  
pd.options.display.expand_frame_repr = False
df
  first                                        description                                             middle  last
0     A                                         short text  very very very very very very very very very v...     1
1     A  1 very 2 very 3 very 4 very 5 very 6 very 7 ve...                                         short text     2
2     A                                   not so long text                                    also short text     3
print(df)
df.columns
Index(['first', 'description', 'middle', 'last'], dtype='object')
df.index
RangeIndex(start=0, stop=3, step=1)

[fabioz]

Regarding the print not showing, this is because pytest will grab the output itself (the issue being that you need to add --capture=no when running so that it doesn't collect the output -- see issue: microsoft/vscode-python#19322).

Regarding pandas customization, we really set up lower limits in the debugger (you could use the print to use the pandas configuration directly if you setup pytest to stop capturing the output).

To setup the limits used in the debugger you need to set the following environment variables (to the values you find appropriate in your use case):

PYDEVD_PANDAS_MAX_ROWS=300
PYDEVD_PANDAS_MAX_COLS=300
PYDEVD_PANDAS_MAX_COLWIDTH=80

Note that raising those values could make interacting with pandas slower when you're stepping in general in the debugger (I'm not sure why, but getting the representation of a pandas data frame is very slow in pandas and the debugger will ask for that representation whenever a pandas data frame is found).

As microsoft/vscode#162965 is tracking the remaining issue (where the contents of the debug console messages seem clipped in your use case), I'm closing this one.

[karthiknadig]

We have a PR on python extension to set --capture=no by default. microsoft/vscode-python#19903

[zljubisic]

You closed the issue, but the problem remains even if settings.json i like this (please notice "-s" -> "--capture=no":

{
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--no-cov", "-s"
    ]
}

[zljubisic]

Regarding pandas customization, we really set up lower limits in the debugger (you could use the print to use the pandas configuration directly if you setup pytest to stop capturing the output).

First, when you say "setup pytest to stop capturing the output" am I doing it correctly with settings.json above or there is another procedure?

To setup the limits used in the debugger you need to set the following environment variables (to the values you find appropriate in your use case):

PYDEVD_PANDAS_MAX_ROWS=300
PYDEVD_PANDAS_MAX_COLS=300
PYDEVD_PANDAS_MAX_COLWIDTH=80

How can I set pd.options.display.expand_frame_repr = False in env. vars case?
How to set env vars to None? unset?

Note that raising those values could make interacting with pandas slower when you're stepping in general in the debugger (I'm not sure why, but getting the representation of a pandas data frame is very slow in pandas and the debugger will ask for that representation whenever a pandas data frame is found).

I really don't get it, why you have so many problems with pandas dataframes.
Under the hood, they are numpy arrays which is very fast type of storage.

Actually, vscode way of debugging code with pandas dataframes is the worst way I have ever experienced.

For example, we have a lot of pydantic models that have several dataframes as attributes, something like this:

class MyObj(BaseModel):
    df1: pd.DataFrame
    df2: pd.DataFrame

To be able to see the dataframes in debugger, I have to add to each model a __repr__() method in which I will print dataframes' len() (because of swallowing last line problem) and print dataframes in their entirety. I have to print few "\n" at the and to raise chances of printing all lines.
So if I create an object x = MyObj(), when I am in debug console I can just type "x " and now chances are that I will get the output I need, but now I am getting all the dataframes all the time.

Do you think this is normal for an IDE?

As microsoft/vscode#162965 is tracking the remaining issue (where the contents of the debug console messages seem clipped in your use case), I'm closing this one.

I think you shouldn't.

[zljubisic]

Furthermore, if this is an idea:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": false,
            "env": {
                "PYDEVD_PANDAS_MAX_COLWIDTH": "1000",
            }
        }
    ]
}

than it doesn't work.

[fabioz]

Many issues in the same place... let's see one at a time:

1. Printing doesn't work (this is due to pytest capturing and setting -s in the args -- in the settings.json -- does work for me):

i.e.:

{
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": [
"--no-cov", "-s"
]
}

When I add the -s printing does work for me (see image below).

Can you check sys.argv to make sure it's actually using that value in your use case? Maybe the settings aren't being loaded from where you expect in your use case?

[fabioz]

2. Customizing the repr for pandas which is given be the debugger.

Right now because pandas is slow to print (it can easily take 0.3 seconds even on small dataframe and having many dataframes or a single big one without the current limits can easily take a few seconds which adds to a very annoying debug experience due to the repr slowness of pandas, which is why the limits were made much smaller).

Regarding pandas customization, we really set up lower limits in the debugger (you could use the print to use the pandas configuration directly if you setup pytest to stop capturing the output).

First, when you say "setup pytest to stop capturing the output" am I doing it correctly with settings.json above or there is another procedure?

To setup the limits used in the debugger you need to set the following environment variables (to the values you find appropriate in your use case):

PYDEVD_PANDAS_MAX_ROWS=300
PYDEVD_PANDAS_MAX_COLS=300
PYDEVD_PANDAS_MAX_COLWIDTH=80

How can I set pd.options.display.expand_frame_repr = False in env. vars case? How to set env vars to None? unset?

That isn't really handled by the debugger (so, you have to set it at runtime as usual).

Note that raising those values could make interacting with pandas slower when you're stepping in general in the debugger (I'm not sure why, but getting the representation of a pandas data frame is very slow in pandas and the debugger will ask for that representation whenever a pandas data frame is found).

I really don't get it, why you have so many problems with pandas dataframes. Under the hood, they are numpy arrays which is very fast type of storage.

Yes, storage is good as is usage of its api, just converting that to a string the user can see is very slow in pandas (they probably didn't really optimize those code paths).

You can't really unset those values, but you can change it to different values as you see fit. If you're interested, the code which handles this is:

https://github.com/microsoft/debugpy/blob/main/src/debugpy/_vendored/pydevd/pydevd_plugins/extensions/types/pydevd_plugin_pandas_types.py

-- you can locally comment the code which customizes the pandas display options (i.e.: customize_pandas_options -- just make it yield and comment all the other code).

-- The values for the PANDAS_MAX_ROWS / PANDAS_MAX_COLS / PANDAS_MAX_COLWIDTH are loaded from those constants. I guess we could provide a way to set it to None there (but that's currently not supported and yours is the first complain regarding it... I guess most of it steams from the fact that you should be able to use print to get any format you want but it's not working in your case due to another library capturing your output).

[fabioz]

Furthermore, if this is an idea:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": false,
            "env": {
                "PYDEVD_PANDAS_MAX_COLWIDTH": "1000",
            }
        }
    ]
}

than it doesn't work.

Note that this will only override the current if the current representation is too big (i.e.: it only makes the repr from pandas smaller, but if it's small already it'll not make it bigger).

So, you probably want to check the current values you have for pandas and customize those...

i.e.:

import pandas
pandas.get_option("display.max_rows")
pandas.get_option("display.max_columns")
pandas.get_option("display.max_colwidth")

if those are smaller you should raise those first...

Also, just to note, if you're running a test from the testing view you also need to add a "purpose": ["debug-test"], to the configuration...

[zljubisic]

@fabioz Do you have an explanation for this ?

Version: 1.72.0 (user setup)
Commit: 64bbfbf67ada9953918d72e1df2f4d8e537d340e
Date: 2022-10-04T23:20:39.912Z
Electron: 19.0.17
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.19044
Sandboxed: No

[fabioz]

@zljubisic on the first line the header is aligned (it's just that it appears in the same line as the tests/...:test_print_one from pytest) and the default print seems right for me.

I don't have an explanation for the smaller print after changing pd.options.display.max_colwidth = 500 though (and I can't reproduce it here) -- could be dependent on the pandas version you're using...

Given that the print just calls __str__ which is implemented by pandas itself you'll have to ask the pandas maintainers... when you're dealing with that print the debugger doesn't customize anything, it just calls the print and I don't know how pandas itself is customized in that case (you could set "justMyCode": false in the launch config and step into it if you're curious about why it does that...)

[fabioz]

Actually, I was able to reproduce the case where it gets smaller by making the terminal window smaller (apparently pandas checks the size of the terminal and as it sees that the string wouldn't fit in the terminal it just elides the column).

You'll have to refer to the pandas docs / ask the pandas maintainers how to override that though...

[zljubisic]

@fabioz you must kidding me. I have just executed pytest -s --no-cov in windows terminal (out of vscode) in two different terminal sizes. Both times dataframe is printed correctly.
Please be careful when you are accusing pandas because pandas is a remarkable python library.
All options from display are very well tested because they are used very often.
No one in data science has a small dataframes, so you have to use different display options to see what you want. There is an options context managers (https://pandas.pydata.org/docs/reference/api/pandas.option_context.html) to change options on the fly only for particular case. I am mentioning this just to explain to you how important it is if you work with dataframes.
Furthermore, if you try to "View Value in Data Viewer", it is slow, but more important, all column widths are the same, you can make them wider (except the very last column) but if you press refresh, columns are again of the same width, so data viewer is useless.

Some of things like jupyter notebooks are quite pleasant to use, vscode as editor is quite good, but working with pandas dataframes is not acceptable.
Try to run this example in pycharm to see how it will look like... and you closed this issue.