BUG: Memory leak when setting Series value via __setitem__

[jzombi]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import gc
import os
import psutil

# create example data
items = [f'item_{i}' for i in range(10_000)]
data = {}
for i, col in enumerate(items[:1000]):
    data[col] = [1]*len(items)
df = pd.DataFrame(index=items, data=data)
gc.collect()


process = psutil.Process(os.getpid())
rss = process.memory_info().rss/1024/1024

for item in df.columns[:10]:
    df[item][item] = -10
    gc.collect()
    new_rss = process.memory_info().rss/1024/1024
    print('{:.2f} MiB'.format(new_rss-rss))

Issue Description

In pandas version 1.4.2 the memory usage increases at each iteration of the last for loop, outputs:

76.55 MiB
152.99 MiB
229.16 MiB
305.08 MiB
381.00 MiB
456.92 MiB
532.58 MiB
608.24 MiB
683.96 MiB
759.51 MiB

Expected Behavior

In pandas version 1.3.5 the memory usage remains constant, output:

0.00 MiB
0.00 MiB
0.00 MiB
0.00 MiB
0.00 MiB
0.00 MiB
0.00 MiB
0.00 MiB
0.00 MiB
0.00 MiB

Installed Versions

INSTALLED VERSIONS
------------------
commit           : 4bfe3d07b4858144c219b9346329027024102ab6
python           : 3.8.12.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.10.0-13-amd64
Version          : #1 SMP Debian 5.10.106-1 (2022-03-17)
machine          : x86_64
processor        :
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.4.2
numpy            : 1.21.6
pytz             : 2022.1
dateutil         : 2.8.2
pip              : 22.1.1
setuptools       : 56.0.0
Cython           : 0.29.30
pytest           : None
hypothesis       : None
sphinx           : 4.5.0
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : 1.0.2
psycopg2         : None
jinja2           : 3.1.2
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : None
gcsfs            : None
markupsafe       : 2.1.1
matplotlib       : 3.5.1
numba            : 0.55.1
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.8.0
snappy           :
sqlalchemy       : 1.4.36
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
zstandard        : None

[phofl]

Before 1.4 setitem probably wrote into the existing array, since 1.4 this always makes a copy. Could you try this without chained indexing?

[jzombi]

Before 1.4 setitem probably wrote into the existing array, since 1.4 this always makes a copy.

I think so too (maybe related: #43406?)

Could you try this without chained indexing?

Could you elaborate on this? The following variants also have the same problem (but I think it is to be expected):

Variant 1:

[...]
for item in df.columns[:10]:
    series = df[item]
    series[item] = -10
    [...]

Variant 2:

[...]
def f(item, series):
    series[item] = -10

for item in df.columns[:10]:
    f(item, df[item])
    [...]

If series are placed on a list (instead of using a dataframe), the memory usage also increases, but the increment is much smaller. I guess the former copied the whole dataframe in each iteration, this one copies only the selected series. (This example shows increasing memory usage in both 1.4.2 and 1.3.5.)

import pandas as pd
import numpy as np
import gc
import os
import psutil

items = [f'item_{i}' for i in range(10_000)]
data = []
for i, col in enumerate(items[:1000]):
    data.append(pd.Series(data=np.random.rand(len(items)), index=items))
gc.collect()

process = psutil.Process(os.getpid())
rss = process.memory_info().rss/1024/1024

for i, item in enumerate(items[:10]):
    data[i][item] = 10
#    data[i].loc[item] = 10  # <- rss increases the same way as with previous line
    gc.collect()
    new_rss = process.memory_info().rss/1024/1024
    print('{:.2f} MiB'.format(new_rss-rss))

If a single series __setitem__ is called multiple times (eg. on different elements), the RSS does not increase. Seems like after the first copy, further copies are not made, or are correctly garbage collected.

[phofl]

I meant using loc, e.g. df.loc[item, item] (I think this does what you want)

[phofl]

The internal cache is not updated correctly when using chained indexing (since 1.4). I think we already have similar issues about that, could you check?

[jzombi]

Yes, both:

df.loc[item, item] = -10

and

df[item].loc[item] = -10

works fine. Actually I've used the solution you suggested to solve my issue before opening this bugreport. I've reported this as a bug only because I think that changing the values of the dataframe this way, even if it makes a copy, should not increase the memory usage further and further (seems like references are stuck somewhere).

[jzombi]

The internal cache is not updated correctly when using chained indexing (since 1.4). I think we already have similar issues about that, could you check?

I'm sorry, I'm afraid I don't know enough about pandas to recognize which issues are really related.

Another strange thing, if df.memory_usage() is called before the for loop, the memory usage does not increase at each iteration step.

df.memory_usage()
for item in df.columns[:10]:
    df[item][item] = -10

    gc.collect()
    new_rss = process.memory_info().rss/1024/1024
    print('{:.2f} MiB'.format(new_rss-rss))

[simonjayhawkins]

Before 1.4 setitem probably wrote into the existing array, since 1.4 this always makes a copy.

I think so too (maybe related: #43406?)

can confirm.

first bad commit: [03dd698] BUG: DataFrame.setitem sometimes operating inplace (#43406)

cc @jbrockmendel

[simonjayhawkins]

The internal cache is not updated correctly when using chained indexing (since 1.4). I think we already have similar issues about that, could you check?

I'm sorry, I'm afraid I don't know enough about pandas to recognize which issues are really related.

several issue point back to #43406. The cache issue may have been referring to is #45684 which should be fixed.

[simonjayhawkins]

moving to 1.4.4

[simonjayhawkins]

closed in #48215