BUG: Memory leak when using custom accessor

[thehomebrewnerd]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import gc
import pandas as pd


@pd.api.extensions.register_dataframe_accessor("test")
class TestAccessor:
    def __init__(self, pandas_obj):
        self._obj = pandas_obj


def dump_garbage():
    gc.collect()
    
    print(f"\nUNCOLLECTABLE OBJECTS: {len(gc.garbage)}")
    for x in gc.garbage:
        s = str(x)
        if len(s) > 80:
            s = s[:77]+'...'
        print(type(x), "\n  ", s)


gc.enable()
gc.set_debug(gc.DEBUG_SAVEALL)

df = pd.DataFrame({"a": [1, 2, 3]})
del df
dump_garbage()  # <- No uncollectable objects found

df = pd.DataFrame({"a": [1, 2, 3]})
df.test  # <- Initialize custom accessor
del df
dump_garbage()  # <- Multiple uncollectable objects found

Problem description

When using a custom accessor, memory is not being freed as expected. This appears to be a result of the accessor storing a reference to the dataframe, as demonstrated in the example above.

The output of the example code is included below showing that after deletion of the dataframe without the accessor initialized, there are 0 uncollectible objects, but after deleting the dataframe with the accessor initialized 17 uncollectible objects remain.

For additional context refer to this issue and this pull request.

Note, this problem appears to be solvable by storing a weak reference to the dataframe in the accessor instead, but I am concerned that using that approach could create other issues.

UNCOLLECTABLE OBJECTS: 0

UNCOLLECTABLE OBJECTS: 17
<class 'pandas.core.frame.DataFrame'> 
      a
0  1
1  2
2  3
<class 'pandas.core.indexes.base.Index'> 
   Index(['a'], dtype='object')
<class 'dict'> 
   {'_data': array(['a'], dtype=object), '_index_data': array(['a'], dtype=objec...
<class 'pandas.core.indexes.range.RangeIndex'> 
   RangeIndex(start=0, stop=3, step=1)
<class 'dict'> 
   {'_range': range(0, 3), '_name': None, '_cache': {}, '_id': <object object at...
<class 'pandas.core.internals.blocks.IntBlock'> 
   IntBlock: slice(0, 1, 1), 1 x 3, dtype: int64
<class 'pandas._libs.internals.BlockPlacement'> 
   BlockPlacement(slice(0, 1, 1))
<class 'slice'> 
   slice(0, 1, 1)
<class 'pandas.core.internals.managers.BlockManager'> 
   BlockManager
Items: Index(['a'], dtype='object')
Axis 1: RangeIndex(start=0, ...
<class 'list'> 
   [Index(['a'], dtype='object'), RangeIndex(start=0, stop=3, step=1)]
<class 'tuple'> 
   (IntBlock: slice(0, 1, 1), 1 x 3, dtype: int64,)
<class 'dict'> 
   {'_is_copy': None, '_mgr': BlockManager
Items: Index(['a'], dtype='object')
A...
<class 'pandas.core.flags.Flags'> 
   <Flags(allows_duplicate_labels=True)>
<class 'weakref'> 
   <weakref at 0x150273900; dead>
<class 'dict'> 
   {'_allows_duplicate_labels': True, '_obj': <weakref at 0x150273900; dead>}
<class '__main__.TestAccessor'> 
   <__main__.TestAccessor object at 0x10af93520>
<class 'dict'> 
   {'_obj':    a
0  1
1  2
2  3}
<class 'pandas.core.series.Series'> 
   0       a
1    0  1
2    1  2
3    2  3
dtype: object
<class 'pandas.core.internals.blocks.ObjectBlock'> 
   ObjectBlock: 4 dtype: object
<class 'pandas._libs.internals.BlockPlacement'> 
   BlockPlacement(slice(0, 4, 1))
<class 'slice'> 
   slice(0, 4, 1)
<class 'pandas.core.internals.managers.SingleBlockManager'> 
   SingleBlockManager
Items: RangeIndex(start=0, stop=4, step=1)
ObjectBlock: 4 ...
<class 'list'> 
   [RangeIndex(start=0, stop=4, step=1)]
<class 'tuple'> 
   (ObjectBlock: 4 dtype: object,)
<class 'dict'> 
   {'_is_copy': None, '_mgr': SingleBlockManager
Items: RangeIndex(start=0, stop...
<class 'pandas.core.flags.Flags'> 
   <Flags(allows_duplicate_labels=True)>
<class 'weakref'> 
   <weakref at 0x1502ba810; dead>
<class 'dict'> 
   {'_allows_duplicate_labels': True, '_obj': <weakref at 0x1502ba810; dead>}
<class 'pandas.core.strings.accessor.StringMethods'> 
   <pandas.core.strings.accessor.StringMethods object at 0x1502aa0d0>
<class 'dict'> 
   {'_inferred_dtype': 'string', '_is_categorical': False, '_is_string': False, ...

Expected Output

Both example cases above should release the memory used by the dataframe once it is deleted.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.8.5.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
Version : Darwin Kernel Version 18.7.0: Mon Aug 31 20:53:32 PDT 2020; root:xnu-4903.278.44~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.19.5
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.1
setuptools : 47.1.0
Cython : None
pytest : 6.0.1
hypothesis : None
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.04.0
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 4.0.0
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None