BUG: memory leak in unique() with object dtype

[adbull]

Code Sample, a copy-pastable example if possible

The following consumes an increasing amount of memory, and eventually dies.

import gc
import pandas as pd
import numpy as np

x = pd.Series(np.arange(10**6) % 10**3).astype(str)
while True:
    x.unique()
    gc.collect()

Problem description

When calling unique() on a series of objects, memory is allocated which cannot be freed from Python. This affects the current GitHub version of Pandas.

Expected Output

Code should run forever with stable memory usage.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.8-100.fc24.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C
LANG: C
LOCALE: None.None

pandas: 0.19.0+824.gf114af0
pytest: 3.0.5
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.18.1
xarray: 0.9.1
IPython: 4.2.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: 0.999
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
s3fs: None
pandas_gbq: None
pandas_datareader: None

[chris-b1]

Thanks for the report - if I change the data to use the object path (x[0] = 1) it no longer leaks, so something must be not getting freed in the StringHashTable path.

pandas/pandas/_libs/hashtable_class_helper.pxi.in

Line 464 in f114af0

cdef class StringHashTable(HashTable):

[chris-b1]

Helpful todo

pandas/pandas/_libs/src/numpy_helper.h

Line 97 in f114af0

// TODO(general): memory leak here

I think we're only freeing the char** vectors, not the actual strings. xref #14859, cc @jreback

[jreback]

ok this is easy to fix

boy have to love c strings and manual memory management