Segfaults on Python 3.8 (CPython master)

[tirkarthi]

Segfaults:

• indexing/test_indexing.py::TestFancy::test_setitem_list (see Collection of segfaults in CPython master #22718)
• ujson.encode(np.array(1)) (see Collection of segfaults in CPython master #22718)
• Issue detailed as follows:

Code Sample, a copy-pastable example if possible

from pandas import DataFrame
import pandas as pd

df = DataFrame({'A': [1, 2, 3],
                'B': [1., 2., 3.],
                'C': ['foo', 'bar', 'baz'],
                'D': pd.date_range('20130101', periods=3)})

print(df.agg('sum'))

Problem description

The above code works fine on Python 3.7 producing the below output

A            6
B            6
C    foobarbaz
dtype: object

The same code when it runs on CPython master causes segfault. I have opened an upstream issue that contains the stack trace when running under GDB. I am not sure if it's due to Pandas or Numpy since it's present in traceback. Since the above works fine on Python 3.7 but segfaults on Python 3.8 I want to report it here along with opening an upstream bug. I found this while running pytest -x pandas/frame/test_apply.py::TestDataFrameAggregate::test_nuiscance_columns

Expected Output

A            6
B            6
C    foobarbaz
dtype: object

Output of pd.show_versions()

(pandas-foo-venv) karthi@ubuntu-s-1vcpu-1gb-blr1-01:~$ python
Python 3.8.0a0 (heads/master:73820a6, Sep 14 2018, 18:33:26)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
<frozen importlib._bootstrap>:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
/home/karthi/pandas-foo-venv/lib/python3.8/site-packages/pandas/core/dtypes/inference.py:6: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable
o/home/karthi/pandas-foo-venv/lib/python3.8/site-packages/pandas/core/tools/datetimes.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
>>> pd.show_versions()
/home/karthi/pandas-foo-venv/lib/python3.8/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

INSTALLED VERSIONS
------------------
commit: None
python: 3.8.0.alpha.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-127-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 10.0.1
setuptools: 39.0.1
Cython: None
numpy: 1.15.1
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
>>>

Thanks

[jbrockmendel]

If you leave out column D, do you still get the segfault? Do you get it with just column D?

[tirkarthi]

If I have only D or no D then there is no segfault. It occurs when I have either A, D or B, D, or C, D.

Leaving out column D

from pandas import DataFrame
import pandas as pd

df = DataFrame({'A': [1, 2, 3],
                'B': [1., 2., 3.],
                'C': ['foo', 'bar', 'baz'],
                #'D': pd.date_range('20130101', periods=3)
})

print(df.agg('sum'))

output

A            6
B            6
C    foobarbaz
dtype: object

Only with Column D

from pandas import DataFrame
import pandas as pd

df = DataFrame({'D': pd.date_range('20130101', periods=3)})

print(df.agg('sum'))

output

Series([], dtype: float64)

[TomAugspurger]

@tirkarthi can you try with 3.8 and pandas master? We should be testing this in #28730, and it's not segfaulting.

[tirkarthi]

Sorry, I don't have enough time to test this. Given that the CI passes in linked PR and 3.8.0 is also released as a stable release I think we can close this. Thanks for the fix.

[TomAugspurger]

K. We'll reopen if things are still a problem.