pandas is no longer importable with -OO optimization

[shoyer]

Code Sample, a copy-pastable example if possible

In your shell:

$ python -OO -c 'import pandas'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/__init__.py", line 42, in <module>
    from pandas.core.api import *
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/api.py", line 10, in <module>
    from pandas.core.groupby.groupby import Grouper
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/groupby/__init__.py", line 2, in <module>
    from pandas.core.groupby.groupby import (
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 46, in <module>
    from pandas.core.index import (Index, MultiIndex,
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/index.py", line 2, in <module>
    from pandas.core.indexes.api import *
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/indexes/api.py", line 4, in <module>
    from pandas.core.indexes.base import (Index,
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 7, in <module>
    from pandas._libs import (lib, index as libindex, tslib as libts,
  File "pandas/_libs/index.pyx", line 28, in init pandas._libs.index
  File "pandas/_libs/tslibs/period.pyx", line 59, in init pandas._libs.tslibs.period
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/tseries/offsets.py", line 1092, in <module>
    class CustomBusinessMonthEnd(_CustomBusinessMonth):
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/tseries/offsets.py", line 1093, in CustomBusinessMonthEnd
    __doc__ = _CustomBusinessMonth.__doc__.replace('[BEGIN/END]', 'end')
AttributeError: 'NoneType' object has no attribute 'replace'

Problem description

-OO optimization strips out docstrings, which may give a minor performance boost (I honestly don't know). Nonetheless, users requested that xarray import properly properly with the -OO flag (pydata/xarray#1706), so we added a regression test that caught this in the latest pandas release (pydata/xarray#1708).

Expected Output

Pandas should be imported without any errors.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: 3.5.0
pip: 9.0.1
setuptools: 39.0.1
Cython: None
numpy: 1.14.2
scipy: None
pyarrow: None
xarray: None
IPython: 6.3.0
sphinx: None
patsy: None
dateutil: 2.7.2
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[WillAyd]

Hmm OK. Just running locally it looks like it could be fixed adding a condition in 3 places that assume we have a docstring, but don't if you run with those flags.

Just out of curiosity do you only have this in your travis configuration or have you placed it somewhere in your unit testing? Think it would be easier to track if we had the latter but off the top of my head not sure how to do that - curious if you have any insight

[shoyer]

You could probably test this with a Python subprocess, but I didn’t bother for xarray so it’s just in our Travis config.

[bryevdv]

FWIW in case it's helpful: Bokeh had to define a format_docstring function to avoid direct access to docstrings:

https://github.com/bokeh/bokeh/blob/master/bokeh/util/string.py#L84-L100

and added a test to maintain working imports with -OO

https://github.com/bokeh/bokeh/blob/master/tests/test_python_execution_with_OO.py

[jorisvandenbossche]

@bryevdv thanks for the links. @mattpap pointed out the same test file in the PR, that is certainly useful.
We have some machinery (decorators) to deal with docstring manipulations, we should just be careful to always use it :-)

[Siecje]

I ran into this bug when packaging an application with cx_Freeze. I have reverted to pandas==0.22.0 for now.

[jorisvandenbossche]

@stonebig you removed your comment, does that mean it was false alarm?

[stonebig]

my problem is still there, but I'm not so sure if this is THIS root cause. I can leave it again : holoviz/holoviews#2738

uploading typical problem in 40' at https://sourceforge.net/projects/winpython/files/WinPython_3.7/3.7.0.1/betas/WinPython64-3.7.0.1rc.exe

[stonebig]

living outside miniconda brings original issues, sorry.

[jorisvandenbossche]

How is the issue that you posted above related to this one? You don't get the error as in the top post?

[stonebig]

the relationship is that I followed the cause of bokeh not being tested against pandas-0.23.x.