dateutil 2.6 gives segfault in normalizing timestamp with datetutil timezone #14621
Comments
|
Can you run the test with verbose mode so you can see for which test it segfaults? |
|
Some of our PR builds are segfaulting as well, e.g. https://travis-ci.org/pandas-dev/pandas/jobs/174412414 No failures on master yet though. Haven't had a chance to dig in yet. |
|
On that PR, it are all the builds using python 3.5 that fail with a segfault |
|
If I compare the installed versions before and after the moment tests started failing, it is |
|
https://pypi.python.org/pypi/python-dateutil/2.6.0 shocker that this breaks something |
|
OK, small reproducible example: gives |
|
So, it's related to using their timezones and normalizing offsets, so rather a specific use case that won't affect to much people I think. |
|
Trimmed down a bit further: |
well maybe though we really really are thoughtful /. try hard /. give notice |
If you have problems with backwards compatibility, feel free to kick off builds against |
|
we have been biten by downstream things before (by other deps) yes we could test against master but that makes our matrix even bigger |
|
@pganssle all respect for your hard work on dateutil! That's why I pointed out to Jeff that people could say the same for pandas, because even when you care about backwards compatibility, it always happen that people are relying on your project in a way you did not expect or don't want to. And that for sure happens in case of pandas as well. Apart from including dateutil master in our builds (as @jreback said, our test matrix is also already huge), what would also be helpful is getting notified of an upcoming release (not sure if you do a release candidate? or if there is a communication channel for such things? (where there is not too much other noise)) |
|
@jorisvandenbossche I have set up a python-dateutil mailing list, but no one has joined it, so I do not usually announce releases there. Usually I just create a Release issue before at least a major release and tag in it everyone who has submitted an issue or PR that was included in the release and leave that open for a day or so (unless it's a critical bugfix). I think going forward I'll also announce it on the mailing list (you can join here). At the moment, no one has ever sent a message to it, so I would consider it "low noise". |
jorisvandenbossche
changed the title
jorisvandenbossche
added
Bug
Compat
|
Thanks for educating me on the verbose flag, I was wondering how I could tell you which test failed. Looks like (from later posts in the thread) the problem has been figured out, but when I add the flag I get: (ana-1.0.5) psanaphi106: ~/rel/slaclab_conda/anarel-manage/recipes/psana $ python -c "import pandas as pd; pd.test('fast', verbose=10)" ... test_week_of_month_index_creation (pandas.tseries.tests.test_offsets.TestCaching) ... ok so that is probably the datetime issue. If it is simple, can you tell me how to re-run a specific test? Maybe it is worth my downgrading the datetime package to get the more recent pandas. best, David Schneider From: Joris Van den Bossche [[email protected]] Can you run the test with verbose mode so you can see for which test it segfaults? pd.test(verbose=10) � |
|
It was indeed the
Note that it is |
|
Based on this: >>> import pandas as pd
>>> from datetime import datetime
>>> dt = pd.Timestamp('2016-01-01 09:00:00')
>>> datetime.replace(dt, hour=0)
Segmentation fault (core dumped)And the fact that the problem came about because the way dateutil calculates timestamps under the hood has changed to a function that uses Also, I am not sure what |
|
so this is hitting this issue now: #7825 we are using the datetime.datetime.replace (iirc) and should simply override and fix it |
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621
…ones are present closes pandas-dev#14621 Author: Jeff Reback <[email protected]> Closes pandas-dev#14631 from jreback/replace and squashes the following commits: 3f95042 [Jeff Reback] BUG: segfault manifesting with dateutil=2.6 w.r.t. replace when timezones are present (cherry picked from commit f8bd08e)
Newly release dateutil 2.6.0 breaks some of the tests related to the use of dateutil timezones (travis is therefore currently failing)
Original report:
A small, complete example of the issue
I maintain central installs of miniconda environments that include pandas. My previous environment with pandas 0.19.0, if I did this
it worked. Now with pandas 0.19.1, it seg faults. Other packages may have been updated in the new environment.
Below are details - first the failure in my ana-1.0.5 environment, it is clearly segfaulting on a test maybe 2/3 the way through? Then the success in my ana-1.0.4 environment, then the pd.get_versions() in the working old environment, and finally in the newer environment where it fails:
OK (SKIP=537)
(ana-1.0.4) pslogin7a: ~/rel/slaclab_conda/anarel-test $ python -c "import pandas; pandas.show_versions()"
INSTALLED VERSIONS
commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-327.13.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C
LOCALE: None.None
pandas: 0.19.0
nose: 1.3.7
pip: 8.1.2
setuptools: 27.2.0
Cython: 0.24.1
numpy: 1.11.2
scipy: 0.18.1
statsmodels: None
xarray: 0.8.2
IPython: 5.1.0
sphinx: 1.4.8
patsy: None
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: 1.1.0
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None
(ana-1.0.4) pslogin7a: ~/rel/slaclab_conda/anarel-test $ source activate ana-1.0.5
(ana-1.0.5) pslogin7a: ~/rel/slaclab_conda/anarel-test $ python -c "import pandas; pandas.show_versions()"
INSTALLED VERSIONS
commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-327.13.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C
LOCALE: None.None
pandas: 0.19.1
nose: 1.3.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.1
numpy: 1.11.2
scipy: 0.18.1
statsmodels: None
xarray: 0.8.2
IPython: 5.1.0
sphinx: 1.4.8
patsy: None
dateutil: 2.6.0
pytz: 2016.7
blosc: None
bottleneck: 1.1.0
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None
(ana-1.0.5) pslogin7a: ~/rel/slaclab_conda/anarel-test $
The text was updated successfully, but these errors were encountered: