[BUG]: Wrong unix timestamp parsing with floating point using pd.to_datetime

[lopezco]

Hello!

I'm trying to parse floating point timestamps with pd.to_datetime but the only way that it works is using the deprecated parameter coerce=True

For example, without coerce=True the result is wrong.

>>> pd.to_datetime(1.1, unit='s')  # Expected result Timestamp('1970-01-01 00:00:01.100000')
Timestamp('1970-01-01 00:00:01')

However, with coerce=True the result is correct.

>>> pd.to_datetime(1.1, unit='s', coerce=True)  # Expected result Timestamp('1970-01-01 00:00:01.100000')
Timestamp('1970-01-01 00:00:01.100000')

I'm happy to help if you need more details.

Code Sample (copy-pastable example)

import pandas as pd
expected_result = pd.Timestamp('1970-01-01 00:00:01.100000')
timestamp1 = pd.to_datetime(1.1, unit='s')
timestamp2 = pd.to_datetime(1.1, unit='s', coerce=True)

print("Result without coerce: {}".format(timestamp1 == expected_result))
print("Result with coerce: {}".format(timestamp2 == expected_result))

pd.show_versions() output

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.19.0-65-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.2
setuptools: 19.2
Cython: 0.24.1
numpy: 1.11.1
scipy: 0.17.0
statsmodels: 0.6.1
xarray: None
IPython: 4.2.0
sphinx: 1.4.1
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: 1.1.0
tables: None
numexpr: 2.5.2
matplotlib: 1.5.1
openpyxl: None
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 0.999
httplib2: 0.8
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
boto: None
pandas_datareader: None

[chris-b1]

Problem is around here - there's some kind of fastpath that tries directly converting to integers, but doesn't seem to handle truncation correctly.
https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L2098

Note that you can also workaround this using the non-deprecated errors kwarg.

In [41]: pd.to_datetime(1.1, unit='s', errors='coerce')
Out[41]: Timestamp('1970-01-01 00:00:01.100000')

[lopezco]

Thank you the workaround!

[jreback]

@chris-b1

This should cast like this

 In [14]: np.array([1.1]).astype('i8', casting='no')
TypeError: Cannot cast array from dtype('float64') to dtype('int64') according to the rule 'no'

instead I think (so this will then hit the other path)