Skip to content

BUG: formating integers datetimes using sql GH17855 #17882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Nov 22, 2017
Merged

BUG: formating integers datetimes using sql GH17855 #17882

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0498dd1
BUG: formating integers datetimes using sql GH17855
drorata Oct 15, 2017
385ad78
BUG: formating integers datetimes using sql GH17855
drorata Oct 15, 2017
e3a438b
Merge branch 'fix-17855' of github.com:drorata/pandas into fix-17855
drorata Nov 20, 2017
26454e7
Merge branch 'master' of git://github.com/pandas-dev/pandas into fix-…
drorata Nov 21, 2017
102ab25
Improve fix as per discussion
drorata Nov 21, 2017
5b56686
Improved testing
drorata Nov 21, 2017
987f588
Fixed typo in SQL data setting
drorata Nov 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions pandas/io/sql.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,12 @@ def _handle_date_column(col, utc=None, format=None):
if isinstance(format, dict):
return to_datetime(col, errors='ignore', **format)
else:
if format in ['D', 's', 'ms', 'us', 'ns']:
return to_datetime(col, errors='coerce', unit=format, utc=utc)
elif (issubclass(col.dtype.type, np.floating) or
issubclass(col.dtype.type, np.integer)):
# parse dates as timestamp
format = 's' if format is None else format
# Allow passing of formatting string for integers
# GH17855
if format is None and (issubclass(col.dtype.type, np.floating) or
issubclass(col.dtype.type, np.integer)):
format = 's'
if format in ['D', 'd', 'h', 'm', 's', 'ms', 'us', 'ns']:
return to_datetime(col, errors='coerce', unit=format, utc=utc)
elif is_datetime64tz_dtype(col):
# coerce to UTC timezone
Expand Down
50 changes: 39 additions & 11 deletions pandas/tests/io/test_sql.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@
"TextCol" TEXT,
"DateCol" TEXT,
"IntDateCol" INTEGER,
"IntDateOnlyCol" INTEGER,
"FloatCol" REAL,
"IntCol" INTEGER,
"BoolCol" INTEGER,
Expand All @@ -98,6 +99,7 @@
`TextCol` TEXT,
`DateCol` DATETIME,
`IntDateCol` INTEGER,
"IntDateOnlyCol" INTEGER,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are using a different quoting here, that's probably the reason for the failure

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'll fix it.

`FloatCol` DOUBLE,
`IntCol` INTEGER,
`BoolCol` BOOLEAN,
Expand All @@ -109,6 +111,7 @@
"DateCol" TIMESTAMP,
"DateColWithTz" TIMESTAMP WITH TIME ZONE,
"IntDateCol" INTEGER,
"IntDateOnlyCol" INTEGER,
"FloatCol" DOUBLE PRECISION,
"IntCol" INTEGER,
"BoolCol" BOOLEAN,
Expand All @@ -120,31 +123,33 @@
'sqlite': {
'query': """
INSERT INTO types_test_data
VALUES(?, ?, ?, ?, ?, ?, ?, ?)
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
'fields': (
'TextCol', 'DateCol', 'IntDateCol', 'FloatCol',
'IntCol', 'BoolCol', 'IntColWithNull', 'BoolColWithNull'
'TextCol', 'DateCol', 'IntDateCol', 'IntDateOnlyCol',
'FloatCol', 'IntCol', 'BoolCol', 'IntColWithNull',
'BoolColWithNull'
)
},
'mysql': {
'query': """
INSERT INTO types_test_data
VALUES("%s", %s, %s, %s, %s, %s, %s, %s)
VALUES("%s", %s, %s, %s, %s, %s, %s, %s, %s)
""",
'fields': (
'TextCol', 'DateCol', 'IntDateCol', 'FloatCol',
'IntCol', 'BoolCol', 'IntColWithNull', 'BoolColWithNull'
'TextCol', 'DateCol', 'IntDateCol', 'IntDateOnlyCol',
'FloatCol', 'IntCol', 'BoolCol', 'IntColWithNull',
'BoolColWithNull'
)
},
'postgresql': {
'query': """
INSERT INTO types_test_data
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s)
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
'fields': (
'TextCol', 'DateCol', 'DateColWithTz',
'IntDateCol', 'FloatCol',
'IntDateCol', 'IntDateOnlyCol', 'FloatCol',
'IntCol', 'BoolCol', 'IntColWithNull', 'BoolColWithNull'
)
},
Expand Down Expand Up @@ -313,13 +318,13 @@ def _load_raw_sql(self):
self.drop_table('types_test_data')
self._get_exec().execute(SQL_STRINGS['create_test_types'][self.flavor])
ins = SQL_STRINGS['insert_test_types'][self.flavor]

data = [
{
'TextCol': 'first',
'DateCol': '2000-01-03 00:00:00',
'DateColWithTz': '2000-01-01 00:00:00-08:00',
'IntDateCol': 535852800,
'IntDateOnlyCol': 20101010,
'FloatCol': 10.10,
'IntCol': 1,
'BoolCol': False,
Expand All @@ -331,6 +336,7 @@ def _load_raw_sql(self):
'DateCol': '2000-01-04 00:00:00',
'DateColWithTz': '2000-06-01 00:00:00-07:00',
'IntDateCol': 1356998400,
'IntDateOnlyCol': 20101212,
'FloatCol': 10.10,
'IntCol': 1,
'BoolCol': False,
Expand Down Expand Up @@ -610,20 +616,42 @@ def test_date_parsing(self):
df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
parse_dates=['DateCol'])
assert issubclass(df.DateCol.dtype.type, np.datetime64)
assert df.DateCol.tolist() == [
pd.Timestamp(2000, 1, 3, 0, 0, 0),
pd.Timestamp(2000, 1, 4, 0, 0, 0)
]

df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
parse_dates={'DateCol': '%Y-%m-%d %H:%M:%S'})
assert issubclass(df.DateCol.dtype.type, np.datetime64)
assert df.DateCol.tolist() == [
pd.Timestamp(2000, 1, 3, 0, 0, 0),
pd.Timestamp(2000, 1, 4, 0, 0, 0)
]

df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
parse_dates=['IntDateCol'])

assert issubclass(df.IntDateCol.dtype.type, np.datetime64)
assert df.IntDateCol.tolist() == [
pd.Timestamp(1986, 12, 25, 0, 0, 0),
pd.Timestamp(2013, 1, 1, 0, 0, 0)
]

df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
parse_dates={'IntDateCol': 's'})

assert issubclass(df.IntDateCol.dtype.type, np.datetime64)
assert df.IntDateCol.tolist() == [
pd.Timestamp(1986, 12, 25, 0, 0, 0),
pd.Timestamp(2013, 1, 1, 0, 0, 0)
]

df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
parse_dates={'IntDateOnlyCol': '%Y%m%d'})
assert issubclass(df.IntDateOnlyCol.dtype.type, np.datetime64)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback I included a test. I also noticed that the existing tests are not enough. It might be that the parsing returns a NaT which satisfies the classing condition but the values are wrong. This is the reason I added an explicit test that checks that the resulting values are correct. As a matter of fact while implementing the improvement suggested by @jorisvandenbossche the tests passed but the returned values were NaTs (the reason was a missing , in the code @jorisvandenbossche suggested)

Would you agree there's a problem with the tests?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, those are indeed not very thorough. Do you want to add such a similar check to the others as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorisvandenbossche Within this same PR?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that would be good yes (because as you said, with your changes you could actually silently 'break' them now)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorisvandenbossche I have included more explicit tests.

assert df.IntDateOnlyCol.tolist() == [
pd.Timestamp('2010-10-10'),
pd.Timestamp('2010-12-12')
]

def test_date_and_index(self):
# Test case where same column appears in parse_date and index_col
Expand Down