You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When reading from SQL queries - especially when reading chunk-wise - issues with type conversion can occur.
It would be ideal if one could specify the types of the columns, so that type conversions go right.
A very common case is reading pure float results chunk-wise from a large table. The nuisance comes in when and chunk contains NULL values only in a column. In such cases these NULL values are stored as None rather than float("nan"). I see that this is an issue with inconsistent NULL types in pandas and missing type information in read_sql_query.
Another case could be when text columns happen to contain numbers only in a chunk.
Specifying types for a query manually could resolve these issues.
The text was updated successfully, but these errors were encountered:
TomAugspurger
changed the title
Add control over types in read_sql_query (to resolve NULL inconsistency)
Add dtype keyword to read_sql_query to control per column dtypes.
Jun 6, 2019
When reading from SQL queries - especially when reading chunk-wise - issues with type conversion can occur.
It would be ideal if one could specify the types of the columns, so that type conversions go right.
A very common case is reading pure float results chunk-wise from a large table. The nuisance comes in when and chunk contains NULL values only in a column. In such cases these NULL values are stored as
None
rather thanfloat("nan")
. I see that this is an issue with inconsistent NULL types in pandas and missing type information in read_sql_query.Another case could be when text columns happen to contain numbers only in a chunk.
Specifying types for a query manually could resolve these issues.
The text was updated successfully, but these errors were encountered: