BUG: Fix some more arrow CSV tests #52087
Conversation
| @@ -158,6 +160,19 @@ def read(self) -> DataFrame: | |||
| parse_options=pyarrow_csv.ParseOptions(**self.parse_options), | |||
| convert_options=pyarrow_csv.ConvertOptions(**self.convert_options), | |||
| ) | |||
|
|
|||
| # Convert all pa.null() cols -> float64 | |||
| # TODO: There has to be a better way... right? | |||
There was a problem hiding this comment.
What I'm looking for here is something like a combination of select_dtypes + astype.
I couldn't find anything in the pyarrow docs, but maybe I'm just bad at Googling.
Sadly the types_mapper doesn't work, since you can only feed that EA types.
There was a problem hiding this comment.
Maybe @jorisvandenbossche knows a better way?
There was a problem hiding this comment.
I don't think there is an easier way. We currently don't have something like select_dtypes, so what you do (iterating over the schema, checking the type, and if needed adapt the new schema) seems the best option.
| dtype_backend = self.kwds["dtype_backend"] | ||
|
|
||
| # Convert all pa.null() cols -> float64 (non nullable) | ||
| # else Int64 (nullable case) |
There was a problem hiding this comment.
This was really confusing for me, but apparently even convert_dtypes on an all null column(float64) will change it to an Int64.
There was a problem hiding this comment.
I am not sure we should replicate that here.
In convert_dtypes, I suppose that is the consequence of trying to cast a float column where all non-null values can faithfully be represented as an integer. Which gives this corner case where this set of values is empty.
But if you don't know the dtype you are starting with (like here), I think using float is safer (it won't give problems later on if you then want to use it for floats and not just integers). Or actually using object dtype might even be safest (as that can represent everything), but of course we don't (yet) have a nullable object dtype, so that's probably not really an option.
There was a problem hiding this comment.
The problem is that the C and python parsers already convert to an Int64. Is this something that we need to fix before 2.0?
>>> import pandas as pd
>>> from io import StringIO
>>> f = StringIO("a\n1,")
>>> pd.read_csv(f, dtype_backend="numpy_nullable")
a
1 <NA>
>>> f.seek(0)
0
>>> pd.read_csv(f, dtype_backend="numpy_nullable").dtypes
a Int64
dtype: object
>>> f.seek(0)
0
>>> pd.read_csv(f)
a
1 NaN
>>> f.seek(0)
0
>>> pd.read_csv(f).dtypes
a float64
dtype: object
There was a problem hiding this comment.
We do this everywhere, so I am +1 on doing Int64 here as well. Initially this came from read_parquet I think
pandas/io/_util.py
Outdated
| @@ -8,6 +8,9 @@ | |||
| def _arrow_dtype_mapping() -> dict: | |||
| pa = import_optional_dependency("pyarrow") | |||
| return { | |||
| # All nulls should still give Float64 not object | |||
| # TODO: This breaks parquet | |||
| # pa.null(): pd.Float64Dtype(), | |||
There was a problem hiding this comment.
In what way does it break parquet?
There was a problem hiding this comment.
Well, first I hit this error.
>>> import pandas as pd
>>> import pyarrow as pa
>>> a = pa.nulls(0)
>>> pd.Float64Dtype().__from__arrow__(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Float64Dtype' object has no attribute '__from__arrow__'
After patching that(another PR incoming),
now an empty DataFrame of dtype object(pd.DataFrame({"value": pd.array([], dtype=object)})) returns a Float64Dtype when roundtripped.
Best guess here is that somehow the types_mapper is overriding the pandas metadata somehow.
|
Anyone want to have another look? |
| if dtype_backend != "pyarrow": | ||
| new_schema = table.schema | ||
| if dtype_backend == "numpy_nullable": | ||
| new_type = pa.int64() |
| "h": pd.Series( | ||
| [pd.NA if parser.engine != "pyarrow" else "", "a"], dtype="string" | ||
| ), | ||
| "h": pd.Series([pd.NA, "a"], dtype="string"), |
|
thx @lithomas1 |
|
@meeseeksdev backport 2.0.x. |
|
Owee, I'm MrMeeseeks, Look at me. There seem to be a conflict, please backport manually. Here are approximate instructions:
And apply the correct labels and milestones. Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon! Remember to remove the If these instructions are inaccurate, feel free to suggest an improvement. |
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.