ENH: Implement arrow string option for various I/O methods #54431
Conversation
pandas/io/orc.py
Outdated
| @@ -132,7 +135,12 @@ def read_orc( | |||
| df = pa_table.to_pandas(types_mapper=mapping.get) | |||
| return df | |||
| else: | |||
| return pa_table.to_pandas() | |||
| print("Ts") | |||
mroeschke
added
IO Data
pandas/_libs/lib.pyx
Outdated
| elif seen.str_: | ||
| if is_string_array(objects): | ||
| from pandas._config import get_option | ||
| opt = get_option("future.infer_string") |
There was a problem hiding this comment.
Can we pass this in as a kwarg to maybe_convert_objects instead?
There was a problem hiding this comment.
I'd rather only get the option if actually needed
pandas/core/dtypes/cast.py
Outdated
| @@ -796,6 +798,12 @@ def infer_dtype_from_scalar(val) -> tuple[DtypeObj, Any]: | |||
| # coming out as np.str_! | |||
|
|
|||
| dtype = _dtype_obj | |||
| opt = get_option("future.infer_string") | |||
There was a problem hiding this comment.
follow-up. This is introduced in the other pr (little bit confusing, sorry)
| def arrow_string_types_mapper() -> Callable: | ||
| pa = import_optional_dependency("pyarrow") | ||
|
|
||
| return {pa.string(): pd.ArrowDtype(pa.string())}.get |
There was a problem hiding this comment.
Thinking about this a little, is there a situation where you would want to mix pyarrow and numpy dtypes?
(I'm thinking maybe we should force users to pick the pyarrow dtype backend if you are using the pyarrow string type)
There was a problem hiding this comment.
Yes there are a lot of situations.
NumPy numeric and Arrow strings is still the fastest, numpy numeric is 2D. Forcing them right now is not a good idea
pandas/io/pytables.py
Outdated
| @@ -3219,7 +3221,12 @@ def read( | |||
| self.validate_read(columns, where) | |||
| index = self.read_index("index", start=start, stop=stop) | |||
| values = self.read_array("values", start=start, stop=stop) | |||
| return Series(values, index=index, name=self.name, copy=False) | |||
| result = Series(values, index=index, name=self.name, copy=False) | |||
| if result.dtype.kind == "O" and using_pyarrow_string_dtype(): | |||
There was a problem hiding this comment.
Not too familiar with this code, but do we need to check if results is a string array first if doing this?
| dtype = pd.ArrowDtype(pa.string()) | ||
|
|
||
| data = """a,b | ||
| x,1 |
There was a problem hiding this comment.
Can you add a test case with null/nan/None like in your other PR?
There was a problem hiding this comment.
I can add a missing field, actually having these values doesn't make much sense
pandas/_libs/lib.pyx
Outdated
| @@ -2669,6 +2678,20 @@ def maybe_convert_objects(ndarray[object] objects, | |||
| return pi._data | |||
| seen.object_ = True | |||
|
|
|||
| elif seen.str_: | |||
| if is_string_array(objects): | |||
There was a problem hiding this comment.
I know everywhere else does this, but is there a way to avoid this double parsing?
(Maybe we check the other flags are all false?)
There was a problem hiding this comment.
No, you exit the first loop as soon as you find one string
|
Nice thanks @phofl |
…v#54431) * ENH: Implement arrow string option for various I/O methods * ENH: allow opt-in to inferring pyarrow strings * Remove comments and add tests * Add string option to arrow parsers * Update * Update * Adjust csv * Update * Update * Add test * Fix mypy --------- Co-authored-by: Brock <[email protected]>
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.