ENH: allow opt-in to inferring pyarrow strings #54430
Conversation
| {"a": ["a", "b"]}, dtype="object", columns=Index(["a"], dtype=dtype) | ||
| ) | ||
| with pd.option_context("future.infer_string", True): | ||
| df = DataFrame({"a": ["a", "b"]}, dtype="object") |
There was a problem hiding this comment.
Can you add a case with null/nan/None in it?
| y,2""" | ||
| parser = all_parsers | ||
| if parser.engine == "pyarrow": | ||
| pytest.skip("TODO: Follow up") |
There was a problem hiding this comment.
Did we ever discuss pyarrow backed engines returning pyarrow types by default? I think from a user perspective it is less than ideal to have to specify both this option and the dtype backend for any pyarrow backed IO methods
There was a problem hiding this comment.
Let's move this discussion to Basel, but there is an issue about this: #51846
There was a problem hiding this comment.
Could you link the GH issue in this skip message?
There was a problem hiding this comment.
#54431 already addresses this
|
cc @jbrockmendel thoughts about the option name? |
I think future.infer_string is reasonable |
mroeschke
added
Strings
Co-authored-by: Matthew Roeschke <[email protected]>
| import pyarrow as pa | ||
|
|
||
| pa_dtype = pa.string() | ||
| dtype = ArrowDtype(pa_dtype) |
There was a problem hiding this comment.
are there any tests that hit this?
|
I’ll add this With the general arrow announcement |
|
Thanks @phofl |
* ENH: allow opt-in to inferring pyarrow strings * Remove comments and add tests * Add json tests * Update * Update pandas/_libs/lib.pyx Co-authored-by: Matthew Roeschke <[email protected]> * Update * Add test --------- Co-authored-by: Brock <[email protected]> Co-authored-by: Matthew Roeschke <[email protected]>
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.This is a simple opt-in for now without warning by default. Can adjust this for 2.2
cc @jbrockmendel