Skip to content

Remove Sequence[str] as type used in DataFrame.to_string() #47233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 24, 2023
Merged

Remove Sequence[str] as type used in DataFrame.to_string() #47233

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
08fb709
API: Disallow sets as index and columns argument in DataFrame constru…
Dr-Irv Jun 4, 2022
3e49016
TYP: Remove Sequence[str] from to_string, to_html
Dr-Irv Jun 4, 2022
a9e2849
use list[str] for header argument. Remove dict as index and columns …
Dr-Irv Jun 5, 2022
d65511c
Merge branch 'main' into typtostring
jreback Jun 10, 2022
d7d82f5
merge with main
Dr-Irv Oct 4, 2022
ca9b7ac
use Axes rather than Sequence[Hashable]
Dr-Irv Oct 4, 2022
6da7d4f
fix up whatsnew - no entry
Dr-Irv Oct 4, 2022
bc33a68
merge in main
Dr-Irv Feb 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.5.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ Other enhancements
- Added ``numeric_only`` argument to :meth:`Resampler.sum`, :meth:`Resampler.prod`, :meth:`Resampler.min`, :meth:`Resampler.max`, :meth:`Resampler.first`, and :meth:`Resampler.last` (:issue:`46442`)
- ``times`` argument in :class:`.ExponentialMovingWindow` now accepts ``np.timedelta64`` (:issue:`47003`)
- :class:`DataError`, :class:`SpecificationError`, :class:`SettingWithCopyError`, and :class:`SettingWithCopyWarning` are now exposed in ``pandas.errors`` (:issue:`27656`)
- :class:`DataFrame` constructor raises if ``index`` or ``columns`` arguments are sets (:issue:`47215`)

.. ---------------------------------------------------------------------------
.. _whatsnew_150.notable_bug_fixes:
Expand Down
3 changes: 1 addition & 2 deletions pandas/_typing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
TYPE_CHECKING,
Any,
Callable,
Collection,
Dict,
Hashable,
Iterator,
Expand Down Expand Up @@ -115,7 +114,7 @@
Ordered = Optional[bool]
JSONSerializable = Optional[Union[PythonScalar, List, Dict]]
Frequency = Union[str, "DateOffset"]
Axes = Collection[Any]
Axes = Union[AnyArrayLike, List, Dict, range]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

List and Dict are probably too restrictive in general? If we use this in places where we accept list-like or dict-like.

Also from https://github.com/python/typeshed/blob/master/CONTRIBUTING.md#conventions

avoid invariant collection types (list, dict) in argument positions, in favor of covariant types like Mapping or Sequence;

This also helps us develop robust code since mypy will report if we try to write into the passed argument.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only getting applied to the index and columns argument of pd.DataFrame. We can't use Sequence, as that would allow a string. We document the arguments as Index or array-like. I don't think List is too restrictive - the problem is that Sequence is too broad because of the issue with strings. I could see removing Dict since we don't document that it is acceptable.


RandomState = Union[
int,
Expand Down
41 changes: 19 additions & 22 deletions pandas/core/frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,12 @@ def __init__(

manager = get_option("mode.data_manager")

# GH47215
if index is not None and isinstance(index, set):
raise ValueError("index cannot be a set")
if columns is not None and isinstance(columns, set):
raise ValueError("columns cannot be a set")

if copy is None:
if isinstance(data, dict):
# retain pre-GH#38939 default behavior
Expand Down Expand Up @@ -730,10 +736,7 @@ def __init__(
if not isinstance(data, np.ndarray) and treat_as_nested(data):
# exclude ndarray as we may have cast it a few lines above
if columns is not None:
# error: Argument 1 to "ensure_index" has incompatible type
# "Collection[Any]"; expected "Union[Union[Union[ExtensionArray,
# ndarray], Index, Series], Sequence[Any]]"
columns = ensure_index(columns) # type: ignore[arg-type]
columns = ensure_index(columns)
arrays, columns, index = nested_data_to_arrays(
# error: Argument 3 to "nested_data_to_arrays" has incompatible
# type "Optional[Collection[Any]]"; expected "Optional[Index]"
Expand Down Expand Up @@ -771,14 +774,8 @@ def __init__(
if index is None or columns is None:
raise ValueError("DataFrame constructor not properly called!")

# Argument 1 to "ensure_index" has incompatible type "Collection[Any]";
# expected "Union[Union[Union[ExtensionArray, ndarray],
# Index, Series], Sequence[Any]]"
index = ensure_index(index) # type: ignore[arg-type]
# Argument 1 to "ensure_index" has incompatible type "Collection[Any]";
# expected "Union[Union[Union[ExtensionArray, ndarray],
# Index, Series], Sequence[Any]]"
columns = ensure_index(columns) # type: ignore[arg-type]
index = ensure_index(index)
columns = ensure_index(columns)

if not dtype:
dtype, _ = infer_dtype_from_scalar(data, pandas_dtype=True)
Expand Down Expand Up @@ -1108,9 +1105,9 @@ def _repr_html_(self) -> str | None:
def to_string(
self,
buf: None = ...,
columns: Sequence[str] | None = ...,
columns: Axes | None = ...,
col_space: int | list[int] | dict[Hashable, int] | None = ...,
header: bool | Sequence[str] = ...,
header: bool | list = ...,
index: bool = ...,
na_rep: str = ...,
formatters: fmt.FormattersType | None = ...,
Expand All @@ -1133,9 +1130,9 @@ def to_string(
def to_string(
self,
buf: FilePath | WriteBuffer[str],
columns: Sequence[str] | None = ...,
columns: Axes | None = ...,
col_space: int | list[int] | dict[Hashable, int] | None = ...,
header: bool | Sequence[str] = ...,
header: bool | list = ...,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid invariant collection types (list, dict) in argument positions, in favor of covariant types like Mapping or Sequence;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue Sequence allows a regular string to be passed, which is invalid.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing it to list[str]

index: bool = ...,
na_rep: str = ...,
formatters: fmt.FormattersType | None = ...,
Expand All @@ -1155,8 +1152,8 @@ def to_string(
...

@Substitution(
header_type="bool or sequence of str",
header="Write out the column names. If a list of strings "
header_type="bool or array-like of column names",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my comment #47215 (comment) about the (imo) vague definition of array-like. what about list-like?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we do accept a Series or Index here, which I guess is "list-like", so I could do that instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we do accept a Series or Index here, which I guess is "list-like", so I could do that instead.

Turns out we don't accept a Series or Index, but do accept a numpy array, but I think we're better off documenting it as a list, and restricting it to a list[str] . It does have to be a list of strings.

header="Write out the column names. If a list of columns "
"is given, it is assumed to be aliases for the "
"column names",
col_space_type="int, list or dict of int",
Expand All @@ -1168,9 +1165,9 @@ def to_string(
def to_string(
self,
buf: FilePath | WriteBuffer[str] | None = None,
columns: Sequence[str] | None = None,
columns: Axes | None = None,
col_space: int | list[int] | dict[Hashable, int] | None = None,
header: bool | Sequence[str] = True,
header: bool | list = True,
index: bool = True,
na_rep: str = "NaN",
formatters: fmt.FormattersType | None = None,
Expand Down Expand Up @@ -2914,9 +2911,9 @@ def to_parquet(
def to_html(
self,
buf: FilePath | WriteBuffer[str] | None = None,
columns: Sequence[str] | None = None,
columns: Axes | None = None,
col_space: ColspaceArgType | None = None,
header: bool | Sequence[str] = True,
header: bool = True,
index: bool = True,
na_rep: str = "NaN",
formatters: FormattersType | None = None,
Expand Down
4 changes: 2 additions & 2 deletions pandas/core/indexes/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@
tz_compare,
)
from pandas._typing import (
AnyArrayLike,
ArrayLike,
Axes,
Dtype,
DtypeObj,
F,
Expand Down Expand Up @@ -7273,7 +7273,7 @@ def ensure_index_from_sequences(sequences, names=None) -> Index:
return MultiIndex.from_arrays(sequences, names=names)


def ensure_index(index_like: AnyArrayLike | Sequence, copy: bool = False) -> Index:
def ensure_index(index_like: Axes, copy: bool = False) -> Index:
"""
Ensure that we have an index from some index-like object.

Expand Down
2 changes: 1 addition & 1 deletion pandas/io/formats/csvs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ def decimal(self) -> str:
return self.fmt.decimal

@property
def header(self) -> bool | Sequence[str]:
def header(self) -> bool | list:
return self.fmt.header

@property
Expand Down
9 changes: 5 additions & 4 deletions pandas/io/formats/format.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
from pandas._libs.tslibs.nattype import NaTType
from pandas._typing import (
ArrayLike,
Axes,
ColspaceArgType,
ColspaceType,
CompressionOptions,
Expand Down Expand Up @@ -119,7 +120,7 @@
----------
buf : str, Path or StringIO-like, optional, default None
Buffer to write to. If None, the output is returned as a string.
columns : sequence, optional, default None
columns : array-like, optional, default None
The subset of columns to write. Writes all columns by default.
col_space : %(col_space_type)s, optional
%(col_space)s.
Expand Down Expand Up @@ -561,9 +562,9 @@ class DataFrameFormatter:
def __init__(
self,
frame: DataFrame,
columns: Sequence[str] | None = None,
columns: Axes | None = None,
col_space: ColspaceArgType | None = None,
header: bool | Sequence[str] = True,
header: bool | list = True,
index: bool = True,
na_rep: str = "NaN",
formatters: FormattersType | None = None,
Expand Down Expand Up @@ -683,7 +684,7 @@ def _initialize_justify(self, justify: str | None) -> str:
else:
return justify

def _initialize_columns(self, columns: Sequence[str] | None) -> Index:
def _initialize_columns(self, columns: Axes | None) -> Index:
if columns is not None:
cols = ensure_index(columns)
self.frame = self.frame[cols]
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/frame/test_constructors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2976,6 +2976,14 @@ def test_construction_from_ndarray_with_eadtype_mismatched_columns(self):
with pytest.raises(ValueError, match=msg):
DataFrame(arr2, columns=["foo", "bar"])

def test_columns_indexes_raise_on_sets(self):
# GH 47215
data = [[1, 2, 3], [4, 5, 6]]
with pytest.raises(ValueError, match="index cannot be a set"):
DataFrame(data, index={"a", "b"})
with pytest.raises(ValueError, match="columns cannot be a set"):
DataFrame(data, columns={"a", "b", "c"})


def get1(obj): # TODO: make a helper in tm?
if isinstance(obj, Series):
Expand Down