Merge remote-tracking branch 'upstream/main' into arrow/csv_arrow

Author: mroeschke

.pre-commit-config.yaml

@@ -230,7 +230,7 @@ repos:
         language: python
         additional_dependencies:
         - flake8==4.0.1
-        - flake8-pyi==22.5.1
+        - flake8-pyi==22.7.0
     -   id: future-annotations
         name: import annotations from __future__
         entry: 'from __future__ import annotations'

doc/source/ecosystem.rst

@@ -161,10 +161,10 @@ A good implementation for Python users is `has2k1/plotnine <https://github.com/h
 `IPython Vega <https://github.com/vega/ipyvega>`__ leverages `Vega
 <https://github.com/vega/vega>`__ to create plots within Jupyter Notebook.
 
-`Plotly <https://poltly.com/python>`__
+`Plotly <https://plotly.com/python>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-`Plotly’s <https://poltly.com/>`__ `Python API <https://poltly.com/python/>`__ enables interactive figures and web shareability. Maps, 2D, 3D, and live-streaming graphs are rendered with WebGL and `D3.js <https://d3js.org/>`__. The library supports plotting directly from a pandas DataFrame and cloud-based collaboration. Users of `matplotlib, ggplot for Python, and Seaborn <https://poltly.com/python/matplotlib-to-plotly-tutorial/>`__ can convert figures into interactive web-based plots. Plots can be drawn in `IPython Notebooks <https://plotly.com/ipython-notebooks/>`__ , edited with R or MATLAB, modified in a GUI, or embedded in apps and dashboards. Plotly is free for unlimited sharing, and has `offline <https://poltly.com/python/offline/>`__, or `on-premise <https://poltly.com/product/enterprise/>`__ accounts for private use.
+`Plotly’s <https://plotly.com/>`__ `Python API <https://plotly.com/python/>`__ enables interactive figures and web shareability. Maps, 2D, 3D, and live-streaming graphs are rendered with WebGL and `D3.js <https://d3js.org/>`__. The library supports plotting directly from a pandas DataFrame and cloud-based collaboration. Users of `matplotlib, ggplot for Python, and Seaborn <https://plotly.com/python/matplotlib-to-plotly-tutorial/>`__ can convert figures into interactive web-based plots. Plots can be drawn in `IPython Notebooks <https://plotly.com/ipython-notebooks/>`__ , edited with R or MATLAB, modified in a GUI, or embedded in apps and dashboards. Plotly is free for unlimited sharing, and has `offline <https://plotly.com/python/offline/>`__, or `on-premise <https://plotly.com/product/enterprise/>`__ accounts for private use.
 
 `Lux <https://github.com/lux-org/lux>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -591,12 +591,12 @@ Library            Accessor     Classes                              Description
 Development tools
 -----------------
 
-`pandas-stubs <https://github.com/VirtusLab/pandas-stubs>`__
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+`pandas-stubs <https://github.com/pandas-dev/pandas-stubs>`__
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 While pandas repository is partially typed, the package itself doesn't expose this information for external use.
 Install pandas-stubs to enable basic type coverage of pandas API.
 
 Learn more by reading through :issue:`14468`, :issue:`26766`, :issue:`28142`.
 
-See installation and usage instructions on the `github page <https://github.com/VirtusLab/pandas-stubs>`__.
+See installation and usage instructions on the `github page <https://github.com/pandas-dev/pandas-stubs>`__.

doc/source/getting_started/intro_tutorials/10_text_data.rst

@@ -179,7 +179,7 @@ applied to integers, so no ``str`` is used.
 
 Based on the index name of the row (``307``) and the column (``Name``),
 we can do a selection using the ``loc`` operator, introduced in the
-`tutorial on subsetting <3_subset_data.ipynb>`__.
+:ref:`tutorial on subsetting <10min_tut_03_subset>`.
 
 .. raw:: html
 

doc/source/user_guide/io.rst

@@ -107,9 +107,10 @@ index_col : int, str, sequence of int / str, or False, optional, default ``None`
   string name or column index. If a sequence of int / str is given, a
   MultiIndex is used.
 
-  Note: ``index_col=False`` can be used to force pandas to *not* use the first
-  column as the index, e.g. when you have a malformed file with delimiters at
-  the end of each line.
+  .. note::
+     ``index_col=False`` can be used to force pandas to *not* use the first
+     column as the index, e.g. when you have a malformed file with delimiters at
+     the end of each line.
 
   The default value of ``None`` instructs pandas to guess. If the number of
   fields in the column header row is equal to the number of fields in the body
@@ -182,15 +183,16 @@ General parsing configuration
 +++++++++++++++++++++++++++++
 
 dtype : Type name or dict of column -> type, default ``None``
-  Data type for data or columns. E.g. ``{'a': np.float64, 'b': np.int32}``
-  (unsupported with ``engine='python'``). Use ``str`` or ``object`` together
-  with suitable ``na_values`` settings to preserve and
-  not interpret dtype.
+  Data type for data or columns. E.g. ``{'a': np.float64, 'b': np.int32, 'c': 'Int64'}``
+  Use ``str`` or ``object`` together with suitable ``na_values`` settings to preserve
+  and not interpret dtype. If converters are specified, they will be applied INSTEAD
+  of dtype conversion.
+
   .. versionadded:: 1.5.0
 
-    Support for defaultdict was added. Specify a defaultdict as input where
-    the default determines the dtype of the columns which are not explicitly
-    listed.
+     Support for defaultdict was added. Specify a defaultdict as input where
+     the default determines the dtype of the columns which are not explicitly
+     listed.
 engine : {``'c'``, ``'python'``, ``'pyarrow'``}
   Parser engine to use. The C and pyarrow engines are faster, while the python engine
   is currently more feature-complete. Multithreading is currently only supported by
@@ -283,7 +285,9 @@ parse_dates : boolean or list of ints or names or list of lists or dict, default
   * If ``[[1, 3]]`` -> combine columns 1 and 3 and parse as a single date
     column.
   * If ``{'foo': [1, 3]}`` -> parse columns 1, 3 as date and call result 'foo'.
-    A fast-path exists for iso8601-formatted dates.
+
+  .. note::
+     A fast-path exists for iso8601-formatted dates.
 infer_datetime_format : boolean, default ``False``
   If ``True`` and parse_dates is enabled for a column, attempt to infer the
   datetime format to speed up the processing.
@@ -1593,8 +1597,10 @@ of multi-columns indices.
 
    pd.read_csv("mi2.csv", header=[0, 1], index_col=0)
 
-Note: If an ``index_col`` is not specified (e.g. you don't have an index, or wrote it
-with ``df.to_csv(..., index=False)``, then any ``names`` on the columns index will be *lost*.
+.. note::
+   If an ``index_col`` is not specified (e.g. you don't have an index, or wrote it
+   with ``df.to_csv(..., index=False)``, then any ``names`` on the columns index will
+   be *lost*.
 
 .. ipython:: python
    :suppress:

doc/source/whatsnew/v1.5.0.rst

@@ -802,6 +802,7 @@ Performance improvements
 - Performance improvement in datetime arrays string formatting when one of the default strftime formats ``"%Y-%m-%d %H:%M:%S"`` or ``"%Y-%m-%d %H:%M:%S.%f"`` is used. (:issue:`44764`)
 - Performance improvement in :meth:`Series.to_sql` and :meth:`DataFrame.to_sql` (:class:`SQLiteTable`) when processing time arrays. (:issue:`44764`)
 - Performance improvements to :func:`read_sas` (:issue:`47403`, :issue:`47404`, :issue:`47405`)
+- Performance improvement in ``argmax`` and ``argmin`` for :class:`arrays.SparseArray` (:issue:`34197`)
 -
 
 .. ---------------------------------------------------------------------------

pandas/_libs/algos.pyi

@@ -1,5 +1,3 @@
-from __future__ import annotations
-
 from typing import Any
 
 import numpy as np

pandas/_libs/interval.pyi

@@ -1,5 +1,3 @@
-from __future__ import annotations
-
 from typing import (
     Any,
     Generic,
@@ -84,7 +82,7 @@ class Interval(IntervalMixin, Generic[_OrderableT]):
         self: Interval[_OrderableTimesT], key: _OrderableTimesT
     ) -> bool: ...
     @overload
-    def __contains__(self: Interval[_OrderableScalarT], key: int | float) -> bool: ...
+    def __contains__(self: Interval[_OrderableScalarT], key: float) -> bool: ...
     @overload
     def __add__(
         self: Interval[_OrderableTimesT], y: Timedelta
@@ -94,7 +92,7 @@ class Interval(IntervalMixin, Generic[_OrderableT]):
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __add__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __add__(self: Interval[float], y: float) -> Interval[float]: ...
     @overload
     def __radd__(
         self: Interval[_OrderableTimesT], y: Timedelta
@@ -104,7 +102,7 @@ class Interval(IntervalMixin, Generic[_OrderableT]):
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __radd__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __radd__(self: Interval[float], y: float) -> Interval[float]: ...
     @overload
     def __sub__(
         self: Interval[_OrderableTimesT], y: Timedelta
@@ -114,7 +112,7 @@ class Interval(IntervalMixin, Generic[_OrderableT]):
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __sub__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __sub__(self: Interval[float], y: float) -> Interval[float]: ...
     @overload
     def __rsub__(
         self: Interval[_OrderableTimesT], y: Timedelta
@@ -124,31 +122,31 @@ class Interval(IntervalMixin, Generic[_OrderableT]):
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __rsub__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __rsub__(self: Interval[float], y: float) -> Interval[float]: ...
     @overload
     def __mul__(
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __mul__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __mul__(self: Interval[float], y: float) -> Interval[float]: ...
     @overload
     def __rmul__(
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __rmul__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __rmul__(self: Interval[float], y: float) -> Interval[float]: ...
     @overload
     def __truediv__(
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __truediv__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __truediv__(self: Interval[float], y: float) -> Interval[float]: ...
     @overload
     def __floordiv__(
         self: Interval[int], y: _OrderableScalarT
     ) -> Interval[_OrderableScalarT]: ...
     @overload
-    def __floordiv__(self: Interval[float], y: int | float) -> Interval[float]: ...
+    def __floordiv__(self: Interval[float], y: float) -> Interval[float]: ...
     def overlaps(self: Interval[_OrderableT], other: Interval[_OrderableT]) -> bool: ...
 
 def intervals_to_interval_bounds(

pandas/_libs/join.pyi

@@ -55,7 +55,7 @@ def asof_join_backward_on_X_by_Y(
     left_by_values: np.ndarray,  # by_t[:]
     right_by_values: np.ndarray,  # by_t[:]
     allow_exact_matches: bool = ...,
-    tolerance: np.number | int | float | None = ...,
+    tolerance: np.number | float | None = ...,
     use_hashtable: bool = ...,
 ) -> tuple[npt.NDArray[np.intp], npt.NDArray[np.intp]]: ...
 def asof_join_forward_on_X_by_Y(
@@ -64,7 +64,7 @@ def asof_join_forward_on_X_by_Y(
     left_by_values: np.ndarray,  # by_t[:]
     right_by_values: np.ndarray,  # by_t[:]
     allow_exact_matches: bool = ...,
-    tolerance: np.number | int | float | None = ...,
+    tolerance: np.number | float | None = ...,
     use_hashtable: bool = ...,
 ) -> tuple[npt.NDArray[np.intp], npt.NDArray[np.intp]]: ...
 def asof_join_nearest_on_X_by_Y(
@@ -73,6 +73,6 @@ def asof_join_nearest_on_X_by_Y(
     left_by_values: np.ndarray,  # by_t[:]
     right_by_values: np.ndarray,  # by_t[:]
     allow_exact_matches: bool = ...,
-    tolerance: np.number | int | float | None = ...,
+    tolerance: np.number | float | None = ...,
     use_hashtable: bool = ...,
 ) -> tuple[npt.NDArray[np.intp], npt.NDArray[np.intp]]: ...

pandas/_libs/json.pyi

@@ -12,7 +12,7 @@ def dumps(
     date_unit: str = ...,
     iso_dates: bool = ...,
     default_handler: None
-    | Callable[[Any], str | int | float | bool | list | dict | None] = ...,
+    | Callable[[Any], str | float | bool | list | dict | None] = ...,
 ) -> str: ...
 def loads(
     s: str,

pandas/_libs/tslibs/offsets.pyi

@@ -1,5 +1,3 @@
-from __future__ import annotations
-
 from datetime import (
     datetime,
     timedelta,

pandas/_libs/tslibs/timedeltas.pyi

@@ -86,7 +86,7 @@ class Timedelta(timedelta):
         cls: type[_S],
         value=...,
         unit: str = ...,
-        **kwargs: int | float | np.integer | np.floating,
+        **kwargs: float | np.integer | np.floating,
     ) -> _S: ...
     # GH 46171
     # While Timedelta can return pd.NaT, having the constructor return
@@ -123,7 +123,7 @@ class Timedelta(timedelta):
     @overload  # type: ignore[override]
     def __floordiv__(self, other: timedelta) -> int: ...
     @overload
-    def __floordiv__(self, other: int | float) -> Timedelta: ...
+    def __floordiv__(self, other: float) -> Timedelta: ...
     @overload
     def __floordiv__(
         self, other: npt.NDArray[np.timedelta64]

pandas/_libs/tslibs/timestamps.pyi

@@ -33,13 +33,7 @@ class Timestamp(datetime):
     value: int  # np.int64
     def __new__(
         cls: type[_DatetimeT],
-        ts_input: int
-        | np.integer
-        | float
-        | str
-        | _date
-        | datetime
-        | np.datetime64 = ...,
+        ts_input: np.integer | float | str | _date | datetime | np.datetime64 = ...,
         freq: int | None | str | BaseOffset = ...,
         tz: str | _tzinfo | None | int = ...,
         unit: str | int | None = ...,

pandas/_libs/writers.pyi

@@ -1,5 +1,3 @@
-from __future__ import annotations
-
 import numpy as np
 
 from pandas._typing import ArrayLike

pandas/_typing.py

@@ -82,7 +82,7 @@
 
 # scalars
 
-PythonScalar = Union[str, int, float, bool]
+PythonScalar = Union[str, float, bool]
 DatetimeLikeScalar = Union["Period", "Timestamp", "Timedelta"]
 PandasScalar = Union["Period", "Timestamp", "Timedelta", "Interval"]
 Scalar = Union[PythonScalar, PandasScalar, np.datetime64, np.timedelta64, datetime]
@@ -92,10 +92,10 @@
 # timestamp and timedelta convertible types
 
 TimestampConvertibleTypes = Union[
-    "Timestamp", datetime, np.datetime64, int, np.int64, float, str
+    "Timestamp", datetime, np.datetime64, np.int64, float, str
 ]
 TimedeltaConvertibleTypes = Union[
-    "Timedelta", timedelta, np.timedelta64, int, np.int64, float, str
+    "Timedelta", timedelta, np.timedelta64, np.int64, float, str
 ]
 Timezone = Union[str, tzinfo]
 
@@ -126,7 +126,7 @@
 ]
 
 # dtypes
-NpDtype = Union[str, np.dtype, type_t[Union[str, float, int, complex, bool, object]]]
+NpDtype = Union[str, np.dtype, type_t[Union[str, complex, bool, object]]]
 Dtype = Union["ExtensionDtype", NpDtype]
 AstypeArg = Union["ExtensionDtype", "npt.DTypeLike"]
 # DtypeArg specifies all allowable dtypes in a functions its dtype argument

pandas/core/arrays/datetimelike.py

@@ -2170,11 +2170,11 @@ def validate_periods(periods: None) -> None:
 
 
 @overload
-def validate_periods(periods: int | float) -> int:
+def validate_periods(periods: float) -> int:
     ...
 
 
-def validate_periods(periods: int | float | None) -> int | None:
+def validate_periods(periods: float | None) -> int | None:
     """
     If a `periods` argument is passed to the Datetime/Timedelta Array/Index
     constructor, cast it to an integer.

pandas/core/arrays/sparse/array.py

@@ -42,7 +42,10 @@
 from pandas.compat.numpy import function as nv
 from pandas.errors import PerformanceWarning
 from pandas.util._exceptions import find_stack_level
-from pandas.util._validators import validate_insert_loc
+from pandas.util._validators import (
+    validate_bool_kwarg,
+    validate_insert_loc,
+)
 
 from pandas.core.dtypes.astype import astype_nansafe
 from pandas.core.dtypes.cast import (
@@ -1646,6 +1649,45 @@ def _min_max(self, kind: Literal["min", "max"], skipna: bool) -> Scalar:
         else:
             return na_value_for_dtype(self.dtype.subtype, compat=False)
 
+    def _argmin_argmax(self, kind: Literal["argmin", "argmax"]) -> int:
+
+        values = self._sparse_values
+        index = self._sparse_index.indices
+        mask = np.asarray(isna(values))
+        func = np.argmax if kind == "argmax" else np.argmin
+
+        idx = np.arange(values.shape[0])
+        non_nans = values[~mask]
+        non_nan_idx = idx[~mask]
+
+        _candidate = non_nan_idx[func(non_nans)]
+        candidate = index[_candidate]
+
+        if isna(self.fill_value):
+            return candidate
+        if kind == "argmin" and self[candidate] < self.fill_value:
+            return candidate
+        if kind == "argmax" and self[candidate] > self.fill_value:
+            return candidate
+        _loc = self._first_fill_value_loc()
+        if _loc == -1:
+            # fill_value doesn't exist
+            return candidate
+        else:
+            return _loc
+
+    def argmax(self, skipna: bool = True) -> int:
+        validate_bool_kwarg(skipna, "skipna")
+        if not skipna and self._hasna:
+            raise NotImplementedError
+        return self._argmin_argmax("argmax")
+
+    def argmin(self, skipna: bool = True) -> int:
+        validate_bool_kwarg(skipna, "skipna")
+        if not skipna and self._hasna:
+            raise NotImplementedError
+        return self._argmin_argmax("argmin")
+
     # ------------------------------------------------------------------------
     # Ufuncs
     # ------------------------------------------------------------------------

pandas/core/frame.py

@@ -2563,7 +2563,7 @@ def to_stata(
         compression: CompressionOptions = "infer",
         storage_options: StorageOptions = None,
         *,
-        value_labels: dict[Hashable, dict[float | int, str]] | None = None,
+        value_labels: dict[Hashable, dict[float, str]] | None = None,
     ) -> None:
         """
         Export DataFrame object to Stata dta format.