Merge branch 'main' into dev/depr/literal-str-read_xml

Author: rmhowe425

ci/code_checks.sh

@@ -105,17 +105,6 @@ if [[ -z "$CHECK" || "$CHECK" == "docstrings" ]]; then
         pandas.errors.UnsupportedFunctionCall \
         pandas.test \
         pandas.NaT \
-        pandas.SparseDtype \
-        pandas.DatetimeTZDtype.unit \
-        pandas.DatetimeTZDtype.tz \
-        pandas.PeriodDtype.freq \
-        pandas.IntervalDtype.subtype \
-        pandas_dtype \
-        pandas.api.types.is_bool \
-        pandas.api.types.is_complex \
-        pandas.api.types.is_float \
-        pandas.api.types.is_integer \
-        pandas.api.types.pandas_dtype \
         pandas.read_clipboard \
         pandas.ExcelFile \
         pandas.ExcelFile.parse \

doc/source/whatsnew/v2.0.3.rst

@@ -14,6 +14,7 @@ including other versions of pandas.
 Fixed regressions
 ~~~~~~~~~~~~~~~~~
 - Fixed performance regression in merging on datetime-like columns (:issue:`53231`)
+- Fixed regression when :meth:`DataFrame.to_string` creates extra space for string dtypes (:issue:`52690`)
 - For external ExtensionArray implementations, restored the default use of ``_values_for_factorize`` for hashing arrays (:issue:`53475`)
 -
 

doc/source/whatsnew/v2.1.0.rst

@@ -366,12 +366,14 @@ Datetimelike
 - :meth:`DatetimeIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`)
 - Bug in :class:`DateOffset` which had inconsistent behavior when multiplying a :class:`DateOffset` object by a constant (:issue:`47953`)
 - Bug in :func:`date_range` when ``freq`` was a :class:`DateOffset` with ``nanoseconds`` (:issue:`46877`)
+- Bug in :meth:`DataFrame.to_sql` raising ``ValueError`` for pyarrow-backed date like dtypes (:issue:`53854`)
 - Bug in :meth:`Timestamp.date`, :meth:`Timestamp.isocalendar`, :meth:`Timestamp.timetuple`, and :meth:`Timestamp.toordinal` were returning incorrect results for inputs outside those supported by the Python standard library's datetime module (:issue:`53668`)
 - Bug in :meth:`Timestamp.round` with values close to the implementation bounds returning incorrect results instead of raising ``OutOfBoundsDatetime`` (:issue:`51494`)
 - Bug in :meth:`arrays.DatetimeArray.map` and :meth:`DatetimeIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`)
 - Bug in constructing a :class:`Series` or :class:`DataFrame` from a datetime or timedelta scalar always inferring nanosecond resolution instead of inferring from the input (:issue:`52212`)
 - Bug in parsing datetime strings with weekday but no day e.g. "2023 Sept Thu" incorrectly raising ``AttributeError`` instead of ``ValueError`` (:issue:`52659`)
 
+
 Timedelta
 ^^^^^^^^^
 - :meth:`TimedeltaIndex.map` with ``na_action="ignore"`` now works as expected (:issue:`51644`)
@@ -495,6 +497,7 @@ Reshaping
 - Bug in :func:`crosstab` when ``dropna=False`` would not keep ``np.nan`` in the result (:issue:`10772`)
 - Bug in :func:`merge_asof` raising ``KeyError`` for extension dtypes (:issue:`52904`)
 - Bug in :func:`merge_asof` raising ``ValueError`` for data backed by read-only ndarrays (:issue:`53513`)
+- Bug in :func:`merge_asof` with ``left_index=True`` or ``right_index=True`` with mismatched index dtypes giving incorrect results in some cases instead of raising ``MergeError`` (:issue:`53870`)
 - Bug in :meth:`DataFrame.agg` and :meth:`Series.agg` on non-unique columns would return incorrect type when dist-like argument passed in (:issue:`51099`)
 - Bug in :meth:`DataFrame.combine_first` ignoring other's columns if ``other`` is empty (:issue:`53792`)
 - Bug in :meth:`DataFrame.idxmin` and :meth:`DataFrame.idxmax`, where the axis dtype would be lost for empty frames (:issue:`53265`)
@@ -503,6 +506,7 @@ Reshaping
 - Bug in :meth:`DataFrame.stack` sorting columns lexicographically (:issue:`53786`)
 - Bug in :meth:`DataFrame.transpose` inferring dtype for object column (:issue:`51546`)
 - Bug in :meth:`Series.combine_first` converting ``int64`` dtype to ``float64`` and losing precision on very large integers (:issue:`51764`)
+-
 
 Sparse
 ^^^^^^
@@ -537,11 +541,12 @@ Other
 - Bug in :func:`assert_almost_equal` now throwing assertion error for two unequal sets (:issue:`51727`)
 - Bug in :func:`assert_frame_equal` checks category dtypes even when asked not to check index type (:issue:`52126`)
 - Bug in :meth:`DataFrame.reindex` with a ``fill_value`` that should be inferred with a :class:`ExtensionDtype` incorrectly inferring ``object`` dtype (:issue:`52586`)
+- Bug in :meth:`DataFrame.shift` and :meth:`Series.shift` when passing both "freq" and "fill_value" silently ignoring "fill_value" instead of raising ``ValueError`` (:issue:`53832`)
+- Bug in :meth:`DataFrame.shift` with ``axis=1`` on a :class:`DataFrame` with a single :class:`ExtensionDtype` column giving incorrect results (:issue:`53832`)
 - Bug in :meth:`Series.align`, :meth:`DataFrame.align`, :meth:`Series.reindex`, :meth:`DataFrame.reindex`, :meth:`Series.interpolate`, :meth:`DataFrame.interpolate`, incorrectly failing to raise with method="asfreq" (:issue:`53620`)
 - Bug in :meth:`Series.map` when giving a callable to an empty series, the returned series had ``object`` dtype. It now keeps the original dtype (:issue:`52384`)
 - Bug in :meth:`Series.memory_usage` when ``deep=True`` throw an error with Series of objects and the returned value is incorrect, as it does not take into account GC corrections (:issue:`51858`)
 - Fixed incorrect ``__name__`` attribute of ``pandas._libs.json`` (:issue:`52898`)
--
 
 .. ***DO NOT USE THIS SECTION***
 

pandas/_libs/lib.pyx

@@ -1056,6 +1056,14 @@ def is_float(obj: object) -> bool:
     Returns
     -------
     bool
+
+    Examples
+    --------
+    >>> pd.api.types.is_float(1.0)
+    True
+
+    >>> pd.api.types.is_float(1)
+    False
     """
     return util.is_float_object(obj)
 
@@ -1067,6 +1075,14 @@ def is_integer(obj: object) -> bool:
     Returns
     -------
     bool
+
+    Examples
+    --------
+    >>> pd.api.types.is_integer(1)
+    True
+
+    >>> pd.api.types.is_integer(1.0)
+    False
     """
     return util.is_integer_object(obj)
 
@@ -1089,6 +1105,14 @@ def is_bool(obj: object) -> bool:
     Returns
     -------
     bool
+
+    Examples
+    --------
+    >>> pd.api.types.is_bool(True)
+    True
+
+    >>> pd.api.types.is_bool(1)
+    False
     """
     return util.is_bool_object(obj)
 
@@ -1100,6 +1124,14 @@ def is_complex(obj: object) -> bool:
     Returns
     -------
     bool
+
+    Examples
+    --------
+    >>> pd.api.types.is_complex(1 + 1j)
+    True
+
+    >>> pd.api.types.is_complex(1)
+    False
     """
     return util.is_complex_object(obj)
 

pandas/_libs/tslibs/period.pyx

@@ -2674,7 +2674,7 @@ class Period(_Period):
     freq : str, default None
         One of pandas period strings or corresponding objects. Accepted
         strings are listed in the
-        :ref:`offset alias section <timeseries.offset_aliases>` in the user docs.
+        :ref:`period alias section <timeseries.period_aliases>` in the user docs.
         If value is datetime, freq is required.
     ordinal : int, default None
         The period offset from the proleptic Gregorian epoch.

pandas/core/algorithms.py

@@ -450,6 +450,9 @@ def unique_with_mask(values, mask: npt.NDArray[np.bool_] | None = None):
 unique1d = unique
 
 
+_MINIMUM_COMP_ARR_LEN = 1_000_000
+
+
 def isin(comps: ListLike, values: ListLike) -> npt.NDArray[np.bool_]:
     """
     Compute the isin boolean array.
@@ -518,7 +521,7 @@ def isin(comps: ListLike, values: ListLike) -> npt.NDArray[np.bool_]:
     # Albeit hashmap has O(1) look-up (vs. O(logn) in sorted array),
     # in1d is faster for small sizes
     if (
-        len(comps_array) > 1_000_000
+        len(comps_array) > _MINIMUM_COMP_ARR_LEN
         and len(values) <= 26
         and comps_array.dtype != object
     ):

pandas/core/apply.py

@@ -735,7 +735,7 @@ def apply(self) -> DataFrame | Series:
             with np.errstate(all="ignore"):
                 results = self.obj._mgr.apply("apply", func=self.func)
             # _constructor will retain self.index and self.columns
-            return self.obj._constructor(data=results)
+            return self.obj._constructor_from_mgr(results, axes=results.axes)
 
         # broadcasting
         if self.result_type == "broadcast":

pandas/core/arraylike.py

@@ -349,7 +349,7 @@ def _reconstruct(result):
             return result
         if isinstance(result, BlockManager):
             # we went through BlockManager.apply e.g. np.sqrt
-            result = self._constructor(result, **reconstruct_kwargs, copy=False)
+            result = self._constructor_from_mgr(result, axes=result.axes)
         else:
             # we converted an array, lost our axes
             result = self._constructor(

pandas/core/arrays/datetimelike.py

@@ -2235,7 +2235,7 @@ def interpolate(
         *,
         method,
         axis: int,
-        index: Index | None,
+        index: Index,
         limit,
         limit_direction,
         limit_area,
@@ -2255,7 +2255,7 @@ def interpolate(
         else:
             out_data = self._ndarray.copy()
 
-        missing.interpolate_array_2d(
+        missing.interpolate_2d_inplace(
             out_data,
             method=method,
             axis=axis,

pandas/core/arrays/numpy_.py

@@ -1,6 +1,9 @@
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+from typing import (
+    TYPE_CHECKING,
+    Literal,
+)
 
 import numpy as np
 
@@ -32,6 +35,7 @@
     from pandas._typing import (
         AxisInt,
         Dtype,
+        FillnaOptions,
         NpDtype,
         Scalar,
         Self,
@@ -224,12 +228,42 @@ def _values_for_factorize(self) -> tuple[np.ndarray, float | None]:
             fv = np.nan
         return self._ndarray, fv
 
+    def pad_or_backfill(
+        self,
+        *,
+        method: FillnaOptions,
+        axis: int,
+        limit: int | None,
+        limit_area: Literal["inside", "outside"] | None = None,
+        copy: bool = True,
+    ) -> Self:
+        """
+        ffill or bfill
+        """
+        if copy:
+            out_data = self._ndarray.copy()
+        else:
+            out_data = self._ndarray
+
+        meth = missing.clean_fill_method(method)
+        missing.pad_or_backfill_inplace(
+            out_data,
+            method=meth,
+            axis=axis,
+            limit=limit,
+            limit_area=limit_area,
+        )
+
+        if not copy:
+            return self
+        return type(self)._simple_new(out_data, dtype=self.dtype)
+
     def interpolate(
         self,
         *,
         method,
         axis: int,
-        index: Index | None,
+        index: Index,
         limit,
         limit_direction,
         limit_area,
@@ -246,7 +280,8 @@ def interpolate(
         else:
             out_data = self._ndarray.copy()
 
-        missing.interpolate_array_2d(
+        # TODO: assert we have floating dtype?
+        missing.interpolate_2d_inplace(
             out_data,
             method=method,
             axis=axis,

pandas/core/arrays/sparse/array.py

@@ -79,7 +79,7 @@
     check_array_indexer,
     unpack_tuple_and_ellipses,
 )
-from pandas.core.missing import interpolate_2d
+from pandas.core.missing import pad_or_backfill_inplace
 from pandas.core.nanops import check_below_min_count
 
 from pandas.io.formats import printing
@@ -764,11 +764,11 @@ def fillna(
                 stacklevel=find_stack_level(),
             )
             new_values = np.asarray(self)
-            # interpolate_2d modifies new_values inplace
-            # error: Argument "method" to "interpolate_2d" has incompatible type
-            # "Literal['backfill', 'bfill', 'ffill', 'pad']"; expected
+            # pad_or_backfill_inplace modifies new_values inplace
+            # error: Argument "method" to "pad_or_backfill_inplace" has incompatible
+            # type "Literal['backfill', 'bfill', 'ffill', 'pad']"; expected
             # "Literal['pad', 'backfill']"
-            interpolate_2d(
+            pad_or_backfill_inplace(
                 new_values, method=method, limit=limit  # type: ignore[arg-type]
             )
             return type(self)(new_values, fill_value=self.fill_value)

pandas/core/dtypes/common.py

@@ -1603,6 +1603,11 @@ def pandas_dtype(dtype) -> DtypeObj:
     Raises
     ------
     TypeError if not a dtype
+
+    Examples
+    --------
+    >>> pd.api.types.pandas_dtype(int)
+    dtype('int64')
     """
     # short-circuit
     if isinstance(dtype, np.ndarray):

pandas/core/dtypes/dtypes.py

@@ -697,16 +697,17 @@ class DatetimeTZDtype(PandasExtensionDtype):
 
     Raises
     ------
-    pytz.UnknownTimeZoneError
+    ZoneInfoNotFoundError
         When the requested timezone cannot be found.
 
     Examples
     --------
-    >>> pd.DatetimeTZDtype(tz='UTC')
+    >>> from zoneinfo import ZoneInfo
+    >>> pd.DatetimeTZDtype(tz=ZoneInfo('UTC'))
     datetime64[ns, UTC]
 
-    >>> pd.DatetimeTZDtype(tz='dateutil/US/Central')
-    datetime64[ns, tzfile('/usr/share/zoneinfo/US/Central')]
+    >>> pd.DatetimeTZDtype(tz=ZoneInfo('Europe/Paris'))
+    datetime64[ns, Europe/Paris]
     """
 
     type: type[Timestamp] = Timestamp
@@ -772,13 +773,27 @@ def _creso(self) -> int:
     def unit(self) -> str_type:
         """
         The precision of the datetime data.
+
+        Examples
+        --------
+        >>> from zoneinfo import ZoneInfo
+        >>> dtype = pd.DatetimeTZDtype(tz=ZoneInfo('America/Los_Angeles'))
+        >>> dtype.unit
+        'ns'
         """
         return self._unit
 
     @property
     def tz(self) -> tzinfo:
         """
         The timezone.
+
+        Examples
+        --------
+        >>> from zoneinfo import ZoneInfo
+        >>> dtype = pd.DatetimeTZDtype(tz=ZoneInfo('America/Los_Angeles'))
+        >>> dtype.tz
+        zoneinfo.ZoneInfo(key='America/Los_Angeles')
         """
         return self._tz
 
@@ -967,6 +982,12 @@ def __reduce__(self):
     def freq(self):
         """
         The frequency object of this PeriodDtype.
+
+        Examples
+        --------
+        >>> dtype = pd.PeriodDtype(freq='D')
+        >>> dtype.freq
+        <Day>
         """
         return self._freq
 
@@ -1217,6 +1238,12 @@ def closed(self) -> IntervalClosedType:
     def subtype(self):
         """
         The dtype of the Interval bounds.
+
+        Examples
+        --------
+        >>> dtype = pd.IntervalDtype(subtype='int64', closed='both')
+        >>> dtype.subtype
+        dtype('int64')
         """
         return self._subtype
 
@@ -1565,6 +1592,17 @@ class SparseDtype(ExtensionDtype):
     Methods
     -------
     None
+
+    Examples
+    --------
+    >>> ser = pd.Series([1, 0, 0], dtype=pd.SparseDtype(dtype=int, fill_value=0))
+    >>> ser
+    0    1
+    1    0
+    2    0
+    dtype: Sparse[int64, 0]
+    >>> ser.sparse.density
+    0.3333333333333333
     """
 
     # We include `_is_na_fill_value` in the metadata to avoid hash collisions