Merge remote-tracking branch 'upstream/main' into allow-mixed-iso

Author: MarcoGorelli

.github/workflows/sdist.yml

@@ -92,5 +92,4 @@ jobs:
     - name: Import pandas
       run: |
         cd ..
-        conda list
         python -c "import pandas; pandas.show_versions();"

.github/workflows/ubuntu.yml

@@ -73,7 +73,6 @@ jobs:
             env_file: actions-pypy-38.yaml
             pattern: "not slow and not network and not single_cpu"
             test_args: "--max-worker-restart 0"
-            error_on_warnings: "0"
           - name: "Numpy Dev"
             env_file: actions-310-numpydev.yaml
             pattern: "not slow and not network and not single_cpu"

doc/source/development/maintaining.rst

@@ -349,10 +349,10 @@ The release process makes a snapshot of pandas (a git commit) available to users
 a particular version number. After the release the new pandas version will be available
 in the next places:
 
-- Git repo with a [new tag](https://github.com/pandas-dev/pandas/tags)
-- Source distribution in a [GitHub release](https://github.com/pandas-dev/pandas/releases)
-- Pip packages in the [PyPI](https://pypi.org/project/pandas/)
-- Conda/Mamba packages in [conda-forge](https://anaconda.org/conda-forge/pandas)
+- Git repo with a `new tag <https://github.com/pandas-dev/pandas/tags>`_
+- Source distribution in a `GitHub release <https://github.com/pandas-dev/pandas/releases>`_
+- Pip packages in the `PyPI <https://pypi.org/project/pandas/>`_
+- Conda/Mamba packages in `conda-forge <https://anaconda.org/conda-forge/pandas>`_
 
 The process for releasing a new version of pandas is detailed next section.
 
@@ -368,11 +368,11 @@ Prerequisites
 
 In order to be able to release a new pandas version, the next permissions are needed:
 
-- Merge rights to the [pandas](https://github.com/pandas-dev/pandas/),
-  [pandas-wheels](https://github.com/MacPython/pandas-wheels), and
-  [pandas-feedstock](https://github.com/conda-forge/pandas-feedstock/) repositories.
+- Merge rights to the `pandas <https://github.com/pandas-dev/pandas/>`_,
+  `pandas-wheels <https://github.com/MacPython/pandas-wheels>`_, and
+  `pandas-feedstock <https://github.com/conda-forge/pandas-feedstock/>`_ repositories.
 - Permissions to push to main in the pandas repository, to push the new tags.
-- Write permissions to [PyPI](https://github.com/conda-forge/pandas-feedstock/pulls)
+- `Write permissions to PyPI <https://github.com/conda-forge/pandas-feedstock/pulls>`_
 - Access to the social media accounts, to publish the announcements.
 
 Pre-release
@@ -408,7 +408,7 @@ Pre-release
 Release
 ```````
 
-1. Create an empty commit and a tag in the last commit of the branch to be released:
+1. Create an empty commit and a tag in the last commit of the branch to be released::
 
     git checkout <branch>
     git pull --ff-only upstream <branch>
@@ -423,7 +423,7 @@ which will be triggered when the tag is pushed.
 2. Only if the release is a release candidate, we want to create a new branch for it, immediately
    after creating the tag. For example, if we are releasing pandas 1.4.0rc0, we would like to
    create the branch 1.4.x to backport commits to the 1.4 versions. As well as create a tag to
-   mark the start of the development of 1.5.0 (assuming it is the next version):
+   mark the start of the development of 1.5.0 (assuming it is the next version)::
 
     git checkout -b 1.4.x
     git push upstream 1.4.x
@@ -436,7 +436,7 @@ which will be triggered when the tag is pushed.
 
     ./setup.py sdist --formats=gztar --quiet
 
-4. Create a [new GitHub release](https://github.com/pandas-dev/pandas/releases/new):
+4. Create a `new GitHub release <https://github.com/pandas-dev/pandas/releases/new>`_:
 
    - Title: ``Pandas <version>``
    - Tag: ``<version>``
@@ -447,13 +447,13 @@ which will be triggered when the tag is pushed.
      (e.g. releasing 1.4.5 after 1.5 has been released)
 
 5. The GitHub release will after some hours trigger an
-   [automated conda-forge PR](https://github.com/conda-forge/pandas-feedstock/pulls).
+   `automated conda-forge PR <https://github.com/conda-forge/pandas-feedstock/pulls>`_.
    Merge it once the CI is green, and it will generate the conda-forge packages.
 
 6. Packages for supported versions in PyPI are built in the
-   [MacPython repo](https://github.com/MacPython/pandas-wheels).
+   `MacPython repo <https://github.com/MacPython/pandas-wheels>`_.
    Open a PR updating the build commit to the released version, and merge it once the
-   CI is green.
+   CI is green. To do this type::
 
     git checkout master
     git pull --ff-only upstream master
@@ -486,7 +486,7 @@ Post-Release
 4. Create a new issue for the next release, with the estimated date of release.
 
 5. Open a PR with the placeholder for the release notes of the next version. See
-   for example [the PR for 1.5.3](https://github.com/pandas-dev/pandas/pull/49843/files).
+   for example `the PR for 1.5.3 <https://github.com/pandas-dev/pandas/pull/49843/files>`_.
 
 6. Announce the new release in the official channels (use previous announcements
    for reference):

doc/source/reference/testing.rst

@@ -28,6 +28,7 @@ Exceptions and warnings
    errors.AccessorRegistrationWarning
    errors.AttributeConflictWarning
    errors.CategoricalConversionWarning
+   errors.ChainedAssignmentError
    errors.ClosedFileError
    errors.CSSWarning
    errors.DatabaseError

doc/source/whatsnew/v2.0.0.rst

@@ -49,6 +49,16 @@ The ``use_nullable_dtypes`` keyword argument has been expanded to the following
 * :func:`read_feather`
 * :func:`to_numeric`
 
+To simplify opting-in to nullable dtypes for these functions, a new option ``nullable_dtypes`` was added that allows setting
+the keyword argument globally to ``True`` if not specified directly. The option can be enabled
+through:
+
+.. ipython:: python
+
+    pd.options.mode.nullable_dtypes = True
+
+The option will only work for functions with the keyword ``use_nullable_dtypes``.
+
 Additionally a new global configuration, ``mode.dtype_backend`` can now be used in conjunction with the parameter ``use_nullable_dtypes=True`` in the following functions
 to select the nullable dtypes implementation.
 
@@ -125,6 +135,14 @@ Copy-on-Write improvements
   a modification to the data happens) when constructing a Series from an existing
   Series with the default of ``copy=False`` (:issue:`50471`)
 
+- Trying to set values using chained assignment (for example, ``df["a"][1:3] = 0``)
+  will now always raise an exception when Copy-on-Write is enabled. In this mode,
+  chained assignment can never work because we are always setting into a temporary
+  object that is the result of an indexing operation (getitem), which under
+  Copy-on-Write always behaves as a copy. Thus, assigning through a chain
+  can never update the original Series or DataFrame. Therefore, an informative
+  error is raised to the user instead of silently doing nothing (:issue:`49467`)
+
 Copy-on-Write can be enabled through
 
 .. code-block:: python
@@ -608,6 +626,7 @@ Other API changes
   methods to get a full slice (for example ``df.loc[:]`` or ``df[:]``) (:issue:`49469`)
 - Disallow computing ``cumprod`` for :class:`Timedelta` object; previously this returned incorrect values (:issue:`50246`)
 - Loading a JSON file with duplicate columns using ``read_json(orient='split')`` renames columns to avoid duplicates, as :func:`read_csv` and the other readers do (:issue:`50370`)
+- The levels of the index of the :class:`Series` returned from ``Series.sparse.from_coo`` now always have dtype ``int32``. Previously they had dtype ``int64`` (:issue:`50926`)
 - :func:`to_datetime` with ``unit`` of either "Y" or "M" will now raise if a sequence contains a non-round ``float`` value, matching the ``Timestamp`` behavior (:issue:`50301`)
 -
 
@@ -623,6 +642,7 @@ Deprecations
 - :meth:`Index.is_floating` has been deprecated. Use :func:`pandas.api.types.is_float_dtype` instead (:issue:`50042`)
 - :meth:`Index.holds_integer` has been deprecated. Use :func:`pandas.api.types.infer_dtype` instead (:issue:`50243`)
 - :meth:`Index.is_categorical` has been deprecated. Use :func:`pandas.api.types.is_categorical_dtype` instead (:issue:`50042`)
+- :meth:`Index.is_interval` has been deprecated. Use :func:`pandas.api.types.is_intterval_dtype` instead (:issue:`50042`)
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_200.prior_deprecations:
@@ -906,6 +926,7 @@ Performance improvements
 - Performance improvements to :func:`read_sas` (:issue:`47403`, :issue:`47405`, :issue:`47656`, :issue:`48502`)
 - Memory improvement in :meth:`RangeIndex.sort_values` (:issue:`48801`)
 - Performance improvement in :meth:`Series.to_numpy` if ``copy=True`` by avoiding copying twice (:issue:`24345`)
+- Performance improvement in :meth:`Series.rename` with :class:`MultiIndex` (:issue:`21055`)
 - Performance improvement in :class:`DataFrameGroupBy` and :class:`SeriesGroupBy` when ``by`` is a categorical type and ``sort=False`` (:issue:`48976`)
 - Performance improvement in :class:`DataFrameGroupBy` and :class:`SeriesGroupBy` when ``by`` is a categorical type and ``observed=False`` (:issue:`49596`)
 - Performance improvement in :func:`read_stata` with parameter ``index_col`` set to ``None`` (the default). Now the index will be a :class:`RangeIndex` instead of :class:`Int64Index` (:issue:`49745`)
@@ -961,6 +982,8 @@ Datetimelike
 - Bug in :func:`Timestamp.utctimetuple` raising a ``TypeError`` (:issue:`32174`)
 - Bug in :func:`to_datetime` was raising ``ValueError`` when parsing mixed-offset :class:`Timestamp` with ``errors='ignore'`` (:issue:`50585`)
 - Bug in :func:`to_datetime` was incorrectly handling floating-point inputs within 1 ``unit`` of the overflow boundaries (:issue:`50183`)
+- Bug in :func:`to_datetime` with unit of "Y" or "M" giving incorrect results, not matching pointwise :class:`Timestamp` results (:issue:`50870`)
+-
 
 Timedelta
 ^^^^^^^^^
@@ -1026,6 +1049,7 @@ Indexing
 - Bug in :meth:`DataFrame.iloc` raising ``IndexError`` when indexer is a :class:`Series` with numeric extension array dtype (:issue:`49521`)
 - Bug in :func:`~DataFrame.describe` when formatting percentiles in the resulting index showed more decimals than needed (:issue:`46362`)
 - Bug in :meth:`DataFrame.compare` does not recognize differences when comparing ``NA`` with value in nullable dtypes (:issue:`48939`)
+- Bug in :meth:`Series.rename` with :class:`MultiIndex` losing extension array dtypes (:issue:`21055`)
 - Bug in :meth:`DataFrame.isetitem` coercing extension array dtypes in :class:`DataFrame` to object (:issue:`49922`)
 - Bug in :class:`BusinessHour` would cause creation of :class:`DatetimeIndex` to fail when no opening hour was included in the index (:issue:`49835`)
 -

pandas/_config/__init__.py

@@ -33,3 +33,8 @@
 def using_copy_on_write():
     _mode_options = _global_config["mode"]
     return _mode_options["copy_on_write"] and _mode_options["data_manager"] == "block"
+
+
+def using_nullable_dtypes():
+    _mode_options = _global_config["mode"]
+    return _mode_options["nullable_dtypes"]

pandas/_libs/tslib.pyx

@@ -220,19 +220,6 @@ def format_array_from_datetime(
     return result
 
 
-cdef int64_t _wrapped_cast_from_unit(object val, str unit) except? -1:
-    """
-    Call cast_from_unit and re-raise OverflowError as OutOfBoundsDatetime
-    """
-    # See also timedeltas._maybe_cast_from_unit
-    try:
-        return cast_from_unit(val, unit)
-    except OverflowError as err:
-        raise OutOfBoundsDatetime(
-            f"cannot convert input {val} with the unit '{unit}'"
-        ) from err
-
-
 def array_with_unit_to_datetime(
     ndarray[object] values,
     str unit,
@@ -302,7 +289,7 @@ def array_with_unit_to_datetime(
                 if val != val or val == NPY_NAT:
                     iresult[i] = NPY_NAT
                 else:
-                    iresult[i] = _wrapped_cast_from_unit(val, unit)
+                    iresult[i] = cast_from_unit(val, unit)
 
             elif isinstance(val, str):
                 if len(val) == 0 or val in nat_strings:
@@ -317,7 +304,7 @@ def array_with_unit_to_datetime(
                             f"non convertible value {val} with the unit '{unit}'"
                         )
 
-                    iresult[i] = _wrapped_cast_from_unit(fval, unit)
+                    iresult[i] = cast_from_unit(fval, unit)
 
             else:
                 # TODO: makes more sense as TypeError, but that would be an
@@ -362,7 +349,7 @@ cdef _array_with_unit_to_datetime_object_fallback(ndarray[object] values, str un
             else:
                 try:
                     oresult[i] = Timestamp(val, unit=unit)
-                except OverflowError:
+                except OutOfBoundsDatetime:
                     oresult[i] = val
 
         elif isinstance(val, str):

pandas/_libs/tslibs/conversion.pyx

@@ -108,22 +108,41 @@ cdef int64_t cast_from_unit(object ts, str unit) except? -1:
     if ts is None:
         return m
 
-    if unit in ["Y", "M"] and  is_float_object(ts) and not ts.is_integer():
-        # GH#47267 it is clear that 2 "M" corresponds to 1970-02-01,
-        #  but not clear what 2.5 "M" corresponds to, so we will
-        #  disallow that case.
-        raise ValueError(
-            f"Conversion of non-round float with unit={unit} "
-            "is ambiguous"
-        )
+    if unit in ["Y", "M"]:
+        if is_float_object(ts) and not ts.is_integer():
+            # GH#47267 it is clear that 2 "M" corresponds to 1970-02-01,
+            #  but not clear what 2.5 "M" corresponds to, so we will
+            #  disallow that case.
+            raise ValueError(
+                f"Conversion of non-round float with unit={unit} "
+                "is ambiguous"
+            )
+        # GH#47266 go through np.datetime64 to avoid weird results e.g. with "Y"
+        #  and 150 we'd get 2120-01-01 09:00:00
+        if is_float_object(ts):
+            ts = int(ts)
+        dt64obj = np.datetime64(ts, unit)
+        return get_datetime64_nanos(dt64obj, NPY_FR_ns)
 
     # cast the unit, multiply base/frace separately
     # to avoid precision issues from float -> int
-    base = <int64_t>ts
+    try:
+        base = <int64_t>ts
+    except OverflowError as err:
+        raise OutOfBoundsDatetime(
+            f"cannot convert input {ts} with the unit '{unit}'"
+        ) from err
+
     frac = ts - base
     if p:
         frac = round(frac, p)
-    return <int64_t>(base * m) + <int64_t>(frac * m)
+
+    try:
+        return <int64_t>(base * m) + <int64_t>(frac * m)
+    except OverflowError as err:
+        raise OutOfBoundsDatetime(
+            f"cannot convert input {ts} with the unit '{unit}'"
+        ) from err
 
 
 cpdef inline (int64_t, int) precision_from_unit(str unit):
@@ -278,25 +297,13 @@ cdef _TSObject convert_to_tsobject(object ts, tzinfo tz, str unit,
         if ts == NPY_NAT:
             obj.value = NPY_NAT
         else:
-            if unit in ["Y", "M"]:
-                # GH#47266 cast_from_unit leads to weird results e.g. with "Y"
-                #  and 150 we'd get 2120-01-01 09:00:00
-                ts = np.datetime64(ts, unit)
-                return convert_to_tsobject(ts, tz, None, False, False)
-
-            ts = ts * cast_from_unit(None, unit)
+            ts = cast_from_unit(ts, unit)
             obj.value = ts
             pandas_datetime_to_datetimestruct(ts, NPY_FR_ns, &obj.dts)
     elif is_float_object(ts):
         if ts != ts or ts == NPY_NAT:
             obj.value = NPY_NAT
         else:
-            if unit in ["Y", "M"]:
-                if ts == int(ts):
-                    # GH#47266 Avoid cast_from_unit, which would give weird results
-                    #  e.g. with "Y" and 150.0 we'd get 2120-01-01 09:00:00
-                    return convert_to_tsobject(int(ts), tz, unit, False, False)
-
             ts = cast_from_unit(ts, unit)
             obj.value = ts
             pandas_datetime_to_datetimestruct(ts, NPY_FR_ns, &obj.dts)

pandas/_libs/tslibs/timedeltas.pyx

@@ -373,7 +373,7 @@ cdef _maybe_cast_from_unit(ts, str unit):
     #  assert unit not in ["Y", "y", "M"]
     try:
         ts = cast_from_unit(ts, unit)
-    except OverflowError as err:
+    except OutOfBoundsDatetime as err:
         raise OutOfBoundsTimedelta(
             f"Cannot cast {ts} from {unit} to 'ns' without overflow."
         ) from err