Skip to content

DOC: cleanup whatsnew entry for na_action='ignore' #52355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 4, 2023
Merged

DOC: cleanup whatsnew entry for na_action='ignore' #52355

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0b162f5
DOC: cleanup whatsnew entry for na_action='ignore'
topper-123 Apr 2, 2023
ed53473
Update doc/source/whatsnew/v2.1.0.rst
topper-123 Apr 3, 2023
acf7636
Update doc/source/whatsnew/v2.1.0.rst
topper-123 Apr 3, 2023
3710f37
Update doc/source/whatsnew/v2.1.0.rst
topper-123 Apr 3, 2023
305dd02
Update doc/source/whatsnew/v2.1.0.rst
topper-123 Apr 3, 2023
8d50b8f
update comment
topper-123 Apr 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 49 additions & 6 deletions doc/source/whatsnew/v2.1.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,51 @@ enhancement1

.. _whatsnew_210.enhancements.enhancement2:

enhancement2
^^^^^^^^^^^^
``map(func, na_action="ignore")`` now works for all array types
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When given a callable, :meth:`Series.map` applies the callable to all elements of the :class:`Series`.
Similarly, :meth:`DataFrame.applymap` applies the callable to all elements of the :class:`DataFrame`,
while :meth:`Index.map` applies the callable to all elements of the :class:`Index`.

Frequently, it is not desirable to apply the callable to nan-like values of the array and to avoid doing
that, the ``map`` method could be called with ``na_action="ignore"``, i.e. ``ser.map(func, na_action="ignore")``.
However, ``na_action="ignore"`` was not implemented for many ``ExtensionArray`` and ``Index`` types
Copy link
Member

@mroeschke mroeschke Apr 3, 2023 •
edited by topper-123
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you note specifically which types na_action is now consistently respected? (the datetime likes and period)?

I made a new version. I think its difficult to explain what didn't work correctly succintly. There were some patterns that worked correctly, other failed, while other gave a wrong result for the same ExtensionArray (e.g. the Categorical example below).

In short, .map could behave very erratic 😄.

and ``na_action="ignore"`` did not work correctly for any ``ExtensionArray`` subclass except the nullable numeric ones (i.e. with dtype :class:`Int64` etc.).

``na_action="ignore"`` now works for all array types (:issue:`52219`, :issue:`51645`, :issue:`51809`, :issue:`51936`, :issue:`52033`; :issue:`52096`).

*Previous behavior*:

.. code-block:: ipython

In [1]: ser = pd.Series(["a", "b", np.nan], dtype="category")
In [2]: ser.map(str.upper, na_action="ignore")
NotImplementedError
In [3]: df = pd.DataFrame(ser)
In [4]: df.applymap(str.upper, na_action="ignore") # worked for DataFrame
0
0 A
1 B
2 NaN
In [5]: idx = pd.Index(ser)
In [6]: idx.map(str.upper, na_action="ignore")
TypeError: CategoricalIndex.map() got an unexpected keyword argument 'na_action'

*New behavior*:

.. ipython:: python

ser = pd.Series(["a", "b", np.nan], dtype="category")
ser.map(str.upper, na_action="ignore")
df = pd.DataFrame(ser)
df.applymap(str.upper, na_action="ignore")
idx = pd.Index(ser)
idx.map(str.upper, na_action="ignore")

Also, note that :meth:`Categorical.map` implicitly has had its ``na_action`` set to ``"ignore"`` by default.
This has been deprecated and will :meth:`Categorical.map` in the future change the default
to ``na_action=None``, like for all the other array types.

.. _whatsnew_210.enhancements.other:

Expand All @@ -34,6 +77,7 @@ Other enhancements
- Implemented ``__pandas_priority__`` to allow custom types to take precedence over :class:`DataFrame`, :class:`Series`, :class:`Index`, or :class:`ExtensionArray` for arithmetic operations, :ref:`see the developer guide <extending.pandas_priority>` (:issue:`48347`)
- :meth:`MultiIndex.sort_values` now supports ``na_position`` (:issue:`51612`)
- :meth:`MultiIndex.sortlevel` and :meth:`Index.sortlevel` gained a new keyword ``na_position`` (:issue:`51612`)
- :meth:`arrays.DatetimeArray.map`, :meth:`arrays.TimedeltaArray.map` and :meth:`arrays.PeriodArray.map` can now take a ``na_action`` argument (:issue:`51644`)
- Improve error message when setting :class:`DataFrame` with wrong number of columns through :meth:`DataFrame.isetitem` (:issue:`51701`)
- Let :meth:`DataFrame.to_feather` accept a non-default :class:`Index` and non-string column names (:issue:`51787`)
- :class:`api.extensions.ExtensionArray` now has a :meth:`~api.extensions.ExtensionArray.map` method (:issue:`51809`)
Expand All @@ -42,7 +86,6 @@ Other enhancements
- Improved error message when creating a DataFrame with empty data (0 rows), no index and an incorrect number of columns. (:issue:`52084`)
- :meth:`DataFrame.applymap` now uses the :meth:`~api.extensions.ExtensionArray.map` method of underlying :class:`api.extensions.ExtensionArray` instances (:issue:`52219`)
- :meth:`arrays.SparseArray.map` now supports ``na_action`` (:issue:`52096`).
- :meth:`Categorical.map` and :meth:`CategoricalIndex.map` now have a ``na_action`` parameter (:issue:`44279`)
- Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)
-

Expand Down Expand Up @@ -163,15 +206,15 @@ Categorical
Datetimelike
^^^^^^^^^^^^
- Bug in :meth:`Timestamp.round` with values close to the implementation bounds returning incorrect results instead of raising ``OutOfBoundsDatetime`` (:issue:`51494`)
- :meth:`arrays.DatetimeArray.map` can now take a ``na_action`` argument. :meth:`DatetimeIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`)
- :meth:`DatetimeIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`)
- Bug in :meth:`arrays.DatetimeArray.map` and :meth:`DatetimeIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`)
-

Timedelta
^^^^^^^^^
- Bug in :meth:`Timedelta.round` with values close to the implementation bounds returning incorrect results instead of raising ``OutOfBoundsTimedelta`` (:issue:`51494`)
- Bug in :class:`TimedeltaIndex` division or multiplication leading to ``.freq`` of "0 Days" instead of ``None`` (:issue:`51575`)
- :meth:`arrays.TimedeltaArray.map` can now take a ``na_action`` argument. :meth:`TimedeltaIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`)
- :meth:`TimedeltaIndex.map` with ``na_action="ignore"`` now works as expected (:issue:`51644`)
- Bug in :meth:`arrays.TimedeltaArray.map` and :meth:`TimedeltaIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`)
-

Expand Down Expand Up @@ -227,7 +270,7 @@ I/O
Period
^^^^^^
- Bug in :class:`PeriodDtype` constructor failing to raise ``TypeError`` when no argument is passed or when ``None`` is passed (:issue:`27388`)
- :meth:`arrays.PeriodArray.map` can now take a ``na_action`` argument. :meth:`PeriodIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`)
- :meth:`PeriodIndex.map` with ``na_action="ignore"`` now works as expected (:issue:`51644`)
- Bug in :class:`PeriodDtype` constructor raising ``ValueError`` instead of ``TypeError`` when an invalid type is passed (:issue:`51790`)
- Bug in :meth:`arrays.PeriodArray.map` and :meth:`PeriodIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`)
-
Expand Down