v2.0.0.rst

What's new in 2.0.0 (??)

These are the changes in pandas 2.0.0. See :ref:`release` for a full changelog including other versions of pandas.

{{ header }}

Enhancements

Optional dependencies version management

Optional pandas dependencies can be managed as extras in a requirements/setup file, for example:

pandas[performance, aws]>=2.0.0

Available optional dependencies (listed in order of appearance at install guide) are [all, performance, computation, timezone, fss, aws, gcp, excel, parquet, feather, hdf5, spss, postgresql, mysql, sql-other, html, xml, plot, output_formatting, clipboard, compression, test] (:issue:`39164`).

enhancement2

Other enhancements

• :func:`read_sas` now supports using encoding='infer' to correctly read and use the encoding specified by the sas file. (:issue:`48048`)
• :meth:`.DataFrameGroupBy.quantile` and :meth:`.SeriesGroupBy.quantile` now preserve nullable dtypes instead of casting to numpy dtypes (:issue:`37493`)
• :meth:`Series.add_suffix`, :meth:`DataFrame.add_suffix`, :meth:`Series.add_prefix` and :meth:`DataFrame.add_prefix` support an axis argument. If axis is set, the default behaviour of which axis to consider can be overwritten (:issue:`47819`)
• :func:`assert_frame_equal` now shows the first element where the DataFrames differ, analogously to pytest's output (:issue:`47910`)
• Added new argument use_nullable_dtypes to :func:`read_csv` and :func:`read_excel` to enable automatic conversion to nullable dtypes (:issue:`36712`)
• Added new global configuration, io.nullable_backend to allow use_nullable_dtypes=True to return pyarrow-backed dtypes when set to "pyarrow" in :func:`read_parquet` (:issue:`48957`)
• Added index parameter to :meth:`DataFrame.to_dict` (:issue:`46398`)
• Added metadata propagation for binary operators on :class:`DataFrame` (:issue:`28283`)
• :class:`.CategoricalConversionWarning`, :class:`.InvalidComparison`, :class:`.InvalidVersion`, :class:`.LossySetitemError`, and :class:`.NoBufferPresent` are now exposed in pandas.errors (:issue:`27656`)
• Fix test optional_extra by adding missing test package pytest-asyncio (:issue:`48361`)
• :func:`DataFrame.astype` exception message thrown improved to include column name when type conversion is not possible. (:issue:`47571`)
• :meth:`DataFrame.to_json` now supports a mode keyword with supported inputs 'w' and 'a'. Defaulting to 'w', 'a' can be used when lines=True and orient='records' to append record oriented json lines to an existing json file. (:issue:`35849`)

Notable bug fixes

These are bug fixes that might have notable behavior changes.

:meth:`.GroupBy.cumsum` and :meth:`.GroupBy.cumprod` overflow instead of lossy casting to float

In previous versions we cast to float when applying cumsum and cumprod which lead to incorrect results even if the result could be hold by int64 dtype. Additionally, the aggregation overflows consistent with numpy and the regular :meth:`DataFrame.cumprod` and :meth:`DataFrame.cumsum` methods when the limit of int64 is reached (:issue:`37493`).

Old Behavior

In [1]: df = pd.DataFrame({"key": ["b"] * 7, "value": 625})
In [2]: df.groupby("key")["value"].cumprod()[5]
Out[2]: 5.960464477539062e+16

We return incorrect results with the 6th value.

New Behavior

.. ipython:: python

    df = pd.DataFrame({"key": ["b"] * 7, "value": 625})
    df.groupby("key")["value"].cumprod()

We overflow with the 7th value, but the 6th value is still correct.

notable_bug_fix2

Backwards incompatible API changes

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated. If installed, we now require:

Package	Minimum Version	Required	Changed
mypy (dev)	0.981		X
python-dateutil	2.8.2	X	X

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package	Minimum Version	Changed
pyarrow	6.0.0	X

See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.

Other API changes

• Passing nanoseconds greater than 999 or less than 0 in :class:`Timestamp` now raises a ValueError (:issue:`48538`, :issue:`48255`)
• :func:`read_csv`: specifying an incorrect number of columns with index_col of now raises ParserError instead of IndexError when using the c parser.
• Default value of dtype in :func:`get_dummies` is changed to bool from uint8 (:issue:`45848`)
• :meth:`DataFrame.astype`, :meth:`Series.astype`, and :meth:`DatetimeIndex.astype` casting datetime64 data to any of "datetime64[s]", "datetime64[ms]", "datetime64[us]" will return an object with the given resolution instead of coercing back to "datetime64[ns]" (:issue:`48928`)
• :meth:`DataFrame.astype`, :meth:`Series.astype`, and :meth:`DatetimeIndex.astype` casting timedelta64 data to any of "timedelta64[s]", "timedelta64[ms]", "timedelta64[us]" will return an object with the given resolution instead of coercing to "float64" dtype (:issue:`48963`)
• Passing data with dtype of "timedelta64[s]", "timedelta64[ms]", or "timedelta64[us]" to :class:`TimedeltaIndex`, :class:`Series`, or :class:`DataFrame` constructors will now retain that dtype instead of casting to "timedelta64[ns]"; timedelta64 data with lower resolution will be cast to the lowest supported resolution "timedelta64[s]" (:issue:`49014`)
• Passing dtype of "timedelta64[s]", "timedelta64[ms]", or "timedelta64[us]" to :class:`TimedeltaIndex`, :class:`Series`, or :class:`DataFrame` constructors will now retain that dtype instead of casting to "timedelta64[ns]"; passing a dtype with lower resolution for :class:`Series` or :class:`DataFrame` will be cast to the lowest supported resolution "timedelta64[s]" (:issue:`49014`)
• Passing a np.datetime64 object with non-nanosecond resolution to :class:`Timestamp` will retain the input resolution if it is "s", "ms", or "ns"; otherwise it will be cast to the closest supported resolution (:issue:`49008`)
• The other argument in :meth:`DataFrame.mask` and :meth:`Series.mask` now defaults to no_default instead of np.nan consistent with :meth:`DataFrame.where` and :meth:`Series.where`. Entries will be filled with the corresponding NULL value (np.nan for numpy dtypes, pd.NA for extension dtypes). (:issue:`49111`)
• When creating a :class:`Series` with a object-dtype :class:`Index` of datetime objects, pandas no longer silently converts the index to a :class:`DatetimeIndex` (:issue:`39307`, :issue:`23598`)
• :meth:`Series.unique` with dtype "timedelta64[ns]" or "datetime64[ns]" now returns :class:`TimedeltaArray` or :class:`DatetimeArray` instead of numpy.ndarray (:issue:`49176`)

Deprecations

Removal of prior version deprecations/changes

• Removed deprecated :meth:`Categorical.to_dense`, use np.asarray(cat) instead (:issue:`32639`)
• Removed deprecated :meth:`Categorical.take_nd` (:issue:`27745`)
• Removed deprecated :meth:`Categorical.mode`, use Series(cat).mode() instead (:issue:`45033`)
• Removed deprecated :meth:`Categorical.is_dtype_equal` and :meth:`CategoricalIndex.is_dtype_equal` (:issue:`37545`)
• Removed deprecated :meth:`CategoricalIndex.take_nd` (:issue:`30702`)
• Removed deprecated :meth:`Index.is_type_compatible` (:issue:`42113`)
• Removed deprecated :meth:`Index.is_mixed`, check index.inferred_type directly instead (:issue:`32922`)
• Removed deprecated :func:`pandas.api.types.is_categorical`; use :func:`pandas.api.types.is_categorical_dtype` instead (:issue:`33385`)
• Removed deprecated :meth:`Index.asi8` (:issue:`37877`)
• Enforced deprecation changing behavior when passing datetime64[ns] dtype data and timezone-aware dtype to :class:`Series`, interpreting the values as wall-times instead of UTC times, matching :class:`DatetimeIndex` behavior (:issue:`41662`)
• Removed deprecated :meth:`DataFrame._AXIS_NUMBERS`, :meth:`DataFrame._AXIS_NAMES`, :meth:`Series._AXIS_NUMBERS`, :meth:`Series._AXIS_NAMES` (:issue:`33637`)
• Removed deprecated :meth:`Index.to_native_types`, use obj.astype(str) instead (:issue:`36418`)
• Removed deprecated :meth:`Series.iteritems`, :meth:`DataFrame.iteritems`, use obj.items instead (:issue:`45321`)
• Removed deprecated :meth:`DatetimeIndex.union_many` (:issue:`45018`)
• Removed deprecated weekofyear and week attributes of :class:`DatetimeArray`, :class:`DatetimeIndex` and dt accessor in favor of isocalendar().week (:issue:`33595`)
• Removed deprecated :meth:`RangeIndex._start`, :meth:`RangeIndex._stop`, :meth:`RangeIndex._step`, use start, stop, step instead (:issue:`30482`)
• Removed deprecated :meth:`DatetimeIndex.to_perioddelta`, Use dtindex - dtindex.to_period(freq).to_timestamp() instead (:issue:`34853`)
• Enforced deprecation disallowing passing a timezone-aware :class:`Timestamp` and dtype="datetime64[ns]" to :class:`Series` or :class:`DataFrame` constructors (:issue:`41555`)
• Enforced deprecation disallowing passing a sequence of timezone-aware values and dtype="datetime64[ns]" to to :class:`Series` or :class:`DataFrame` constructors (:issue:`41555`)
• Enforced deprecation disallowing using .astype to convert a datetime64[ns] :class:`Series`, :class:`DataFrame`, or :class:`DatetimeIndex` to timezone-aware dtype, use obj.tz_localize or ser.dt.tz_localize instead (:issue:`39258`)
• Enforced deprecation disallowing using .astype to convert a timezone-aware :class:`Series`, :class:`DataFrame`, or :class:`DatetimeIndex` to timezone-naive datetime64[ns] dtype, use obj.tz_localize(None) or obj.tz_convert("UTC").tz_localize(None) instead (:issue:`39258`)
• Removed Date parser functions :func:`~pandas.io.date_converters.parse_date_time`, :func:`~pandas.io.date_converters.parse_date_fields`, :func:`~pandas.io.date_converters.parse_all_fields` and :func:`~pandas.io.date_converters.generic_parser` (:issue:`24518`)
• Remove argument squeeze from :meth:`DataFrame.groupby` and :meth:`Series.groupby` (:issue:`32380`)
• Removed deprecated apply, apply_index, __call__, onOffset, and isAnchored attributes from :class:`DateOffset` (:issue:`34171`)
• Removed keep_tz argument in :meth:`DatetimeIndex.to_series` (:issue:`29731`)
• Remove arguments names and dtype from :meth:`Index.copy` and levels and codes from :meth:`MultiIndex.copy` (:issue:`35853`, :issue:`36685`)
• Remove argument inplace from :meth:`MultiIndex.set_levels` and :meth:`MultiIndex.set_codes` (:issue:`35626`)
• Disallow passing positional arguments to :meth:`MultiIndex.set_levels` and :meth:`MultiIndex.set_codes` (:issue:`41485`)
• Removed :meth:`MultiIndex.is_lexsorted` and :meth:`MultiIndex.lexsort_depth` (:issue:`38701`)
• Removed argument how from :meth:`PeriodIndex.astype`, use :meth:`PeriodIndex.to_timestamp` instead (:issue:`37982`)
• Removed argument try_cast from :meth:`DataFrame.mask`, :meth:`DataFrame.where`, :meth:`Series.mask` and :meth:`Series.where` (:issue:`38836`)
• Removed argument tz from :meth:`Period.to_timestamp`, use obj.to_timestamp(...).tz_localize(tz) instead (:issue:`34522`)
• Removed argument sort_columns in :meth:`DataFrame.plot` and :meth:`Series.plot` (:issue:`47563`)
• Removed argument is_copy from :meth:`DataFrame.take` and :meth:`Series.take` (:issue:`30615`)
• Removed argument kind from :meth:`Index.get_slice_bound`, :meth:`Index.slice_indexer` and :meth:`Index.slice_locs` (:issue:`41378`)
• Disallow subclass-specific keywords (e.g. "freq", "tz", "names", "closed") in the :class:`Index` constructor (:issue:`38597`)
• Removed argument inplace from :meth:`Categorical.remove_unused_categories` (:issue:`37918`)
• Disallow passing non-round floats to :class:`Timestamp` with unit="M" or unit="Y" (:issue:`47266`)
• Remove keywords convert_float and mangle_dupe_cols from :func:`read_excel` (:issue:`41176`)
• Disallow passing non-keyword arguments to :func:`read_excel` except io and sheet_name (:issue:`34418`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.set_index` except keys (:issue:`41495`)
• Disallow passing non-keyword arguments to :meth:`Resampler.interpolate` except method (:issue:`41699`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.reset_index` and :meth:`Series.reset_index` except level (:issue:`41496`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.dropna` and :meth:`Series.dropna` (:issue:`41504`)
• Disallow passing non-keyword arguments to :meth:`ExtensionArray.argsort` (:issue:`46134`)
• Disallow passing non-keyword arguments to :meth:`Categorical.sort_values` (:issue:`47618`)
• Disallow passing non-keyword arguments to :meth:`Index.drop_duplicates` and :meth:`Series.drop_duplicates` (:issue:`41485`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.drop_duplicates` except for subset (:issue:`41485`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.sort_index` and :meth:`Series.sort_index` (:issue:`41506`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.interpolate` and :meth:`Series.interpolate` except for method (:issue:`41510`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.any` and :meth:`Series.any` (:issue:`44896`)
• Disallow passing non-keyword arguments to :meth:`Index.set_names` except for names (:issue:`41551`)
• Disallow passing non-keyword arguments to :meth:`Index.join` except for other (:issue:`46518`)
• Disallow passing non-keyword arguments to :func:`concat` except for objs (:issue:`41485`)
• Disallow passing non-keyword arguments to :func:`pivot` except for data (:issue:`48301`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.pivot` (:issue:`48301`)
• Disallow passing non-keyword arguments to :func:`read_json` except for path_or_buf (:issue:`27573`)
• Disallow passing non-keyword arguments to :func:`read_sas` except for filepath_or_buffer (:issue:`47154`)
• Disallow passing non-keyword arguments to :func:`read_stata` except for filepath_or_buffer (:issue:`48128`)
• Disallow passing non-keyword arguments to :func:`read_csv` except filepath_or_buffer (:issue:`41485`)
• Disallow passing non-keyword arguments to :func:`read_table` except filepath_or_buffer (:issue:`41485`)
• Disallow passing non-keyword arguments to :func:`read_fwf` except filepath_or_buffer (:issue:`44710`)
• Disallow passing non-keyword arguments to :func:`read_xml` except for path_or_buffer (:issue:`45133`)
• Disallow passing non-keyword arguments to :meth:`Series.mask` and :meth:`DataFrame.mask` except cond and other (:issue:`41580`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.to_stata` except for path (:issue:`48128`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.where` and :meth:`Series.where` except for cond and other (:issue:`41523`)
• Disallow passing non-keyword arguments to :meth:`Series.set_axis` and :meth:`DataFrame.set_axis` except for labels (:issue:`41491`)
• Disallow passing non-keyword arguments to :meth:`Series.rename_axis` and :meth:`DataFrame.rename_axis` except for mapper (:issue:`47587`)
• Disallow passing non-keyword arguments to :meth:`Series.clip` and :meth:`DataFrame.clip` (:issue:`41511`)
• Disallow passing non-keyword arguments to :meth:`Series.bfill`, :meth:`Series.ffill`, :meth:`DataFrame.bfill` and :meth:`DataFrame.ffill` (:issue:`41508`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.replace`, :meth:`Series.replace` except for to_replace and value (:issue:`47587`)
• Disallow passing non-keyword arguments to :meth:`DataFrame.sort_values` except for by (:issue:`41505`)
• Disallow passing non-keyword arguments to :meth:`Series.sort_values` (:issue:`41505`)
• Removed :meth:`.Rolling.validate`, :meth:`.Expanding.validate`, and :meth:`.ExponentialMovingWindow.validate` (:issue:`43665`)
• Removed :attr:`Rolling.win_type` returning "freq" (:issue:`38963`)
• Removed :attr:`Rolling.is_datetimelike` (:issue:`38963`)
• Removed deprecated :meth:`Timedelta.delta`, :meth:`Timedelta.is_populated`, and :attr:`Timedelta.freq` (:issue:`46430`, :issue:`46476`)
• Removed deprecated :meth:`Categorical.replace`, use :meth:`Series.replace` instead (:issue:`44929`)
• Removed the numeric_only keyword from :meth:`Categorical.min` and :meth:`Categorical.max` in favor of skipna (:issue:`48821`)
• Removed :func:`is_extension_type` in favor of :func:`is_extension_array_dtype` (:issue:`29457`)
• Removed .ExponentialMovingWindow.vol (:issue:`39220`)
• Removed :meth:`Index.get_value` and :meth:`Index.set_value` (:issue:`33907`, :issue:`28621`)
• Removed :meth:`Series.slice_shift` and :meth:`DataFrame.slice_shift` (:issue:`37601`)
• Remove :meth:`DataFrameGroupBy.pad` and :meth:`DataFrameGroupBy.backfill` (:issue:`45076`)
• Remove numpy argument from :func:`read_json` (:issue:`30636`)
• Disallow passing abbreviations for orient in :meth:`DataFrame.to_dict` (:issue:`32516`)
• Removed get_offset in favor of :func:`to_offset` (:issue:`30340`)
• Removed the warn keyword in :func:`infer_freq` (:issue:`45947`)
• Removed the center keyword in :meth:`DataFrame.expanding` (:issue:`20647`)
• Removed the truediv keyword from :func:`eval` (:issue:`29812`)
• Removed the pandas.datetime submodule (:issue:`30489`)
• Removed the pandas.np submodule (:issue:`30296`)
• Removed pandas.util.testing in favor of pandas.testing (:issue:`30745`)
• Removed :meth:`Series.str.__iter__` (:issue:`28277`)
• Removed pandas.SparseArray in favor of :class:`arrays.SparseArray` (:issue:`30642`)
• Removed pandas.SparseSeries and pandas.SparseDataFrame, including pickle support. (:issue:`30642`)
• Enforced disallowing passing an integer fill_value to :meth:`DataFrame.shift` and :meth:`Series.shift`` with datetime64, timedelta64, or period dtypes (:issue:`32591`)
• Enforced disallowing a string column label into times in :meth:`DataFrame.ewm` (:issue:`43265`)
• Enforced disallowing a tuple of column labels into :meth:`.DataFrameGroupBy.__getitem__` (:issue:`30546`)
• Enforced disallowing setting values with .loc using a positional slice. Use .loc with labels or .iloc with positions instead (:issue:`31840`)
• Removed setting Categorical._codes directly (:issue:`41429`)
• Removed setting Categorical.categories directly (:issue:`47834`)
• Removed argument inplace from :meth:`Categorical.add_categories`, :meth:`Categorical.remove_categories`, :meth:`Categorical.set_categories`, :meth:`Categorical.rename_categories`, :meth:`Categorical.reorder_categories`, :meth:`Categorical.set_ordered`, :meth:`Categorical.as_ordered`, :meth:`Categorical.as_unordered` (:issue:`37981`, :issue:`41118`, :issue:`41133`, :issue:`47834`)
• Enforced :meth:`Rolling.count` with min_periods=None to default to the size of the window (:issue:`31302`)
• Renamed fname to path in :meth:`DataFrame.to_parquet`, :meth:`DataFrame.to_stata` and :meth:`DataFrame.to_feather` (:issue:`30338`)
• Enforced disallowing indexing a :class:`Series` with a single item list with a slice (e.g. ser[[slice(0, 2)]]). Either convert the list to tuple, or pass the slice directly instead (:issue:`31333`)
• Enforced the display.max_colwidth option to not accept negative integers (:issue:`31569`)
• Removed the display.column_space option in favor of df.to_string(col_space=...) (:issue:`47280`)
• Removed the deprecated method mad from pandas classes (:issue:`11787`)
• Removed the deprecated method tshift from pandas classes (:issue:`11631`)
• Changed the behavior of :func:`to_datetime` with argument "now" with utc=False to match Timestamp("now") (:issue:`18705`)
• Changed behavior of :class:`DataFrame` constructor given floating-point data and an integer dtype, when the data cannot be cast losslessly, the floating point dtype is retained, matching :class:`Series` behavior (:issue:`41170`)
• Changed behavior of :class:`DataFrame` constructor when passed a dtype (other than int) that the data cannot be cast to; it now raises instead of silently ignoring the dtype (:issue:`41733`)
• Changed the behavior of :class:`Series` constructor, it will no longer infer a datetime64 or timedelta64 dtype from string entries (:issue:`41731`)
• Changed behavior of :class:`Index` constructor when passed a SparseArray or SparseDtype to retain that dtype instead of casting to numpy.ndarray (:issue:`43930`)
• Changed behavior of :meth:`DataFrame.any` and :meth:`DataFrame.all` with bool_only=True; object-dtype columns with all-bool values will no longer be included, manually cast to bool dtype first (:issue:`46188`)

Performance improvements

• Performance improvement in :meth:`.DataFrameGroupBy.median` and :meth:`.SeriesGroupBy.median` and :meth:`.GroupBy.cumprod` for nullable dtypes (:issue:`37493`)
• Performance improvement in :meth:`MultiIndex.argsort` and :meth:`MultiIndex.sort_values` (:issue:`48406`)
• Performance improvement in :meth:`MultiIndex.size` (:issue:`48723`)
• Performance improvement in :meth:`MultiIndex.union` without missing values and without duplicates (:issue:`48505`, :issue:`48752`)
• Performance improvement in :meth:`MultiIndex.difference` (:issue:`48606`)
• Performance improvement in :class:`MultiIndex` set operations with sort=None (:issue:`49010`)
• Performance improvement in :meth:`.DataFrameGroupBy.mean`, :meth:`.SeriesGroupBy.mean`, :meth:`.DataFrameGroupBy.var`, and :meth:`.SeriesGroupBy.var` for extension array dtypes (:issue:`37493`)
• Performance improvement in :meth:`MultiIndex.isin` when level=None (:issue:`48622`)
• Performance improvement in :meth:`Index.union` and :meth:`MultiIndex.union` when index contains duplicates (:issue:`48900`)
• Performance improvement for :meth:`Series.value_counts` with nullable dtype (:issue:`48338`)
• Performance improvement for :class:`Series` constructor passing integer numpy array with nullable dtype (:issue:`48338`)
• Performance improvement for :class:`DatetimeIndex` constructor passing a list (:issue:`48609`)
• Performance improvement in :func:`merge` and :meth:`DataFrame.join` when joining on a sorted :class:`MultiIndex` (:issue:`48504`)
• Performance improvement in :meth:`DataFrame.loc` and :meth:`Series.loc` for tuple-based indexing of a :class:`MultiIndex` (:issue:`48384`)
• Performance improvement for :meth:`MultiIndex.unique` (:issue:`48335`)
• Performance improvement for :func:`concat` with extension array backed indexes (:issue:`49128`, :issue:`49178`)
• Reduce memory usage of :meth:`DataFrame.to_pickle`/:meth:`Series.to_pickle` when using BZ2 or LZMA (:issue:`49068`)
• Performance improvement for :class:`~arrays.StringArray` constructor passing a numpy array with type np.str_ (:issue:`49109`)
• Performance improvement in :meth:`~arrays.ArrowExtensionArray.factorize` (:issue:`49177`)
• Performance improvement in :meth:`DataFrame.join` when joining on a subset of a :class:`MultiIndex` (:issue:`48611`)
• Performance improvement for :meth:`MultiIndex.intersection` (:issue:`48604`)
• Performance improvement in var for nullable dtypes (:issue:`48379`).
• Performance improvements to :func:`read_sas` (:issue:`47403`, :issue:`47405`, :issue:`47656`, :issue:`48502`)
• Memory improvement in :meth:`RangeIndex.sort_values` (:issue:`48801`)
• Performance improvement in :class:`DataFrameGroupBy` and :class:`SeriesGroupBy` when by is a categorical type and sort=False (:issue:`48976`)

Bug fixes

Categorical

• Bug in :meth:`Categorical.set_categories` losing dtype information (:issue:`48812`)
• Bug in :meth:`DataFrame.groupby` and :meth:`Series.groupby` would reorder categories when used as a grouper (:issue:`48749`)

Datetimelike

• Bug in :func:`pandas.infer_freq`, raising TypeError when inferred on :class:`RangeIndex` (:issue:`47084`)
• Bug in :func:`to_datetime` was raising on invalid offsets with errors='coerce' and infer_datetime_format=True (:issue:`48633`)
• Bug in :class:`DatetimeIndex` constructor failing to raise when tz=None is explicitly specified in conjunction with timezone-aware dtype or data (:issue:`48659`)
• Bug in subtracting a datetime scalar from :class:`DatetimeIndex` failing to retain the original freq attribute (:issue:`48818`)

Timedelta

• Bug in :func:`to_timedelta` raising error when input has nullable dtype Float64 (:issue:`48796`)
• Bug in :class:`Timedelta` constructor incorrectly raising instead of returning NaT when given a np.timedelta64("nat") (:issue:`48898`)
• Bug in :class:`Timedelta` constructor failing to raise when passed both a :class:`Timedelta` object and keywords (e.g. days, seconds) (:issue:`48898`)

Timezones

Numeric

• Bug in :meth:`DataFrame.add` cannot apply ufunc when inputs contain mixed DataFrame type and Series type (:issue:`39853`)

Conversion

• Bug in constructing :class:`Series` with int64 dtype from a string list raising instead of casting (:issue:`44923`)
• Bug in :meth:`DataFrame.eval` incorrectly raising an AttributeError when there are negative values in function call (:issue:`46471`)
• Bug in :meth:`Series.convert_dtypes` not converting dtype to nullable dtype when :class:`Series` contains NA and has dtype object (:issue:`48791`)
• Bug where any :class:`ExtensionDtype` subclass with kind="M" would be interpreted as a timezone type (:issue:`34986`)
• Bug in :class:`.arrays.ArrowExtensionArray` that would raise NotImplementedError when passed a sequence of strings or binary (:issue:`49172`)

Strings

Interval

Indexing

• Bug in :meth:`DataFrame.reindex` filling with wrong values when indexing columns and index for uint dtypes (:issue:`48184`)
• Bug in :meth:`DataFrame.loc` coercing dtypes when setting values with a list indexer (:issue:`49159`)
• Bug in :meth:`DataFrame.__setitem__` raising ValueError when right hand side is :class:`DataFrame` with :class:`MultiIndex` columns (:issue:`49121`)
• Bug in :meth:`DataFrame.reindex` casting dtype to object when :class:`DataFrame` has single extension array column when re-indexing columns and index (:issue:`48190`)
• Bug in :func:`~DataFrame.describe` when formatting percentiles in the resulting index showed more decimals than needed (:issue:`46362`)
• Bug in :meth:`DataFrame.compare` does not recognize differences when comparing NA with value in nullable dtypes (:issue:`48939`)

Missing

• Bug in :meth:`Index.equals` raising TypeError when :class:`Index` consists of tuples that contain NA (:issue:`48446`)
• Bug in :meth:`Series.map` caused incorrect result when data has NaNs and defaultdict mapping was used (:issue:`48813`)
• Bug in :class:`NA` raising a TypeError instead of return :class:`NA` when performing a binary operation with a bytes object (:issue:`49108`)
• Bug in :meth:`Dataframe.update` raising TypeError when column has NaT values (:issue:`16713`)

MultiIndex

• Bug in :meth:`MultiIndex.argsort` raising TypeError when index contains :attr:`NA` (:issue:`48495`)
• Bug in :meth:`MultiIndex.difference` losing extension array dtype (:issue:`48606`)
• Bug in :class:`MultiIndex.set_levels` raising IndexError when setting empty level (:issue:`48636`)
• Bug in :meth:`MultiIndex.unique` losing extension array dtype (:issue:`48335`)
• Bug in :meth:`MultiIndex.intersection` losing extension array (:issue:`48604`)
• Bug in :meth:`MultiIndex.union` losing extension array (:issue:`48498`, :issue:`48505`, :issue:`48900`)
• Bug in :meth:`MultiIndex.union` not sorting when sort=None and index contains missing values (:issue:`49010`)
• Bug in :meth:`MultiIndex.append` not checking names for equality (:issue:`48288`)
• Bug in :meth:`MultiIndex.symmetric_difference` losing extension array (:issue:`48607`)

I/O

• Bug in :func:`read_sas` caused fragmentation of :class:`DataFrame` and raised :class:`.errors.PerformanceWarning` (:issue:`48595`)
• Improved error message in :func:`read_excel` by including the offending sheet name when an exception is raised while reading a file (:issue:`48706`)
• Bug when a pickling a subset PyArrow-backed data that would serialize the entire data instead of the subset (:issue:`42600`)
• Bug in :func:`read_csv` for a single-line csv with fewer columns than names raised :class:`.errors.ParserError` with engine="c" (:issue:`47566`)

Period

• Bug in :meth:`Period.strftime` and :meth:`PeriodIndex.strftime`, raising UnicodeDecodeError when a locale-specific directive was passed (:issue:`46319`)

Plotting

• ax.set_xlim was sometimes raising UserWarning which users couldn't address due to set_xlim not accepting parsing arguments - the converter now uses :func:`Timestamp` instead (:issue:`49148`)

Groupby/resample/rolling

• Bug in :class:`.ExponentialMovingWindow` with online not raising a NotImplementedError for unsupported operations (:issue:`48834`)
• Bug in :meth:`DataFrameGroupBy.sample` raises ValueError when the object is empty (:issue:`48459`)
• Bug in :meth:`Series.groupby` raises ValueError when an entry of the index is equal to the name of the index (:issue:`48567`)
• Bug in :meth:`DataFrameGroupBy.resample` produces inconsistent results when passing empty DataFrame (:issue:`47705`)

Reshaping

• Bug in :meth:`DataFrame.pivot_table` raising TypeError for nullable dtype and margins=True (:issue:`48681`)
• Bug in :meth:`DataFrame.unstack` and :meth:`Series.unstack` unstacking wrong level of :class:`MultiIndex` when :class:`MultiIndex` has mixed names (:issue:`48763`)
• Bug in :meth:`DataFrame.pivot` not respecting None as column name (:issue:`48293`)
• Bug in :func:`join` when left_on or right_on is or includes a :class:`CategoricalIndex` incorrectly raising AttributeError (:issue:`48464`)

Sparse

ExtensionArray

• Bug in :meth:`Series.mean` overflowing unnecessarily with nullable integers (:issue:`48378`)
• Bug when concatenating an empty DataFrame with an ExtensionDtype to another DataFrame with the same ExtensionDtype, the resulting dtype turned into object (:issue:`48510`)

Styler

Metadata

• Fixed metadata propagation in :meth:`DataFrame.corr` and :meth:`DataFrame.cov` (:issue:`28283`)

Other

Contributors