v1.2.0.rst

.. _whatsnew_120:

What's new in 1.2.0 (??)
------------------------

These are the changes in pandas 1.2.0. See :ref:`release` for a full changelog
including other versions of pandas.

{{ header }}

.. ---------------------------------------------------------------------------

Enhancements
~~~~~~~~~~~~

Passing arguments to fsspec backends
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Many read/write functions have acquired the ``storage_options`` optional argument,
to pass a dictionary of parameters to the storage backend. This allows, for
example, for passing credentials to S3 and GCS storage. The details of what
parameters can be passed to which backends can be found in the documentation
of the individual storage backends (detailed from the fsspec docs for
`builtin implementations`_ and linked to `external ones`_). See
Section :ref:`io.remote`.

:issue:`35655` added fsspec support (including ``storage_options``)
for reading excel files.

.. _builtin implementations: https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations
.. _external ones: https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations

.. _whatsnew_120.binary_handle_to_csv:

Support for binary file handles in ``to_csv``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

:meth:`to_csv` supports file handles in binary mode (:issue:`19827` and :issue:`35058`)
with ``encoding`` (:issue:`13068` and :issue:`23854`) and ``compression`` (:issue:`22555`).
``mode`` has to contain a ``b`` for binary handles to be supported.

For example:

.. ipython:: python

   import io

   data = pd.DataFrame([0, 1, 2])
   buffer = io.BytesIO()
   data.to_csv(buffer, mode="w+b", encoding="utf-8", compression="gzip")

.. _whatsnew_120.enhancements.other:

Other enhancements
^^^^^^^^^^^^^^^^^^
- :class:`Index` with object dtype supports division and multiplication (:issue:`34160`)
-
-

.. _whatsnew_120.api_breaking.experimental:

Changes to experimental APIs
----------------------------

- :meth:`pandas.api.extensions.ExtensionDtype.construct_array_type` has changed from a classmethod to a regular method to support one dtype being used for multiple arrays. To migrate, change your definition to a regular method and ensure that your method is called on instances rather than the class (:issue:`36126`).

.. _whatsnew_120.api_breaking.python:

Increased minimum version for Python
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Pandas 1.2.0 supports Python 3.7.1 and higher (:issue:`35214`).

.. _whatsnew_120.api_breaking.deps:

Increased minimum versions for dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some minimum supported versions of dependencies were updated (:issue:`35214`).
If installed, we now require:

+-----------------+-----------------+----------+---------+
| Package         | Minimum Version | Required | Changed |
+=================+=================+==========+=========+
| numpy           | 1.16.5          |    X     |    X    |
+-----------------+-----------------+----------+---------+
| pytz            | 2017.3          |    X     |    X    |
+-----------------+-----------------+----------+---------+
| python-dateutil | 2.7.3           |    X     |         |
+-----------------+-----------------+----------+---------+
| bottleneck      | 1.2.1           |          |         |
+-----------------+-----------------+----------+---------+
| numexpr         | 2.6.8           |          |    X    |
+-----------------+-----------------+----------+---------+
| pytest (dev)    | 5.0.1           |          |    X    |
+-----------------+-----------------+----------+---------+

For `optional libraries <https://dev.pandas.io/docs/install.html#dependencies>`_ the general recommendation is to use the latest version.
The following table lists the lowest version per library that is currently being tested throughout the development of pandas.
Optional libraries below the lowest tested version may still work, but are not considered supported.

+-----------------+-----------------+---------+
| Package         | Minimum Version | Changed |
+=================+=================+=========+
| beautifulsoup4  | 4.6.0           |         |
+-----------------+-----------------+---------+
| fastparquet     | 0.3.2           |         |
+-----------------+-----------------+---------+
| fsspec          | 0.7.4           |         |
+-----------------+-----------------+---------+
| gcsfs           | 0.6.0           |         |
+-----------------+-----------------+---------+
| lxml            | 4.3.0           |    X    |
+-----------------+-----------------+---------+
| matplotlib      | 2.2.3           |    X    |
+-----------------+-----------------+---------+
| numba           | 0.46.0          |         |
+-----------------+-----------------+---------+
| openpyxl        | 2.5.7           |         |
+-----------------+-----------------+---------+
| pyarrow         | 0.15.0          |    X    |
+-----------------+-----------------+---------+
| pymysql         | 0.7.11          |    X    |
+-----------------+-----------------+---------+
| pytables        | 3.4.4           |    X    |
+-----------------+-----------------+---------+
| s3fs            | 0.4.0           |         |
+-----------------+-----------------+---------+
| scipy           | 1.2.0           |         |
+-----------------+-----------------+---------+
| sqlalchemy      | 1.2.8           |    X    |
+-----------------+-----------------+---------+
| xarray          | 0.12.0          |    X    |
+-----------------+-----------------+---------+
| xlrd            | 1.2.0           |    X    |
+-----------------+-----------------+---------+
| xlsxwriter      | 1.0.2           |    X    |
+-----------------+-----------------+---------+
| xlwt            | 1.3.0           |    X    |
+-----------------+-----------------+---------+
| pandas-gbq      | 0.12.0          |         |
+-----------------+-----------------+---------+

See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.

.. ---------------------------------------------------------------------------

.. _whatsnew_120.deprecations:

Deprecations
~~~~~~~~~~~~
- Deprecated parameter ``inplace`` in :meth:`MultiIndex.set_codes` and :meth:`MultiIndex.set_levels` (:issue:`35626`)
- Deprecated parameter ``dtype`` in :~meth:`Index.copy` on method all index classes. Use the :meth:`Index.astype` method instead for changing dtype(:issue:`35853`)
-

.. ---------------------------------------------------------------------------


.. _whatsnew_120.performance:

Performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~

- Performance improvement in :meth:`GroupBy.agg` with the ``numba`` engine (:issue:`35759`)
-

.. ---------------------------------------------------------------------------

.. _whatsnew_120.bug_fixes:

Bug fixes
~~~~~~~~~


Categorical
^^^^^^^^^^^

-
-

Datetimelike
^^^^^^^^^^^^
- Bug in :attr:`DatetimeArray.date` where a ``ValueError`` would be raised with a read-only backing array (:issue:`33530`)
- Bug in ``NaT`` comparisons failing to raise ``TypeError`` on invalid inequality comparisons (:issue:`35046`)
- Bug in :class:`DateOffset` where attributes reconstructed from pickle files differ from original objects when input values exceed normal ranges (e.g months=12) (:issue:`34511`)
-

Timedelta
^^^^^^^^^
- Bug in :class:`TimedeltaIndex`, :class:`Series`, and :class:`DataFrame` floor-division with ``timedelta64`` dtypes and ``NaT`` in the denominator (:issue:`35529`)
-
-

Timezones
^^^^^^^^^

- Bug in :func:`date_range` was raising AmbiguousTimeError for valid input with `ambiguous=False` (:issue:`35297`)
-


Numeric
^^^^^^^
-
-

Conversion
^^^^^^^^^^

-
-

Strings
^^^^^^^

-
-


Interval
^^^^^^^^

-
-

Indexing
^^^^^^^^

-
-

Missing
^^^^^^^

- Bug in :meth:`SeriesGroupBy.transform` now correctly handles missing values for `dropna=False` (:issue:`35014`)
-

MultiIndex
^^^^^^^^^^

- Bug in :meth:`DataFrame.xs` when used with :class:`IndexSlice` raises ``TypeError`` with message `Expected label or tuple of labels` (:issue:`35301`)
-

I/O
^^^

- Bug in :meth:`to_csv` caused a ``ValueError`` when it was called with a filename in combination with ``mode`` containing a ``b`` (:issue:`35058`)
- In :meth:`read_csv` `float_precision='round_trip'` now handles `decimal` and `thousands` parameters (:issue:`35365`)
- :meth:`to_pickle` and :meth:`read_pickle` were closing user-provided file objects (:issue:`35679`)
- :meth:`to_csv` passes compression arguments for `'gzip'` always to `gzip.GzipFile` (:issue:`28103`)

Plotting
^^^^^^^^

-
-

Groupby/resample/rolling
^^^^^^^^^^^^^^^^^^^^^^^^

- Bug in :meth:`DataFrameGroupBy.count` and :meth:`SeriesGroupBy.sum` returning ``NaN`` for missing categories when grouped on multiple ``Categoricals``. Now returning ``0`` (:issue:`35028`)
- Bug in :meth:`DataFrameGroupBy.apply` that would some times throw an erroneous ``ValueError`` if the grouping axis had duplicate entries (:issue:`16646`)
- Bug when combining methods :meth:`DataFrame.groupby` with :meth:`DataFrame.resample` and :meth:`DataFrame.interpolate` raising an ``TypeError`` (:issue:`35325`)
- Bug in :meth:`DataFrameGroupBy.apply` where a non-nuisance grouping column would be dropped from the output columns if another groupby method was called before ``.apply()`` (:issue:`34656`)
- Bug in :meth:`DataFrameGroupby.apply` would drop a :class:`CategoricalIndex` when grouped on. (:issue:`35792`)
- Bug when subsetting columns on a :class:`~pandas.core.groupby.DataFrameGroupBy` (e.g. ``df.groupby('a')[['b']])``) would reset the attributes ``axis``, ``dropna``, ``group_keys``, ``level``, ``mutated``, ``sort``, and ``squeeze`` to their default values. (:issue:`9959`)
-

Reshaping
^^^^^^^^^

- Bug in :meth:`DataFrame.pivot_table` with ``aggfunc='count'`` or ``aggfunc='sum'`` returning ``NaN`` for missing categories when pivoted on a ``Categorical``. Now returning ``0`` (:issue:`31422`)
- Bug in :func:`union_indexes` where input index names are not preserved in some cases. Affects :func:`concat` and :class:`DataFrame` constructor (:issue:`13475`)
- Bug in func :meth:`crosstab` when using multiple columns with ``margins=True`` and ``normalize=True`` (:issue:`35144`)
-

Sparse
^^^^^^

-
-

ExtensionArray
^^^^^^^^^^^^^^

-
-


Other
^^^^^
-
-

.. ---------------------------------------------------------------------------

.. _whatsnew_120.contributors:

Contributors
~~~~~~~~~~~~