Skip to content

Latest commit

 

History

History
executable file
·
1002 lines (773 loc) · 60.2 KB

v1.0.0.rst

File metadata and controls

executable file
·
1002 lines (773 loc) · 60.2 KB

What's new in 1.0.0 (??)

Warning

Starting with the 1.x series of releases, pandas only supports Python 3.6.1 and higher.

New Deprecation Policy

Starting with Pandas 1.0.0, pandas will adopt a version of SemVer.

Historically, pandas has used a "rolling" deprecation policy, with occasional outright breaking API changes. Where possible, we would deprecate the behavior we'd like to change, giving an option to adopt the new behavior (via a keyword or an alternative method), and issuing a warning for users of the old behavior. Sometimes, a deprecation was not possible, and we would make an outright API breaking change.

We'll continue to introduce deprecations in major and minor releases (e.g. 1.0.0, 1.1.0, ...). Those deprecations will be enforced in the next major release.

Note that behavior changes and API breaking changes are not identical. API breaking changes will only be released in major versions. If we consider a behavior to be a bug, and fixing that bug induces a behavior change, we'll release that change in a minor release. This is a sometimes difficult judgment call that we'll do our best on.

This doesn't mean that pandas' pace of development will slow down. In the 2019 Pandas User Survey, about 95% of the respondents said they considered pandas "stable enough". This indicates there's an appetite for new features, even if it comes at the cost of break API. The difference is that now API breaking changes will be accompanied with a bump in the major version number (e.g. pandas 1.5.1 -> 2.0.0).

See :ref:`policies.version` for more.

{{ header }}

These are the changes in pandas 1.0.0. See :ref:`release` for a full changelog including other versions of pandas.

Enhancements

Dedicated string data type

We've added :class:`StringDtype`, an extension type dedicated to string data. Previously, strings were typically stored in object-dtype NumPy arrays. (:issue:`29975`)

Warning

StringDtype is currently considered experimental. The implementation and parts of the API may change without warning.

The 'string' extension type solves several issues with object-dtype NumPy arrays:

  1. You can accidentally store a mixture of strings and non-strings in an object dtype array. A StringArray can only store strings.
  2. object dtype breaks dtype-specific operations like :meth:`DataFrame.select_dtypes`. There isn't a clear way to select just text while excluding non-text, but still object-dtype columns.
  3. When reading code, the contents of an object dtype array is less clear than string.
.. ipython:: python

   pd.Series(['abc', None, 'def'], dtype=pd.StringDtype())

You can use the alias "string" as well.

.. ipython:: python

   s = pd.Series(['abc', None, 'def'], dtype="string")
   s

The usual string accessor methods work. Where appropriate, the return type of the Series or columns of a DataFrame will also have string dtype.

.. ipython:: python

   s.str.upper()
   s.str.split('b', expand=True).dtypes

String accessor methods returning integers will return a value with :class:`Int64Dtype`

.. ipython:: python

   s.str.count("a")

We recommend explicitly using the string data type when working with strings. See :ref:`text.types` for more.

Experimental NA scalar to denote missing values

A new pd.NA value (singleton) is introduced to represent scalar missing values. Up to now, np.nan is used for this for float data, np.nan or None for object-dtype data and pd.NaT for datetime-like data. The goal of pd.NA is provide a "missing" indicator that can be used consistently across data types. For now, the nullable integer and boolean data types and the new string data type make use of pd.NA (:issue:`28095`).

Warning

Experimental: the behaviour of pd.NA can still change without warning.

For example, creating a Series using the nullable integer dtype:

.. ipython:: python

    s = pd.Series([1, 2, None], dtype="Int64")
    s
    s[2]

Compared to np.nan, pd.NA behaves differently in certain operations. In addition to arithmetic operations, pd.NA also propagates as "missing" or "unknown" in comparison operations:

.. ipython:: python

    np.nan > 1
    pd.NA > 1

For logical operations, pd.NA follows the rules of the three-valued logic (or Kleene logic). For example:

.. ipython:: python

    pd.NA | True

For more, see :ref:`NA section <missing_data.NA>` in the user guide on missing data.

Boolean data type with missing values support

We've added :class:`BooleanDtype` / :class:`~arrays.BooleanArray`, an extension type dedicated to boolean data that can hold missing values. With the default 'bool data type based on a numpy bool array, the column can only hold True or False values and not missing values. This new :class:`BooleanDtype` can store missing values as well by keeping track of this in a separate mask. (:issue:`29555`, :issue:`30095`)

.. ipython:: python

   pd.Series([True, False, None], dtype=pd.BooleanDtype())

You can use the alias "boolean" as well.

.. ipython:: python

   s = pd.Series([True, False, None], dtype="boolean")
   s

Using Numba in rolling.apply

We've added an engine keyword to :meth:`~Rolling.apply` that allows the user to execute the routine using Numba instead of Cython. Using the Numba engine can yield significant performance gains if the apply function can operate on numpy arrays and the data set is larger (1 million rows or greater). For more details, see :ref:`rolling apply documentation <stats.rolling_apply>` (:issue:`28987`)

Defining custom windows for rolling operations

We've added a :func:`pandas.api.indexers.BaseIndexer` class that allows users to define how window bounds are created during rolling operations. Users can define their own get_window_bounds method on a :func:`pandas.api.indexers.BaseIndexer` subclass that will generate the start and end indices used for each window during the rolling aggregation. For more details and example usage, see the :ref:`custom window rolling documentation <stats.custom_rolling_window>`

Other enhancements

Build Changes

Pandas has added a pyproject.toml file and will no longer include cythonized files in the source distribution uploaded to PyPI (:issue:`28341`, :issue:`20775`). If you're installing a built distribution (wheel) or via conda, this shouldn't have any effect on you. If you're building pandas from source, you should no longer need to install Cython into your build environment before calling pip install pandas.

Backwards incompatible API changes

Avoid using names from MultiIndex.levels

As part of a larger refactor to :class:`MultiIndex` the level names are now stored separately from the levels (:issue:`27242`). We recommend using :attr:`MultiIndex.names` to access the names, and :meth:`Index.set_names` to update the names.

For backwards compatibility, you can still access the names via the levels.

.. ipython:: python

   mi = pd.MultiIndex.from_product([[1, 2], ['a', 'b']], names=['x', 'y'])
   mi.levels[0].name

However, it is no longer possible to update the names of the MultiIndex via the name of the level. The following will silently fail to update the name of the MultiIndex

.. ipython:: python

   mi.levels[0].name = "new name"
   mi.names

To update, use MultiIndex.set_names, which returns a new MultiIndex.

.. ipython:: python

   mi2 = mi.set_names("new name", level=0)
   mi2.names

New repr for :class:`~pandas.arrays.IntervalArray`

pandas 0.25.x

In [1]: pd.arrays.IntervalArray.from_tuples([(0, 1), (2, 3)])
Out[2]:
IntervalArray([(0, 1], (2, 3]],
              closed='right',
              dtype='interval[int64]')

pandas 1.0.0

.. ipython:: python

   pd.arrays.IntervalArray.from_tuples([(0, 1), (2, 3)])

All :class:`SeriesGroupBy` aggregation methods now respect the observed keyword

The following methods now also correctly output values for unobserved categories when called through groupby(..., observed=False) (:issue:`17605`)

.. ipython:: python

    df = pd.DataFrame({
        "cat_1": pd.Categorical(list("AABB"), categories=list("ABC")),
        "cat_2": pd.Categorical(list("AB") * 2, categories=list("ABC")),
        "value": [0.1] * 4,
    })
    df

pandas 0.25.x

In [2]: df.groupby(["cat_1", "cat_2"], observed=False)["value"].count()
Out[2]:
     cat_1  cat_2
     A      A        1
            B        1
     B      A        1
            B        1
 Name: value, dtype: int64

pandas 1.0.0

.. ipython:: python

   df.groupby(["cat_1", "cat_2"], observed=False)["value"].count()

:meth:`pandas.array` inference changes

:meth:`pandas.array` now infers pandas' new extension types in several cases (:issue:`29791`):

  1. String data (including missing values) now returns a :class:`arrays.StringArray`.
  2. Integer data (including missing values) now returns a :class:`arrays.IntegerArray`.
  3. Boolean data (including missing values) now returns the new :class:`arrays.BooleanArray`

pandas 0.25.x

>>> pd.array(["a", None])
<PandasArray>
['a', None]
Length: 2, dtype: object

>>> pd.array([1, None])
<PandasArray>
[1, None]
Length: 2, dtype: object

pandas 1.0.0

.. ipython:: python

   pd.array(["a", None])
   pd.array([1, None])

As a reminder, you can specify the dtype to disable all inference.

:class:`arrays.IntegerArray` now uses :attr:`pandas.NA`

:class:`arrays.IntegerArray` now uses :attr:`pandas.NA` rather than :attr:`numpy.nan` as its missing value marker (:issue:`29964`).

pandas 0.25.x

>>> a = pd.array([1, 2, None], dtype="Int64")
>>> a
<IntegerArray>
[1, 2, NaN]
Length: 3, dtype: Int64

>>> a[2]
nan

pandas 1.0.0

.. ipython:: python

   a = pd.array([1, 2, None], dtype="Int64")
   a[2]

See :ref:`missing_data.NA` for more on the differences between :attr:`pandas.NA` and :attr:`numpy.nan`.

:class:`arrays.IntegerArray` comparisons return :class:`arrays.BooleanArray`

Comparison operations on a :class:`arrays.IntegerArray` now returns a :class:`arrays.BooleanArray` rather than a NumPy array (:issue:`29964`).

pandas 0.25.x

>>> a = pd.array([1, 2, None], dtype="Int64")
>>> a
<IntegerArray>
[1, 2, NaN]
Length: 3, dtype: Int64

>>> a > 1
array([False,  True, False])

pandas 1.0.0

.. ipython:: python

   a = pd.array([1, 2, None], dtype="Int64")
   a > 1

Note that missing values now propagate, rather than always comparing unequal like :attr:`numpy.nan`. See :ref:`missing_data.NA` for more.

By default :meth:`Categorical.min` now returns the minimum instead of np.nan

When :class:`Categorical` contains np.nan, :meth:`Categorical.min` no longer return np.nan by default (skipna=True) (:issue:`25303`)

pandas 0.25.x

In [1]: pd.Categorical([1, 2, np.nan], ordered=True).min()
Out[1]: nan

pandas 1.0.0

.. ipython:: python

   pd.Categorical([1, 2, np.nan], ordered=True).min()

Default dtype of empty :class:`pandas.Series`

Initialising an empty :class:`pandas.Series` without specifying a dtype will raise a DeprecationWarning now (:issue:`17261`). The default dtype will change from float64 to object in future releases so that it is consistent with the behaviour of :class:`DataFrame` and :class:`Index`.

pandas 1.0.0

In [1]: pd.Series()
Out[2]:
DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.
Series([], dtype: float64)

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated (:issue:`29766`, :issue:`29723`). If installed, we now require:

Package Minimum Version Required Changed
numpy 1.13.3 X  
pytz 2015.4 X  
python-dateutil 2.6.1 X  
bottleneck 1.2.1    
numexpr 2.6.2    
pytest (dev) 4.0.2    

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package Minimum Version Changed
beautifulsoup4 4.6.0  
fastparquet 0.3.2 X
gcsfs 0.2.2  
lxml 3.8.0  
matplotlib 2.2.2  
numba 0.46.0 X
openpyxl 2.5.7 X
pyarrow 0.12.0 X
pymysql 0.7.1  
pytables 3.4.2  
s3fs 0.3.0 X
scipy 0.19.0  
sqlalchemy 1.1.4  
xarray 0.8.2  
xlrd 1.1.0  
xlsxwriter 0.9.8  
xlwt 1.2.0  

See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.

Other API changes

Documentation Improvements

Deprecations

Removed SparseSeries and SparseDataFrame

SparseSeries, SparseDataFrame and the DataFrame.to_sparse method have been removed (:issue:`28425`). We recommend using a Series or DataFrame with sparse values instead. See :ref:`sparse.migration` for help with migrating existing code.

Removal of prior version deprecations/changes

Matplotlib unit registration

Previously, pandas would register converters with matplotlib as a side effect of importing pandas (:issue:`18720`). This changed the output of plots made via matplotlib plots after pandas was imported, even if you were using matplotlib directly rather than :meth:`~DataFrame.plot`.

To use pandas formatters with a matplotlib plot, specify

>>> import pandas as pd
>>> pd.options.plotting.matplotlib.register_converters = True

Note that plots created by :meth:`DataFrame.plot` and :meth:`Series.plot` do register the converters automatically. The only behavior change is when plotting a date-like object via matplotlib.pyplot.plot or matplotlib.Axes.plot. See :ref:`plotting.formatters` for more.

Other removals

Performance improvements

Bug fixes

Categorical

Datetimelike

Timedelta

Timezones

Numeric

Conversion

Strings

Interval

Indexing

Missing

MultiIndex

  • Constructior for :class:`MultiIndex` verifies that the given sortorder is compatible with the actual lexsort_depth if verify_integrity parameter is True (the default) (:issue:`28735`)
  • Series and MultiIndex .drop with MultiIndex raise exception if labels not in given in level (:issue:`8594`)

I/O

Plotting

Groupby/resample/rolling

Reshaping

Sparse

ExtensionArray

Other

Contributors