Merge branch 'master' of https://github.com/pandas-dev/pandas into bug-iloc-ea

Author: jbrockmendel

doc/source/getting_started/intro_tutorials/02_read_write.rst

@@ -118,7 +118,7 @@ done by requesting the pandas ``dtypes`` attribute:
     titanic.dtypes
 
 For each of the columns, the used data type is enlisted. The data types
-in this ``DataFrame`` are integers (``int64``), floats (``float63``) and
+in this ``DataFrame`` are integers (``int64``), floats (``float64``) and
 strings (``object``).
 
 .. note::

doc/source/user_guide/io.rst

@@ -28,7 +28,7 @@ The pandas I/O API is a set of top level ``reader`` functions accessed like
     binary;`HDF5 Format <https://support.hdfgroup.org/HDF5/whatishdf5.html>`__;:ref:`read_hdf<io.hdf5>`;:ref:`to_hdf<io.hdf5>`
     binary;`Feather Format <https://github.com/wesm/feather>`__;:ref:`read_feather<io.feather>`;:ref:`to_feather<io.feather>`
     binary;`Parquet Format <https://parquet.apache.org/>`__;:ref:`read_parquet<io.parquet>`;:ref:`to_parquet<io.parquet>`
-    binary;`ORC Format <//https://orc.apache.org/>`__;:ref:`read_orc<io.orc>`;
+    binary;`ORC Format <https://orc.apache.org/>`__;:ref:`read_orc<io.orc>`;
     binary;`Msgpack <https://msgpack.org/index.html>`__;:ref:`read_msgpack<io.msgpack>`;:ref:`to_msgpack<io.msgpack>`
     binary;`Stata <https://en.wikipedia.org/wiki/Stata>`__;:ref:`read_stata<io.stata_reader>`;:ref:`to_stata<io.stata_writer>`
     binary;`SAS <https://en.wikipedia.org/wiki/SAS_(software)>`__;:ref:`read_sas<io.sas_reader>`;
@@ -4817,7 +4817,7 @@ ORC
 
 .. versionadded:: 1.0.0
 
-Similar to the :ref:`parquet <io.parquet>` format, the `ORC Format <//https://orc.apache.org/>`__ is a binary columnar serialization
+Similar to the :ref:`parquet <io.parquet>` format, the `ORC Format <https://orc.apache.org/>`__ is a binary columnar serialization
 for data frames. It is designed to make reading data frames efficient. Pandas provides *only* a reader for the
 ORC format, :func:`~pandas.read_orc`. This requires the `pyarrow <https://arrow.apache.org/docs/python/>`__ library.
 

doc/source/user_guide/text.rst

@@ -641,21 +641,40 @@ You can check whether elements contain a pattern:
 .. ipython:: python
 
    pattern = r'[0-9][a-z]'
-   pd.Series(['1', '2', '3a', '3b', '03c'],
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
              dtype="string").str.contains(pattern)
 
 Or whether elements match a pattern:
 
 .. ipython:: python
 
-   pd.Series(['1', '2', '3a', '3b', '03c'],
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
              dtype="string").str.match(pattern)
 
-The distinction between ``match`` and ``contains`` is strictness: ``match``
-relies on strict ``re.match``, while ``contains`` relies on ``re.search``.
+.. versionadded:: 1.1.0
 
-Methods like ``match``, ``contains``, ``startswith``, and ``endswith`` take
-an extra ``na`` argument so missing values can be considered True or False:
+.. ipython:: python
+
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
+             dtype="string").str.fullmatch(pattern)
+
+.. note::
+
+    The distinction between ``match``, ``fullmatch``, and ``contains`` is strictness:
+    ``fullmatch`` tests whether the entire string matches the regular expression;
+    ``match`` tests whether there is a match of the regular expression that begins
+    at the first character of the string; and ``contains`` tests whether there is
+    a match of the regular expression at any position within the string.
+
+    The corresponding functions in the ``re`` package for these three match modes are
+    `re.fullmatch <https://docs.python.org/3/library/re.html#re.fullmatch>`_,
+    `re.match <https://docs.python.org/3/library/re.html#re.match>`_, and
+    `re.search <https://docs.python.org/3/library/re.html#re.search>`_,
+    respectively.
+
+Methods like ``match``, ``fullmatch``, ``contains``, ``startswith``, and
+``endswith`` take an extra ``na`` argument so missing values can be considered
+True or False:
 
 .. ipython:: python
 

doc/source/whatsnew/v1.1.0.rst

@@ -69,6 +69,7 @@ Other enhancements
 - `OptionError` is now exposed in `pandas.errors` (:issue:`27553`)
 - :func:`timedelta_range` will now infer a frequency when passed ``start``, ``stop``, and ``periods`` (:issue:`32377`)
 - Positional slicing on a :class:`IntervalIndex` now supports slices with ``step > 1`` (:issue:`31658`)
+- :class:`Series.str` now has a `fullmatch` method that matches a regular expression against the entire string in each row of the series, similar to `re.fullmatch` (:issue:`32806`).
 - :meth:`DataFrame.sample` will now also allow array-like and BitGenerator objects to be passed to ``random_state`` as seeds (:issue:`32503`)
 -
 
@@ -260,7 +261,7 @@ Timedelta
 
 - Bug in constructing a :class:`Timedelta` with a high precision integer that would round the :class:`Timedelta` components (:issue:`31354`)
 - Bug in dividing ``np.nan`` or ``None`` by :class:`Timedelta`` incorrectly returning ``NaT`` (:issue:`31869`)
--
+- Timedeltas now understand ``µs`` as identifier for microsecond (:issue:`32899`)
 
 Timezones
 ^^^^^^^^^

pandas/_libs/tslibs/timedeltas.pyx

@@ -82,6 +82,7 @@ cdef dict timedelta_abbrevs = {
     "us": "us",
     "microseconds": "us",
     "microsecond": "us",
+    "µs": "us",
     "micro": "us",
     "micros": "us",
     "u": "us",

pandas/_testing.py

@@ -2662,3 +2662,34 @@ def external_error_raised(
     import pytest
 
     return pytest.raises(expected_exception, match=None)
+
+
+cython_table = pd.core.base.SelectionMixin._cython_table.items()
+
+
+def get_cython_table_params(ndframe, func_names_and_expected):
+    """
+    Combine frame, functions from SelectionMixin._cython_table
+    keys and expected result.
+
+    Parameters
+    ----------
+    ndframe : DataFrame or Series
+    func_names_and_expected : Sequence of two items
+        The first item is a name of a NDFrame method ('sum', 'prod') etc.
+        The second item is the expected return value.
+
+    Returns
+    -------
+    list
+        List of three items (DataFrame, function, expected result)
+    """
+    results = []
+    for func_name, expected in func_names_and_expected:
+        results.append((ndframe, func_name, expected))
+        results += [
+            (ndframe, func, expected)
+            for func, name in cython_table
+            if name == func_name
+        ]
+    return results

pandas/conftest.py

@@ -368,6 +368,17 @@ def _create_multiindex():
     return mi
 
 
+def _create_mi_with_dt64tz_level():
+    """
+    MultiIndex with a level that is a tzaware DatetimeIndex.
+    """
+    # GH#8367 round trip with pickle
+    return MultiIndex.from_product(
+        [[1, 2], ["a", "b"], pd.date_range("20130101", periods=3, tz="US/Eastern")],
+        names=["one", "two", "three"],
+    )
+
+
 indices_dict = {
     "unicode": tm.makeUnicodeIndex(100),
     "string": tm.makeStringIndex(100),
@@ -384,6 +395,7 @@ def _create_multiindex():
     "interval": tm.makeIntervalIndex(100),
     "empty": Index([]),
     "tuples": MultiIndex.from_tuples(zip(["foo", "bar", "baz"], [1, 2, 3])),
+    "mi-with-dt64tz-level": _create_mi_with_dt64tz_level(),
     "multi": _create_multiindex(),
     "repeats": Index([0, 0, 1, 1, 2, 2]),
 }
@@ -1119,10 +1131,7 @@ def spmatrix(request):
     return getattr(sparse, request.param + "_matrix")
 
 
-_cython_table = pd.core.base.SelectionMixin._cython_table.items()
-
-
-@pytest.fixture(params=list(_cython_table))
+@pytest.fixture(params=list(tm.cython_table))
 def cython_table_items(request):
     """
     Yields a tuple of a function and its corresponding name. Correspond to
@@ -1131,34 +1140,6 @@ def cython_table_items(request):
     return request.param
 
 
-def _get_cython_table_params(ndframe, func_names_and_expected):
-    """
-    Combine frame, functions from SelectionMixin._cython_table
-    keys and expected result.
-
-    Parameters
-    ----------
-    ndframe : DataFrame or Series
-    func_names_and_expected : Sequence of two items
-        The first item is a name of a NDFrame method ('sum', 'prod') etc.
-        The second item is the expected return value.
-
-    Returns
-    -------
-    list
-        List of three items (DataFrame, function, expected result)
-    """
-    results = []
-    for func_name, expected in func_names_and_expected:
-        results.append((ndframe, func_name, expected))
-        results += [
-            (ndframe, func, expected)
-            for func, name in _cython_table
-            if name == func_name
-        ]
-    return results
-
-
 @pytest.fixture(
     params=[
         getattr(pd.offsets, o)

pandas/core/algorithms.py

@@ -700,7 +700,7 @@ def value_counts(
         result = result.sort_index()
 
         # if we are dropna and we have NO values
-        if dropna and (result.values == 0).all():
+        if dropna and (result._values == 0).all():
             result = result.iloc[0:0]
 
         # normalizing is by len of all (regardless of dropna)
@@ -713,7 +713,7 @@ def value_counts(
             # handle Categorical and sparse,
             result = Series(values)._values.value_counts(dropna=dropna)
             result.name = name
-            counts = result.values
+            counts = result._values
 
         else:
             keys, counts = _value_counts_arraylike(values, dropna)
@@ -823,7 +823,7 @@ def mode(values, dropna: bool = True) -> "Series":
     # categorical is a fast-path
     if is_categorical_dtype(values):
         if isinstance(values, Series):
-            return Series(values.values.mode(dropna=dropna), name=values.name)
+            return Series(values._values.mode(dropna=dropna), name=values.name)
         return values.mode(dropna=dropna)
 
     if dropna and needs_i8_conversion(values.dtype):

pandas/core/arrays/datetimelike.py

@@ -905,7 +905,7 @@ def value_counts(self, dropna=False):
         index = Index(
             cls(result.index.view("i8"), dtype=self.dtype), name=result.index.name
         )
-        return Series(result.values, index=index, name=result.name)
+        return Series(result._values, index=index, name=result.name)
 
     def map(self, mapper):
         # TODO(GH-23179): Add ExtensionArray.map

pandas/core/arrays/interval.py

@@ -152,7 +152,7 @@ class IntervalArray(IntervalMixin, ExtensionArray):
     def __new__(cls, data, closed=None, dtype=None, copy=False, verify_integrity=True):
 
         if isinstance(data, ABCSeries) and is_interval_dtype(data):
-            data = data.values
+            data = data._values
 
         if isinstance(data, (cls, ABCIntervalIndex)):
             left = data.left

pandas/core/arrays/masked.py

@@ -244,11 +244,11 @@ def value_counts(self, dropna: bool = True) -> "Series":
         # TODO(extension)
         # if we have allow Index to hold an ExtensionArray
         # this is easier
-        index = value_counts.index.values.astype(object)
+        index = value_counts.index._values.astype(object)
 
         # if we want nans, count the mask
         if dropna:
-            counts = value_counts.values
+            counts = value_counts._values
         else:
             counts = np.empty(len(value_counts) + 1, dtype="int64")
             counts[:-1] = value_counts

pandas/core/base.py

@@ -123,15 +123,11 @@ def __setattr__(self, key: str, value):
         object.__setattr__(self, key, value)
 
 
-class GroupByError(Exception):
+class DataError(Exception):
     pass
 
 
-class DataError(GroupByError):
-    pass
-
-
-class SpecificationError(GroupByError):
+class SpecificationError(Exception):
     pass
 
 
@@ -372,7 +368,7 @@ def _agg_1dim(name, how, subset=None):
                     )
                 return colg.aggregate(how)
 
-            def _agg_2dim(name, how):
+            def _agg_2dim(how):
                 """
                 aggregate a 2-dim with how
                 """
@@ -660,7 +656,7 @@ def item(self):
         ):
             # numpy returns ints instead of datetime64/timedelta64 objects,
             #  which we need to wrap in Timestamp/Timedelta/Period regardless.
-            return self.values.item()
+            return self._values.item()
 
         if len(self) == 1:
             return next(iter(self))
@@ -1132,10 +1128,8 @@ def _map_values(self, mapper, na_action=None):
                 # use the built in categorical series mapper which saves
                 # time by mapping the categories instead of all values
                 return self._values.map(mapper)
-            if is_extension_array_dtype(self.dtype):
-                values = self._values
-            else:
-                values = self.values
+
+            values = self._values
 
             indexer = mapper.index.get_indexer(values)
             new_values = algorithms.take_1d(mapper._values, indexer)

pandas/core/common.py

@@ -213,7 +213,7 @@ def asarray_tuplesafe(values, dtype=None):
     if not (isinstance(values, (list, tuple)) or hasattr(values, "__array__")):
         values = list(values)
     elif isinstance(values, ABCIndexClass):
-        return values.values
+        return values._values
 
     if isinstance(values, list) and dtype in [np.object_, object]:
         return construct_1d_object_array_from_listlike(values)

pandas/core/dtypes/cast.py

@@ -888,7 +888,7 @@ def astype_nansafe(arr, dtype, copy: bool = True, skipna: bool = False):
         elif is_timedelta64_dtype(dtype):
             from pandas import to_timedelta
 
-            return astype_nansafe(to_timedelta(arr).values, dtype, copy=copy)
+            return astype_nansafe(to_timedelta(arr)._values, dtype, copy=copy)
 
     if dtype.name in ("datetime64", "timedelta64"):
         msg = (

pandas/core/dtypes/common.py

@@ -188,7 +188,9 @@ def ensure_python_int(value: Union[int, np.integer]) -> int:
     TypeError: if the value isn't an int or can't be converted to one.
     """
     if not is_scalar(value):
-        raise TypeError(f"Value needs to be a scalar value, was type {type(value)}")
+        raise TypeError(
+            f"Value needs to be a scalar value, was type {type(value).__name__}"
+        )
     try:
         new_value = int(value)
         assert new_value == value

pandas/core/dtypes/missing.py

@@ -229,7 +229,7 @@ def _isna_ndarraylike(obj):
     if not is_extension:
         # Avoid accessing `.values` on things like
         # PeriodIndex, which may be expensive.
-        values = getattr(obj, "values", obj)
+        values = getattr(obj, "_values", obj)
     else:
         values = obj
 
@@ -270,7 +270,7 @@ def _isna_ndarraylike(obj):
 
 
 def _isna_ndarraylike_old(obj):
-    values = getattr(obj, "values", obj)
+    values = getattr(obj, "_values", obj)
     dtype = values.dtype
 
     if is_string_dtype(dtype):

pandas/core/generic.py

@@ -7071,7 +7071,7 @@ def asof(self, where, subset=None):
 
                 return Series(np.nan, index=self.columns, name=where[0])
 
-        locs = self.index.asof_locs(where, ~(nulls.values))
+        locs = self.index.asof_locs(where, ~(nulls._values))
 
         # mask the missing
         missing = locs == -1
@@ -7230,7 +7230,7 @@ def _clip_with_scalar(self, lower, upper, inplace: bool_t = False):
             raise ValueError("Cannot use an NA value as a clip threshold")
 
         result = self
-        mask = isna(self.values)
+        mask = isna(self._values)
 
         with np.errstate(all="ignore"):
             if upper is not None:
@@ -8604,7 +8604,7 @@ def _where(
 
                 if self.ndim == 1:
 
-                    icond = cond.values
+                    icond = cond._values
 
                     # GH 2745 / GH 4192
                     # treat like a scalar