Merge branch 'master' of https://github.com/pandas-dev/pandas into ref-numeric-validate_fill_value

Author: jbrockmendel

README.md

@@ -60,7 +60,7 @@ Here are just a few of the things that pandas does well:
     and saving/loading data from the ultrafast [**HDF5 format**][hdfstore]
   - [**Time series**][timeseries]-specific functionality: date range
     generation and frequency conversion, moving window statistics,
-    date shifting and lagging.
+    date shifting and lagging
 
 
    [missing-data]: https://pandas.pydata.org/pandas-docs/stable/missing_data.html#working-with-missing-data

asv_bench/benchmarks/reshape.py

@@ -103,7 +103,10 @@ def setup(self):
         nidvars = 20
         N = 5000
         self.letters = list("ABCD")
-        yrvars = [l + str(num) for l, num in product(self.letters, range(1, nyrs + 1))]
+        yrvars = [
+            letter + str(num)
+            for letter, num in product(self.letters, range(1, nyrs + 1))
+        ]
         columns = [str(i) for i in range(nidvars)] + yrvars
         self.df = DataFrame(np.random.randn(N, nidvars + len(yrvars)), columns=columns)
         self.df["id"] = self.df.index

doc/source/user_guide/style.ipynb

@@ -793,7 +793,8 @@
    "source": [
     "The next option you have are \"table styles\".\n",
     "These are styles that apply to the table as a whole, but don't look at the data.\n",
-    "Certain stylings, including pseudo-selectors like `:hover` can only be used this way."
+    "Certain stylings, including pseudo-selectors like `:hover` can only be used this way.\n",
+    "These can also be used to set specific row or column based class selectors, as will be shown."
    ]
   },
   {
@@ -831,9 +832,32 @@
     "The value for `props` should be a list of tuples of `('attribute', 'value')`.\n",
     "\n",
     "`table_styles` are extremely flexible, but not as fun to type out by hand.\n",
-    "We hope to collect some useful ones either in pandas, or preferable in a new package that [builds on top](#Extensibility) the tools here."
+    "We hope to collect some useful ones either in pandas, or preferable in a new package that [builds on top](#Extensibility) the tools here.\n",
+    "\n",
+    "`table_styles` can be used to add column and row based class descriptors. For large tables this can increase performance by avoiding repetitive individual css for each cell, and it can also simplify style construction in some cases.\n",
+    "If `table_styles` is given as a dictionary each key should be a specified column or index value and this will map to specific class CSS selectors of the given column or row.\n",
+    "\n",
+    "Note that `Styler.set_table_styles` will overwrite existing styles but can be chained by setting the `overwrite` argument to `False`."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "html = html.set_table_styles({\n",
+    "    'B': [dict(selector='', props=[('color', 'green')])],\n",
+    "    'C': [dict(selector='td', props=[('color', 'red')])], \n",
+    "    }, overwrite=False)\n",
+    "html"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -922,10 +946,12 @@
     "- DataFrame only `(use Series.to_frame().style)`\n",
     "- The index and columns must be unique\n",
     "- No large repr, and performance isn't great; this is intended for summary DataFrames\n",
-    "- You can only style the *values*, not the index or columns\n",
+    "- You can only style the *values*, not the index or columns (except with `table_styles` above)\n",
     "- You can only apply styles, you can't insert new HTML entities\n",
     "\n",
-    "Some of these will be addressed in the future.\n"
+    "Some of these will be addressed in the future.\n",
+    "Performance can suffer when adding styles to each cell in a large DataFrame.\n",
+    "It is recommended to apply table or column based styles where possible to limit overall HTML length, as well as setting a shorter UUID to avoid unnecessary repeated data transmission. \n"
    ]
   },
   {

doc/source/whatsnew/v1.1.5.rst

@@ -17,7 +17,7 @@ Fixed regressions
 - Regression in addition of a timedelta-like scalar to a :class:`DatetimeIndex` raising incorrectly (:issue:`37295`)
 - Fixed regression in :meth:`Series.groupby` raising when the :class:`Index` of the :class:`Series` had a tuple as its name (:issue:`37755`)
 - Fixed regression in :meth:`DataFrame.loc` and :meth:`Series.loc` for ``__setitem__`` when one-dimensional tuple was given to select from :class:`MultiIndex` (:issue:`37711`)
--
+- Fixed regression in inplace operations on :class:`Series` with ``ExtensionDtype`` with NumPy dtyped operand (:issue:`37910`)
 
 .. ---------------------------------------------------------------------------
 

doc/source/whatsnew/v1.2.0.rst

@@ -232,6 +232,7 @@ Other enhancements
 - :class:`Index` with object dtype supports division and multiplication (:issue:`34160`)
 - :meth:`DataFrame.explode` and :meth:`Series.explode` now support exploding of sets (:issue:`35614`)
 - :meth:`DataFrame.hist` now supports time series (datetime) data (:issue:`32590`)
+- :meth:`Styler.set_table_styles` now allows the direct styling of rows and columns and can be chained (:issue:`35607`)
 - ``Styler`` now allows direct CSS class name addition to individual data cells (:issue:`36159`)
 - :meth:`Rolling.mean()` and :meth:`Rolling.sum()` use Kahan summation to calculate the mean to avoid numerical problems (:issue:`10319`, :issue:`11645`, :issue:`13254`, :issue:`32761`, :issue:`36031`)
 - :meth:`DatetimeIndex.searchsorted`, :meth:`TimedeltaIndex.searchsorted`, :meth:`PeriodIndex.searchsorted`, and :meth:`Series.searchsorted` with datetimelike dtypes will now try to cast string arguments (listlike and scalar) to the matching datetimelike type (:issue:`36346`)
@@ -619,6 +620,7 @@ Indexing
 - Bug in indexing on a :class:`Series` or :class:`DataFrame` with a :class:`CategoricalIndex` using listlike indexer that contains elements that are in the index's ``categories`` but not in the index itself failing to raise ``KeyError`` (:issue:`37901`)
 - Bug in :meth:`DataFrame.iloc` and :meth:`Series.iloc` aligning objects in ``__setitem__`` (:issue:`22046`)
 - Bug in :meth:`DataFrame.loc` did not raise ``KeyError`` when missing combination was given with ``slice(None)`` for remaining levels (:issue:`19556`)
+- Bug in :meth:`DataFrame.loc` raising ``TypeError`` when non-integer slice was given to select values from :class:`MultiIndex` (:issue:`25165`, :issue:`24263`)
 - Bug on inserting a boolean label into a :class:`DataFrame` with a numeric :class:`Index` columns incorrectly casting to integer (:issue:`36319`)
 
 Missing

pandas/_libs/index_class_helper.pxi.in

@@ -10,29 +10,29 @@ WARNING: DO NOT edit .pxi FILE directly, .pxi is generated from .pxi.in
 
 {{py:
 
-# name, dtype, hashtable_name
-dtypes = [('Float64', 'float64', 'Float64'),
-          ('Float32', 'float32', 'Float64'),
-          ('Int64', 'int64', 'Int64'),
-          ('Int32', 'int32', 'Int64'),
-          ('Int16', 'int16', 'Int64'),
-          ('Int8', 'int8', 'Int64'),
-          ('UInt64', 'uint64', 'UInt64'),
-          ('UInt32', 'uint32', 'UInt64'),
-          ('UInt16', 'uint16', 'UInt64'),
-          ('UInt8', 'uint8', 'UInt64'),
+# name, dtype
+dtypes = [('Float64', 'float64'),
+          ('Float32', 'float32'),
+          ('Int64', 'int64'),
+          ('Int32', 'int32'),
+          ('Int16', 'int16'),
+          ('Int8', 'int8'),
+          ('UInt64', 'uint64'),
+          ('UInt32', 'uint32'),
+          ('UInt16', 'uint16'),
+          ('UInt8', 'uint8'),
           ]
 }}
 
-{{for name, dtype, hashtable_name in dtypes}}
+{{for name, dtype in dtypes}}
 
 
 cdef class {{name}}Engine(IndexEngine):
     # constructor-caller is responsible for ensuring that vgetter()
     #  returns an ndarray with dtype {{dtype}}_t
 
     cdef _make_hash_table(self, Py_ssize_t n):
-        return _hash.{{hashtable_name}}HashTable(n)
+        return _hash.{{name}}HashTable(n)
 
     {{if name not in {'Float64', 'Float32'} }}
     cdef _check_type(self, object val):
@@ -41,9 +41,7 @@ cdef class {{name}}Engine(IndexEngine):
     {{endif}}
 
     cdef void _call_map_locations(self, values):
-        # self.mapping is of type {{hashtable_name}}HashTable,
-        # so convert dtype of values
-        self.mapping.map_locations(algos.ensure_{{hashtable_name.lower()}}(values))
+        self.mapping.map_locations(algos.ensure_{{name.lower()}}(values))
 
     cdef _maybe_get_bool_indexer(self, object val):
         cdef:

pandas/_testing.py

@@ -749,19 +749,19 @@ def assert_index_equal(
     """
     __tracebackhide__ = True
 
-    def _check_types(l, r, obj="Index"):
+    def _check_types(left, right, obj="Index"):
         if exact:
-            assert_class_equal(l, r, exact=exact, obj=obj)
+            assert_class_equal(left, right, exact=exact, obj=obj)
 
             # Skip exact dtype checking when `check_categorical` is False
             if check_categorical:
-                assert_attr_equal("dtype", l, r, obj=obj)
+                assert_attr_equal("dtype", left, right, obj=obj)
 
             # allow string-like to have different inferred_types
-            if l.inferred_type in ("string"):
-                assert r.inferred_type in ("string")
+            if left.inferred_type in ("string"):
+                assert right.inferred_type in ("string")
             else:
-                assert_attr_equal("inferred_type", l, r, obj=obj)
+                assert_attr_equal("inferred_type", left, right, obj=obj)
 
     def _get_ilevel_values(index, level):
         # accept level number only
@@ -1147,9 +1147,9 @@ def _raise(left, right, err_msg):
                 )
 
             diff = 0
-            for l, r in zip(left, right):
+            for left_arr, right_arr in zip(left, right):
                 # count up differences
-                if not array_equivalent(l, r, strict_nan=strict_nan):
+                if not array_equivalent(left_arr, right_arr, strict_nan=strict_nan):
                     diff += 1
 
             diff = diff * 100.0 / left.size

pandas/conftest.py

@@ -33,8 +33,10 @@
 
 import pandas.util._test_decorators as td
 
+from pandas.core.dtypes.dtypes import DatetimeTZDtype, IntervalDtype
+
 import pandas as pd
-from pandas import DataFrame, Series
+from pandas import DataFrame, Interval, Period, Series, Timedelta, Timestamp
 import pandas._testing as tm
 from pandas.core import ops
 from pandas.core.indexes.api import Index, MultiIndex
@@ -687,6 +689,26 @@ def float_frame():
     return DataFrame(tm.getSeriesData())
 
 
+# ----------------------------------------------------------------
+# Scalars
+# ----------------------------------------------------------------
+@pytest.fixture(
+    params=[
+        (Interval(left=0, right=5), IntervalDtype("int64")),
+        (Interval(left=0.1, right=0.5), IntervalDtype("float64")),
+        (Period("2012-01", freq="M"), "period[M]"),
+        (Period("2012-02-01", freq="D"), "period[D]"),
+        (
+            Timestamp("2011-01-01", tz="US/Eastern"),
+            DatetimeTZDtype(tz="US/Eastern"),
+        ),
+        (Timedelta(seconds=500), "timedelta64[ns]"),
+    ]
+)
+def ea_scalar_and_dtype(request):
+    return request.param
+
+
 # ----------------------------------------------------------------
 # Operators & Operations
 # ----------------------------------------------------------------

pandas/core/algorithms.py

@@ -60,7 +60,7 @@
 from pandas.core.indexers import validate_indices
 
 if TYPE_CHECKING:
-    from pandas import Categorical, DataFrame, Series
+    from pandas import Categorical, DataFrame, Index, Series
 
 _shared_docs: Dict[str, str] = {}
 
@@ -433,10 +433,8 @@ def isin(comps: AnyArrayLike, values: AnyArrayLike) -> np.ndarray:
         return cast("Categorical", comps).isin(values)
 
     if needs_i8_conversion(comps):
-        # Dispatch to DatetimeLikeIndexMixin.isin
-        from pandas import Index
-
-        return Index(comps).isin(values)
+        # Dispatch to DatetimeLikeArrayMixin.isin
+        return array(comps).isin(values)
 
     comps, dtype = _ensure_data(comps)
     values, _ = _ensure_data(values, dtype=dtype)
@@ -542,7 +540,7 @@ def factorize(
     sort: bool = False,
     na_sentinel: Optional[int] = -1,
     size_hint: Optional[int] = None,
-) -> Tuple[np.ndarray, Union[np.ndarray, ABCIndex]]:
+) -> Tuple[np.ndarray, Union[np.ndarray, "Index"]]:
     """
     Encode the object as an enumerated type or categorical variable.
 

pandas/core/arrays/datetimelike.py

@@ -62,7 +62,7 @@
 from pandas.core.dtypes.missing import is_valid_nat_for_dtype, isna
 
 from pandas.core import nanops, ops
-from pandas.core.algorithms import checked_add_with_arr, unique1d, value_counts
+from pandas.core.algorithms import checked_add_with_arr, isin, unique1d, value_counts
 from pandas.core.arraylike import OpsMixin
 from pandas.core.arrays._mixins import NDArrayBackedExtensionArray
 import pandas.core.common as com
@@ -101,6 +101,8 @@ class DatetimeLikeArrayMixin(OpsMixin, NDArrayBackedExtensionArray):
         _generate_range
     """
 
+    # _infer_matches -> which infer_dtype strings are close enough to our own
+    _infer_matches: Tuple[str, ...]
     _is_recognized_dtype: Callable[[DtypeObj], bool]
     _recognized_scalars: Tuple[Type, ...]
     _data: np.ndarray
@@ -697,6 +699,59 @@ def map(self, mapper):
 
         return Index(self).map(mapper).array
 
+    def isin(self, values) -> np.ndarray:
+        """
+        Compute boolean array of whether each value is found in the
+        passed set of values.
+
+        Parameters
+        ----------
+        values : set or sequence of values
+
+        Returns
+        -------
+        ndarray[bool]
+        """
+        if not hasattr(values, "dtype"):
+            values = np.asarray(values)
+
+        if values.dtype.kind in ["f", "i", "u", "c"]:
+            # TODO: de-duplicate with equals, validate_comparison_value
+            return np.zeros(self.shape, dtype=bool)
+
+        if not isinstance(values, type(self)):
+            inferrable = [
+                "timedelta",
+                "timedelta64",
+                "datetime",
+                "datetime64",
+                "date",
+                "period",
+            ]
+            if values.dtype == object:
+                inferred = lib.infer_dtype(values, skipna=False)
+                if inferred not in inferrable:
+                    if inferred == "string":
+                        pass
+
+                    elif "mixed" in inferred:
+                        return isin(self.astype(object), values)
+                    else:
+                        return np.zeros(self.shape, dtype=bool)
+
+            try:
+                values = type(self)._from_sequence(values)
+            except ValueError:
+                return isin(self.astype(object), values)
+
+        try:
+            self._check_compatible_with(values)
+        except (TypeError, ValueError):
+            # Includes tzawareness mismatch and IncompatibleFrequencyError
+            return np.zeros(self.shape, dtype=bool)
+
+        return isin(self.asi8, values.asi8)
+
     # ------------------------------------------------------------------
     # Null Handling
 

pandas/core/arrays/datetimes.py

@@ -154,6 +154,7 @@ class DatetimeArray(dtl.TimelikeOps, dtl.DatelikeOps):
     _scalar_type = Timestamp
     _recognized_scalars = (datetime, np.datetime64)
     _is_recognized_dtype = is_datetime64_any_dtype
+    _infer_matches = ("datetime", "datetime64", "date")
 
     # define my properties & methods for delegation
     _bool_ops = [