Merge branch 'main' into groupby-save-codes-new

* main:
  Preserve `base` and `loffset` arguments in `resample` (pydata#7444)
  ignore the `pkg_resources` deprecation warning (pydata#7594)
  Update contains_cftime_datetimes to avoid loading entire variable array (pydata#7494)
  Support first, last with dask arrays (pydata#7562)
  update the docs environment (pydata#7442)
  Add xCDAT to list of Xarray related projects (pydata#7579)
  [pre-commit.ci] pre-commit autoupdate (pydata#7565)
  fix nczarr when libnetcdf>4.8.1 (pydata#7575)
  use numpys SupportsDtype (pydata#7521)

Author: dcherian

.pre-commit-config.yaml

@@ -16,7 +16,7 @@ repos:
         files: ^xarray/
   - repo: https://github.com/charliermarsh/ruff-pre-commit
     # Ruff version.
-    rev: 'v0.0.248'
+    rev: 'v0.0.253'
     hooks:
       - id: ruff
         args: ["--fix"]

ci/requirements/doc.yml

@@ -4,7 +4,7 @@ channels:
   - conda-forge
   - nodefaults
 dependencies:
-  - python=3.9
+  - python=3.10
   - bottleneck
   - cartopy
   - cfgrib>=0.9
@@ -23,18 +23,17 @@ dependencies:
   - pandas>=1.4
   - pooch
   - pip
-  - pydata-sphinx-theme>=0.4.3
   - pyproj
   - rasterio>=1.1
   - scipy!=1.10.0
   - seaborn
   - setuptools
   - sparse
   - sphinx-autosummary-accessors
-  - sphinx-book-theme >= 0.0.38
+  - sphinx-book-theme >= 0.3.0
   - sphinx-copybutton
   - sphinx-design
-  - sphinx!=4.4.0
+  - sphinx>=5.0
   - zarr>=2.10
   - pip:
       - sphinxext-rediraffe

doc/conf.py

@@ -97,8 +97,8 @@
 
 
 extlinks = {
-    "issue": ("https://github.com/pydata/xarray/issues/%s", "GH"),
-    "pull": ("https://github.com/pydata/xarray/pull/%s", "PR"),
+    "issue": ("https://github.com/pydata/xarray/issues/%s", "GH%s"),
+    "pull": ("https://github.com/pydata/xarray/pull/%s", "PR%s"),
 }
 
 # sphinx-copybutton configurations
@@ -244,12 +244,11 @@
     use_repository_button=True,
     use_issues_button=True,
     home_page_in_toc=False,
-    extra_navbar="",
-    navbar_footer_text="",
     extra_footer="""<p>Xarray is a fiscally sponsored project of <a href="https://numfocus.org">NumFOCUS</a>,
     a nonprofit dedicated to supporting the open-source scientific computing community.<br>
     Theme by the <a href="https://ebp.jupyterbook.org">Executable Book Project</a></p>""",
     twitter_url="https://twitter.com/xarray_devs",
+    icon_links=[],  # workaround for pydata/pydata-sphinx-theme#1220
 )
 
 

doc/ecosystem.rst

@@ -45,6 +45,7 @@ Geosciences
 - `xarray-spatial <https://xarray-spatial.org/>`_: Numba-accelerated raster-based spatial processing tools (NDVI, curvature, zonal-statistics, proximity, hillshading, viewshed, etc.)
 - `xarray-topo <https://xarray-topo.readthedocs.io/>`_: xarray extension for topographic analysis and modelling.
 - `xbpch <https://github.com/darothen/xbpch>`_: xarray interface for bpch files.
+- `xCDAT <https://xcdat.readthedocs.io/>`_: An extension of xarray for climate data analysis on structured grids.
 - `xclim <https://xclim.readthedocs.io/>`_: A library for calculating climate science indices with unit handling built from xarray and dask.
 - `xESMF <https://pangeo-xesmf.readthedocs.io/>`_: Universal regridder for geospatial data.
 - `xgcm <https://xgcm.readthedocs.io/>`_: Extends the xarray data model to understand finite volume grid cells (common in General Circulation Models) and provides interpolation and difference operations for such grids.

doc/user-guide/interpolation.rst

@@ -50,7 +50,7 @@ array-like, which gives the interpolated result as an array.
     # interpolation
     da.interp(time=[2.5, 3.5])
 
-To interpolate data with a :py:doc:`numpy.datetime64 <reference/arrays.datetime>` coordinate you can pass a string.
+To interpolate data with a :py:doc:`numpy.datetime64 <numpy:reference/arrays.datetime>` coordinate you can pass a string.
 
 .. ipython:: python
 

doc/user-guide/weather-climate.rst

@@ -233,7 +233,7 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
 
 .. ipython:: python
 
-    da.resample(time="81T", closed="right", label="right", base=3).mean()
+    da.resample(time="81T", closed="right", label="right", offset="3T").mean()
 
 .. _Timestamp-valid range: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timestamp-limitations
 .. _ISO 8601 standard: https://en.wikipedia.org/wiki/ISO_8601

doc/whats-new.rst

@@ -25,13 +25,22 @@ New Features
 
 - Fix :py:meth:`xr.cov` and :py:meth:`xr.corr` now support complex valued arrays  (:issue:`7340`, :pull:`7392`).
   By `Michael Niklas <https://github.com/headtr1ck>`_.
+- Support dask arrays in ``first`` and ``last`` reductions.
+  By `Deepak Cherian <https://github.com/dcherian>`_.
 
 Breaking changes
 ~~~~~~~~~~~~~~~~
 
 
 Deprecations
 ~~~~~~~~~~~~
+- Following pandas, the ``base`` and ``loffset`` parameters of
+  :py:meth:`xr.DataArray.resample` and :py:meth:`xr.Dataset.resample` have been
+  deprecated and will be removed in a future version of xarray.  Using the
+  ``origin`` or ``offset`` parameters is recommended as a replacement for using
+  the ``base`` parameter and using time offset arithmetic is recommended as a
+  replacement for using the ``loffset`` parameter (:pull:`8459`).  By `Spencer
+  Clark <https://github.com/spencerkclark>`_.
 
 
 Bug fixes
@@ -42,6 +51,8 @@ Bug fixes
 - Fix matplotlib raising a UserWarning when plotting a scatter plot
   with an unfilled marker (:issue:`7313`, :pull:`7318`).
   By `Jimmy Westling <https://github.com/illviljan>`_.
+- Improved performance in ``open_dataset`` for datasets with large object arrays (:issue:`7484`, :pull:`7494`).
+  By `Alex Goodman <https://github.com/agoodm>`_ and `Deepak Cherian <https://github.com/dcherian>`_.
 
 Documentation
 ~~~~~~~~~~~~~

xarray/backends/zarr.py

@@ -207,7 +207,7 @@ def _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr):
                 "which are required for xarray to determine variable dimensions."
             ) from e
 
-    nc_attrs = [attr for attr in zarr_obj.attrs if attr.startswith("_NC")]
+    nc_attrs = [attr for attr in zarr_obj.attrs if attr.lower().startswith("_nc")]
     attributes = HiddenKeyDict(zarr_obj.attrs, [dimension_key] + nc_attrs)
     return dimensions, attributes
 
@@ -495,7 +495,7 @@ def get_attrs(self):
         return {
             k: v
             for k, v in self.zarr_group.attrs.asdict().items()
-            if not k.startswith("_NC")
+            if not k.lower().startswith("_nc")
         }
 
     def get_dimensions(self):

xarray/coding/calendar_ops.py

@@ -147,7 +147,7 @@ def convert_calendar(
     from xarray.core.dataarray import DataArray
 
     time = obj[dim]
-    if not _contains_datetime_like_objects(time):
+    if not _contains_datetime_like_objects(time.variable):
         raise ValueError(f"Coordinate {dim} must contain datetime objects.")
 
     use_cftime = _should_cftime_be_used(time, calendar, use_cftime)
@@ -319,8 +319,8 @@ def interp_calendar(source, target, dim="time"):
         target = DataArray(target, dims=(dim,), name=dim)
 
     if not _contains_datetime_like_objects(
-        source[dim]
-    ) or not _contains_datetime_like_objects(target):
+        source[dim].variable
+    ) or not _contains_datetime_like_objects(target.variable):
         raise ValueError(
             f"Both 'source.{dim}' and 'target' must contain datetime objects."
         )

xarray/coding/cftime_offsets.py

@@ -1267,7 +1267,7 @@ def date_range_like(source, calendar, use_cftime=None):
     if not isinstance(source, (pd.DatetimeIndex, CFTimeIndex)) and (
         isinstance(source, DataArray)
         and (source.ndim != 1)
-        or not _contains_datetime_like_objects(source)
+        or not _contains_datetime_like_objects(source.variable)
     ):
         raise ValueError(
             "'source' must be a 1D array of datetime objects for inferring its range."

xarray/coding/frequencies.py

@@ -79,11 +79,12 @@ def infer_freq(index):
         If there are fewer than three values or the index is not 1D.
     """
     from xarray.core.dataarray import DataArray
+    from xarray.core.variable import Variable
 
     if isinstance(index, (DataArray, pd.Series)):
         if index.ndim != 1:
             raise ValueError("'index' must be 1D")
-        elif not _contains_datetime_like_objects(DataArray(index)):
+        elif not _contains_datetime_like_objects(Variable("dim", index)):
             raise ValueError("'index' must contain datetime-like objects")
         dtype = np.asarray(index).dtype
         if dtype == "datetime64[ns]":

xarray/core/accessor_dt.py

@@ -574,7 +574,7 @@ def __new__(cls, obj: T_DataArray) -> CombinedDatetimelikeAccessor:
         # we need to choose which parent (datetime or timedelta) is
         # appropriate. Since we're checking the dtypes anyway, we'll just
         # do all the validation here.
-        if not _contains_datetime_like_objects(obj):
+        if not _contains_datetime_like_objects(obj.variable):
             raise TypeError(
                 "'.dt' accessor only available for "
                 "DataArray with datetime64 timedelta64 dtype or "

xarray/core/common.py

@@ -11,9 +11,16 @@
 import pandas as pd
 
 from xarray.core import dtypes, duck_array_ops, formatting, formatting_html, ops
+from xarray.core.indexing import BasicIndexer, ExplicitlyIndexed
 from xarray.core.options import OPTIONS, _get_keep_attrs
+from xarray.core.pdcompat import _convert_base_to_offset
 from xarray.core.pycompat import is_duck_dask_array
-from xarray.core.utils import Frozen, either_dict_or_kwargs, is_scalar
+from xarray.core.utils import (
+    Frozen,
+    either_dict_or_kwargs,
+    emit_user_level_warning,
+    is_scalar,
+)
 
 try:
     import cftime
@@ -40,6 +47,7 @@
         ScalarOrArray,
         SideOptions,
         T_DataWithCoords,
+        T_Variable,
     )
     from xarray.core.variable import Variable
 
@@ -843,6 +851,12 @@ def _resample(
             For frequencies that evenly subdivide 1 day, the "origin" of the
             aggregated intervals. For example, for "24H" frequency, base could
             range from 0 through 23.
+
+            .. deprecated:: 2023.03.0
+                Following pandas, the ``base`` parameter is deprecated in favor
+                of the ``origin`` and ``offset`` parameters, and will be removed
+                in a future version of xarray.
+
         origin : {'epoch', 'start', 'start_day', 'end', 'end_day'}, pd.Timestamp, datetime.datetime, np.datetime64, or cftime.datetime, default 'start_day'
             The datetime on which to adjust the grouping. The timezone of origin
             must match the timezone of the index.
@@ -858,6 +872,12 @@ def _resample(
         loffset : timedelta or str, optional
             Offset used to adjust the resampled time labels. Some pandas date
             offset strings are supported.
+
+            .. deprecated:: 2023.03.0
+                Following pandas, the ``loffset`` parameter is deprecated in favor
+                of using time offset arithmetic, and will be removed in a future
+                version of xarray.
+
         restore_coord_dims : bool, optional
             If True, also restore the dimension order of multi-dimensional
             coordinates.
@@ -928,8 +948,8 @@ def _resample(
         """
         # TODO support non-string indexer after removing the old API.
 
-        from xarray.coding.cftimeindex import CFTimeIndex
         from xarray.core.dataarray import DataArray
+        from xarray.core.groupby import TimeResampleGrouper
         from xarray.core.resample import RESAMPLE_DIM
 
         if keep_attrs is not None:
@@ -959,28 +979,36 @@ def _resample(
         dim_name: Hashable = dim
         dim_coord = self[dim]
 
-        if isinstance(self._indexes[dim_name].to_pandas_index(), CFTimeIndex):
-            from xarray.core.resample_cftime import CFTimeGrouper
-
-            grouper = CFTimeGrouper(
-                freq=freq,
-                closed=closed,
-                label=label,
-                base=base,
-                loffset=loffset,
-                origin=origin,
-                offset=offset,
+        if loffset is not None:
+            emit_user_level_warning(
+                "Following pandas, the `loffset` parameter to resample will be deprecated "
+                "in a future version of xarray.  Switch to using time offset arithmetic.",
+                FutureWarning,
             )
-        else:
-            grouper = pd.Grouper(
-                freq=freq,
-                closed=closed,
-                label=label,
-                base=base,
-                offset=offset,
-                origin=origin,
-                loffset=loffset,
+
+        if base is not None:
+            emit_user_level_warning(
+                "Following pandas, the `base` parameter to resample will be deprecated in "
+                "a future version of xarray.  Switch to using `origin` or `offset` instead.",
+                FutureWarning,
             )
+
+        if base is not None and offset is not None:
+            raise ValueError("base and offset cannot be present at the same time")
+
+        if base is not None:
+            index = self._indexes[dim_name].to_pandas_index()
+            offset = _convert_base_to_offset(base, freq, index)
+
+        grouper = TimeResampleGrouper(
+            freq=freq,
+            closed=closed,
+            label=label,
+            origin=origin,
+            offset=offset,
+            loffset=loffset,
+        )
+
         group = DataArray(
             dim_coord, coords=dim_coord.coords, dims=dim_coord.dims, name=RESAMPLE_DIM
         )
@@ -1770,31 +1798,27 @@ def is_np_timedelta_like(dtype: DTypeLike) -> bool:
     return np.issubdtype(dtype, np.timedelta64)
 
 
-def _contains_cftime_datetimes(array) -> bool:
-    """Check if an array contains cftime.datetime objects"""
+def _contains_cftime_datetimes(array: Any) -> bool:
+    """Check if a array inside a Variable contains cftime.datetime objects"""
     if cftime is None:
         return False
-    else:
-        if array.dtype == np.dtype("O") and array.size > 0:
-            sample = np.asarray(array).flat[0]
-            if is_duck_dask_array(sample):
-                sample = sample.compute()
-                if isinstance(sample, np.ndarray):
-                    sample = sample.item()
-            return isinstance(sample, cftime.datetime)
-        else:
-            return False
 
+    if array.dtype == np.dtype("O") and array.size > 0:
+        first_idx = (0,) * array.ndim
+        if isinstance(array, ExplicitlyIndexed):
+            first_idx = BasicIndexer(first_idx)
+        sample = array[first_idx]
+        return isinstance(np.asarray(sample).item(), cftime.datetime)
 
-def contains_cftime_datetimes(var) -> bool:
+    return False
+
+
+def contains_cftime_datetimes(var: T_Variable) -> bool:
     """Check if an xarray.Variable contains cftime.datetime objects"""
-    if var.dtype == np.dtype("O") and var.size > 0:
-        return _contains_cftime_datetimes(var.data)
-    else:
-        return False
+    return _contains_cftime_datetimes(var._data)
 
 
-def _contains_datetime_like_objects(var) -> bool:
+def _contains_datetime_like_objects(var: T_Variable) -> bool:
     """Check if a variable contains datetime like objects (either
     np.datetime64, np.timedelta64, or cftime.datetime)
     """