Merge remote-tracking branch 'upstream/master' into boilerplate-4

Author: jorisvandenbossche

ci/deps/azure-37-locale.yaml

@@ -5,7 +5,8 @@ dependencies:
   - python=3.7.*
 
   # tools
-  - cython>=0.29.16
+  # Cython pin for https://github.com/pandas-dev/pandas/issues/34704
+  - cython==0.29.19
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - pytest-asyncio

doc/source/development/contributing.rst

@@ -136,6 +136,10 @@ want to clone your fork to your machine::
 This creates the directory `pandas-yourname` and connects your repository to
 the upstream (main project) *pandas* repository.
 
+Note that performing a shallow clone (with ``--depth==N``, for some ``N`` greater
+or equal to 1) might break some tests and features as ``pd.show_versions()``
+as the version number cannot be computed anymore.
+
 .. _contributing.dev_env:
 
 Creating a development environment

doc/source/user_guide/merging.rst

@@ -1273,7 +1273,7 @@ columns:
 
 .. ipython:: python
 
-   result = pd.merge(left, right, on='k', suffixes=['_l', '_r'])
+   result = pd.merge(left, right, on='k', suffixes=('_l', '_r'))
 
 .. ipython:: python
    :suppress:

doc/source/user_guide/visualization.rst

@@ -1332,7 +1332,7 @@ otherwise you will see a warning.
 
 .. ipython:: python
 
-   fig, axes = plt.subplots(4, 4, figsize=(6, 6))
+   fig, axes = plt.subplots(4, 4, figsize=(9, 9))
    plt.subplots_adjust(wspace=0.5, hspace=0.5)
    target1 = [axes[0][0], axes[1][1], axes[2][2], axes[3][3]]
    target2 = [axes[3][0], axes[2][1], axes[1][2], axes[0][3]]
@@ -1369,6 +1369,7 @@ Another option is passing an ``ax`` argument to :meth:`Series.plot` to plot on a
 .. ipython:: python
 
    fig, axes = plt.subplots(nrows=2, ncols=2)
+   plt.subplots_adjust(wspace=0.2, hspace=0.5)
    df['A'].plot(ax=axes[0, 0]);
    axes[0, 0].set_title('A');
    df['B'].plot(ax=axes[0, 1]);

doc/source/whatsnew/v1.0.5.rst

@@ -22,7 +22,8 @@ Fixed regressions
 
 Bug fixes
 ~~~~~~~~~
--
+
+- Fixed building from source with Python 3.8 fetching the wrong version of NumPy (:issue:`34666`)
 -
 
 Contributors

doc/source/whatsnew/v1.1.0.rst

@@ -288,9 +288,11 @@ Other enhancements
 - :meth:`HDFStore.put` now accepts `track_times` parameter. Parameter is passed to ``create_table`` method of ``PyTables`` (:issue:`32682`).
 - Make :class:`pandas.core.window.Rolling` and :class:`pandas.core.window.Expanding` iterable（:issue:`11704`)
 - Make ``option_context`` a :class:`contextlib.ContextDecorator`, which allows it to be used as a decorator over an entire function (:issue:`34253`).
+- :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` now accept an ``errors`` argument (:issue:`22610`)
 - :meth:`groupby.transform` now allows ``func`` to be ``pad``, ``backfill`` and ``cumcount`` (:issue:`31269`).
 - :meth:`~pandas.io.json.read_json` now accepts `nrows` parameter. (:issue:`33916`).
 - :meth `~pandas.io.gbq.read_gbq` now allows to disable progress bar (:issue:`33360`).
+- :meth:`~pandas.io.gbq.read_gbq` now supports the ``max_results`` kwarg from ``pandas-gbq`` (:issue:`34639`).
 
 .. ---------------------------------------------------------------------------
 
@@ -401,6 +403,8 @@ Backwards incompatible API changes
 - :func: `pandas.api.dtypes.is_string_dtype` no longer incorrectly identifies categorical series as string.
 - :func:`read_excel` no longer takes ``**kwds`` arguments. This means that passing in keyword ``chunksize`` now raises a ``TypeError``
   (previously raised a ``NotImplementedError``), while passing in keyword ``encoding`` now raises a ``TypeError`` (:issue:`34464`)
+- :func: `merge` now checks ``suffixes`` parameter type to be ``tuple`` and raises ``TypeError``, whereas before a ``list`` or ``set`` were accepted and that the ``set`` could produce unexpected results (:issue:`33740`)
+- :class:`Period` no longer accepts tuples for the ``freq`` argument (:issue:`34658`)
 
 ``MultiIndex.get_indexer`` interprets `method` argument differently
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -826,6 +830,7 @@ Timedelta
 - Bug in :func:`timedelta_range` that produced an extra point on a edge case (:issue:`30353`, :issue:`33498`)
 - Bug in :meth:`DataFrame.resample` that produced an extra point on a edge case (:issue:`30353`, :issue:`13022`, :issue:`33498`)
 - Bug in :meth:`DataFrame.resample` that ignored the ``loffset`` argument when dealing with timedelta (:issue:`7687`, :issue:`33498`)
+- Bug in :class:`Timedelta` and `pandas.to_timedelta` that ignored `unit`-argument for string input (:issue:`12136`)
 
 Timezones
 ^^^^^^^^^
@@ -892,6 +897,7 @@ Indexing
 - Bug in :meth:`DataFrame.truncate` and :meth:`Series.truncate` where index was assumed to be monotone increasing (:issue:`33756`)
 - Indexing with a list of strings representing datetimes failed on :class:`DatetimeIndex` or :class:`PeriodIndex`(:issue:`11278`)
 - Bug in :meth:`Series.at` when used with a :class:`MultiIndex` would raise an exception on valid inputs (:issue:`26989`)
+- Bug in :meth:`DataFrame.loc` with dictionary of values changes columns with dtype of ``int`` to ``float`` (:issue:`34573`)
 - Bug in :meth:`Series.loc` when used with a :class:`MultiIndex` would raise an IndexingError when accessing a None value (:issue:`34318`)
 
 Missing
@@ -1012,6 +1018,7 @@ Reshaping
 - Ensure only named functions can be used in :func:`eval()` (:issue:`32460`)
 - Bug in :func:`Dataframe.aggregate` and :func:`Series.aggregate` was causing recursive loop in some cases (:issue:`34224`)
 - Fixed bug in :func:`melt` where melting MultiIndex columns with ``col_level`` > 0 would raise a ``KeyError`` on ``id_vars`` (:issue:`34129`)
+- Bug in :meth:`Series.where` with an empty Series and empty ``cond`` having non-bool dtype (:issue:`34592`)
 
 Sparse
 ^^^^^^

environment.yml

@@ -12,7 +12,7 @@ dependencies:
   - asv
 
   # building
-  - cython>=0.29.16
+  - cython=0.29.19
 
   # code checks
   - black=19.10b0

pandas/_libs/tslibs/__init__.py

@@ -1,4 +1,5 @@
 __all__ = [
+    "dtypes",
     "localize_pydatetime",
     "NaT",
     "NaTType",
@@ -17,7 +18,7 @@
     "to_offset",
 ]
 
-
+from . import dtypes
 from .conversion import localize_pydatetime
 from .nattype import NaT, NaTType, iNaT, is_null_datetimelike, nat_strings
 from .np_datetime import OutOfBoundsDatetime

pandas/_libs/tslibs/conversion.pyx

@@ -785,7 +785,6 @@ cpdef ndarray[int64_t] normalize_i8_timestamps(const int64_t[:] stamps, tzinfo t
         int64_t[:] deltas
         str typ
         Py_ssize_t[:] pos
-        npy_datetimestruct dts
         int64_t delta, local_val
 
     if tz is None or is_utc(tz):
@@ -795,16 +794,14 @@ cpdef ndarray[int64_t] normalize_i8_timestamps(const int64_t[:] stamps, tzinfo t
                     result[i] = NPY_NAT
                     continue
                 local_val = stamps[i]
-                dt64_to_dtstruct(local_val, &dts)
-                result[i] = _normalized_stamp(&dts)
+                result[i] = _normalize_i8_stamp(local_val)
     elif is_tzlocal(tz):
         for i in range(n):
             if stamps[i] == NPY_NAT:
                 result[i] = NPY_NAT
                 continue
             local_val = tz_convert_utc_to_tzlocal(stamps[i], tz)
-            dt64_to_dtstruct(local_val, &dts)
-            result[i] = _normalized_stamp(&dts)
+            result[i] = _normalize_i8_stamp(local_val)
     else:
         # Adjust datetime64 timestamp, recompute datetimestruct
         trans, deltas, typ = get_dst_info(tz)
@@ -816,38 +813,36 @@ cpdef ndarray[int64_t] normalize_i8_timestamps(const int64_t[:] stamps, tzinfo t
                 if stamps[i] == NPY_NAT:
                     result[i] = NPY_NAT
                     continue
-                dt64_to_dtstruct(stamps[i] + delta, &dts)
-                result[i] = _normalized_stamp(&dts)
+                local_val = stamps[i] + delta
+                result[i] = _normalize_i8_stamp(local_val)
         else:
             pos = trans.searchsorted(stamps, side='right') - 1
             for i in range(n):
                 if stamps[i] == NPY_NAT:
                     result[i] = NPY_NAT
                     continue
-                dt64_to_dtstruct(stamps[i] + deltas[pos[i]], &dts)
-                result[i] = _normalized_stamp(&dts)
+                local_val = stamps[i] + deltas[pos[i]]
+                result[i] = _normalize_i8_stamp(local_val)
 
     return result.base  # `.base` to access underlying ndarray
 
 
-cdef inline int64_t _normalized_stamp(npy_datetimestruct *dts) nogil:
+@cython.cdivision
+cdef inline int64_t _normalize_i8_stamp(int64_t local_val) nogil:
     """
-    Normalize the given datetimestruct to midnight, then convert to int64_t.
+    Round the localized nanosecond timestamp down to the previous midnight.
 
     Parameters
     ----------
-    *dts : pointer to npy_datetimestruct
+    local_val : int64_t
 
     Returns
     -------
-    stamp : int64
-    """
-    dts.hour = 0
-    dts.min = 0
-    dts.sec = 0
-    dts.us = 0
-    dts.ps = 0
-    return dtstruct_to_dt64(dts)
+    int64_t
+    """
+    cdef:
+        int64_t day_nanos = 24 * 3600 * 1_000_000_000
+    return local_val - (local_val % day_nanos)
 
 
 @cython.wraparound(False)

pandas/_libs/tslibs/dtypes.pxd

@@ -1,4 +1,21 @@
 
+cdef enum c_FreqGroup:
+    # Mirrors FreqGroup in the .pxy file
+    FR_ANN = 1000
+    FR_QTR = 2000
+    FR_MTH = 3000
+    FR_WK = 4000
+    FR_BUS = 5000
+    FR_DAY = 6000
+    FR_HR = 7000
+    FR_MIN = 8000
+    FR_SEC = 9000
+    FR_MS = 10000
+    FR_US = 11000
+    FR_NS = 12000
+    FR_UND = -10000  # undefined
+
+
 cdef enum PeriodDtypeCode:
     # Annual freqs with various fiscal year ends.
     # eg, 2005 for A_FEB runs Mar 1, 2004 to Feb 28, 2005
@@ -50,7 +67,9 @@ cdef enum PeriodDtypeCode:
     U = 11000       # Microsecondly
     N = 12000       # Nanosecondly
 
+    UNDEFINED = -10_000
+
 
-cdef class PeriodPseudoDtype:
+cdef class PeriodDtypeBase:
     cdef readonly:
         PeriodDtypeCode dtype_code

pandas/_libs/tslibs/dtypes.pyx

@@ -2,7 +2,7 @@
 # originals
 
 
-cdef class PeriodPseudoDtype:
+cdef class PeriodDtypeBase:
     """
     Similar to an actual dtype, this contains all of the information
     describing a PeriodDtype in an integer code.
@@ -14,9 +14,9 @@ cdef class PeriodPseudoDtype:
         self.dtype_code = code
 
     def __eq__(self, other):
-        if not isinstance(other, PeriodPseudoDtype):
+        if not isinstance(other, PeriodDtypeBase):
             return False
-        if not isinstance(self, PeriodPseudoDtype):
+        if not isinstance(self, PeriodDtypeBase):
             # cython semantics, this is a reversed op
             return False
         return self.dtype_code == other.dtype_code
@@ -106,3 +106,20 @@ _period_code_map.update({
     "W": 4000,   # Weekly
     "C": 5000,   # Custom Business Day
 })
+
+
+class FreqGroup:
+    # Mirrors c_FreqGroup in the .pxd file
+    FR_ANN = 1000
+    FR_QTR = 2000
+    FR_MTH = 3000
+    FR_WK = 4000
+    FR_BUS = 5000
+    FR_DAY = 6000
+    FR_HR = 7000
+    FR_MIN = 8000
+    FR_SEC = 9000
+    FR_MS = 10000
+    FR_US = 11000
+    FR_NS = 12000
+    FR_UND = -10000  # undefined

pandas/_libs/tslibs/frequencies.pxd

@@ -1,5 +1,3 @@
 cdef dict attrname_to_abbrevs
 
-cpdef get_freq_code(freqstr)
 cpdef int get_to_timestamp_base(int base)
-cpdef str get_freq_str(base, mult=*)