Merge remote-tracking branch 'upstream/master' into sparse-frame-accessor

Author: TomAugspurger

doc/source/whatsnew/v0.24.2.rst

@@ -2,8 +2,8 @@
 
 .. _whatsnew_0242:
 
-Whats New in 0.24.2 (February XX, 2019)
----------------------------------------
+Whats New in 0.24.2 (March 12, 2019)
+------------------------------------
 
 .. warning::
 
@@ -33,6 +33,8 @@ Fixed Regressions
 - Fixed regression in :class:`Categorical`, where constructing it from a categorical ``Series`` and an explicit ``categories=`` that differed from that in the ``Series`` created an invalid object which could trigger segfaults. (:issue:`25318`)
 - Fixed regression in :func:`to_timedelta` losing precision when converting floating data to ``Timedelta`` data (:issue:`25077`).
 - Fixed pip installing from source into an environment without NumPy (:issue:`25193`)
+- Fixed regression in :meth:`DataFrame.replace` where large strings of numbers would be coerced into ``int64``, causing an ``OverflowError`` (:issue:`25616`)
+- Fixed regression in :func:`factorize` when passing a custom ``na_sentinel`` value with ``sort=True`` (:issue:`25409`).
 - Fixed regression in :meth:`DataFrame.to_csv` writing duplicate line endings with gzip compress (:issue:`25311`)
 
 .. _whatsnew_0242.bug_fixes:
@@ -89,6 +91,7 @@ A total of 25 people contributed patches to this release. People with a "+" by t
 * Joris Van den Bossche
 * Josh
 * Justin Zheng
+* Kendall Masse
 * Matthew Roeschke
 * Max Bolingbroke +
 * rbenes +

doc/source/whatsnew/v0.25.0.rst

@@ -87,14 +87,15 @@ Other API Changes
 - :class:`DatetimeTZDtype` will now standardize pytz timezones to a common timezone instance (:issue:`24713`)
 - ``Timestamp`` and ``Timedelta`` scalars now implement the :meth:`to_numpy` method as aliases to :meth:`Timestamp.to_datetime64` and :meth:`Timedelta.to_timedelta64`, respectively. (:issue:`24653`)
 - :meth:`Timestamp.strptime` will now rise a ``NotImplementedError`` (:issue:`25016`)
--
+- Bug in :meth:`DatetimeIndex.snap` which didn't preserving the ``name`` of the input :class:`Index` (:issue:`25575`)
 
 .. _whatsnew_0250.deprecations:
 
 Deprecations
 ~~~~~~~~~~~~
 
 - Deprecated the `M (months)` and `Y (year)` `units` parameter of :func: `pandas.to_timedelta`, :func: `pandas.Timedelta` and :func: `pandas.TimedeltaIndex` (:issue:`16344`)
+- The functions :func:`pandas.to_datetime` and :func:`pandas.to_timedelta` have deprecated the ``box`` keyword. Instead, use :meth:`to_numpy` or :meth:`Timestamp.to_datetime64`/:meth:`Timedelta.to_timedelta64`. (:issue:`24416`)
 
 .. _whatsnew_0250.prior_deprecations:
 
@@ -215,10 +216,10 @@ I/O
 - Bug in :func:`read_json` for ``orient='table'`` when it tries to infer dtypes by default, which is not applicable as dtypes are already defined in the JSON schema (:issue:`21345`)
 - Bug in :func:`read_json` for ``orient='table'`` and float index, as it infers index dtype by default, which is not applicable because index dtype is already defined in the JSON schema (:issue:`25433`)
 - Bug in :func:`read_json` for ``orient='table'`` and string of float column names, as it makes a column name type conversion to Timestamp, which is not applicable because column names are already defined in the JSON schema (:issue:`25435`)
+- Bug in :func:`json_normalize` for ``errors='ignore'`` where missing values in the input data, were filled in resulting ``DataFrame`` with the string "nan" instead of ``numpy.nan`` (:issue:`25468`)
 - :meth:`DataFrame.to_html` now raises ``TypeError`` when using an invalid type for the ``classes`` parameter instead of ``AsseertionError`` (:issue:`25608`)
 - Bug in :meth:`DataFrame.to_string` and :meth:`DataFrame.to_latex` that would lead to incorrect output when the ``header`` keyword is used (:issue:`16718`)
 -
--
 
 
 Plotting
@@ -243,6 +244,7 @@ Reshaping
 - Bug in :func:`pandas.merge` adds a string of ``None`` if ``None`` is assigned in suffixes instead of remain the column name as-is (:issue:`24782`).
 - Bug in :func:`merge` when merging by index name would sometimes result in an incorrectly numbered index (:issue:`24212`)
 - :func:`to_records` now accepts dtypes to its `column_dtypes` parameter (:issue:`24895`)
+- Bug in :func:`concat` where order of ``OrderedDict`` (and ``dict`` in Python 3.6+) is not respected, when passed in as  ``objs`` argument (:issue:`21510`)
 
 
 Sparse

environment.yml

@@ -19,6 +19,7 @@ dependencies:
   - hypothesis>=3.82
   - isort
   - moto
+  - pycodestyle=2.4
   - pytest>=4.0.2
   - pytest-mock
   - sphinx

pandas/core/algorithms.py

@@ -619,13 +619,19 @@ def factorize(values, sort=False, order=None, na_sentinel=-1, size_hint=None):
 
     if sort and len(uniques) > 0:
         from pandas.core.sorting import safe_sort
-        try:
-            order = uniques.argsort()
-            order2 = order.argsort()
-            labels = take_1d(order2, labels, fill_value=na_sentinel)
-            uniques = uniques.take(order)
-        except TypeError:
-            # Mixed types, where uniques.argsort fails.
+        if na_sentinel == -1:
+            # GH-25409 take_1d only works for na_sentinels of -1
+            try:
+                order = uniques.argsort()
+                order2 = order.argsort()
+                labels = take_1d(order2, labels, fill_value=na_sentinel)
+                uniques = uniques.take(order)
+            except TypeError:
+                # Mixed types, where uniques.argsort fails.
+                uniques, labels = safe_sort(uniques, labels,
+                                            na_sentinel=na_sentinel,
+                                            assume_unique=True)
+        else:
             uniques, labels = safe_sort(uniques, labels,
                                         na_sentinel=na_sentinel,
                                         assume_unique=True)

pandas/core/categorical.py

@@ -794,10 +794,10 @@ def soft_convert_objects(values, datetime=True, numeric=True, timedelta=True,
         # Immediate return if coerce
         if datetime:
             from pandas import to_datetime
-            return to_datetime(values, errors='coerce', box=False)
+            return to_datetime(values, errors='coerce').to_numpy()
         elif timedelta:
             from pandas import to_timedelta
-            return to_timedelta(values, errors='coerce', box=False)
+            return to_timedelta(values, errors='coerce').to_numpy()
         elif numeric:
             from pandas import to_numeric
             return to_numeric(values, errors='coerce')

pandas/core/dtypes/cast.py

@@ -822,7 +822,7 @@ def _aggregate_multiple_funcs(self, arg, _level):
                     columns.append(com.get_callable_name(f))
             arg = lzip(columns, arg)
 
-        results = {}
+        results = collections.OrderedDict()
         for name, func in arg:
             obj = self
             if name in results:

pandas/core/groupby/generic.py

@@ -300,7 +300,8 @@ def asobject(self):
         return self.astype(object)
 
     def _convert_tolerance(self, tolerance, target):
-        tolerance = np.asarray(to_timedelta(tolerance, box=False))
+        tolerance = np.asarray(to_timedelta(tolerance).to_numpy())
+
         if target.size != tolerance.size and tolerance.size > 1:
             raise ValueError('list-like tolerance size must match '
                              'target index size')

pandas/core/indexes/datetimelike.py

@@ -787,8 +787,8 @@ def snap(self, freq='S'):
             snapped[i] = s
 
         # we know it conforms; skip check
-        return DatetimeIndex._simple_new(snapped, freq=freq)
-        # TODO: what about self.name?  tz? if so, use shallow_copy?
+        return DatetimeIndex._simple_new(snapped, name=self.name, tz=self.tz,
+                                         freq=freq)
 
     def join(self, other, how='left', level=None, return_indexers=False,
              sort=False):

pandas/core/indexes/datetimes.py

@@ -1079,7 +1079,7 @@ def coerce_to_target_dtype(self, other):
 
         try:
             return self.astype(dtype)
-        except (ValueError, TypeError):
+        except (ValueError, TypeError, OverflowError):
             pass
 
         return self.astype(object)
@@ -3210,7 +3210,7 @@ def _putmask_smart(v, m, n):
                 nv = v.copy()
                 nv[m] = nn_at
                 return nv
-    except (ValueError, IndexError, TypeError):
+    except (ValueError, IndexError, TypeError, OverflowError):
         pass
 
     n = np.asarray(n)

pandas/core/internals/blocks.py

@@ -253,7 +253,7 @@ def __init__(self, objs, axis=0, join='outer', join_axes=None,
 
         if isinstance(objs, dict):
             if keys is None:
-                keys = sorted(objs)
+                keys = com.dict_keys_to_ordered_list(objs)
             objs = [objs[k] for k in keys]
         else:
             objs = list(objs)

pandas/core/reshape/concat.py

@@ -9,6 +9,7 @@
     DateParseError, _format_is_iso, _guess_datetime_format, parse_time_string)
 from pandas._libs.tslibs.strptime import array_strptime
 from pandas.compat import zip
+from pandas.util._decorators import deprecate_kwarg
 
 from pandas.core.dtypes.common import (
     ensure_object, is_datetime64_dtype, is_datetime64_ns_dtype,
@@ -398,6 +399,7 @@ def _adjust_to_origin(arg, origin, unit):
     return arg
 
 
+@deprecate_kwarg(old_arg_name='box', new_arg_name=None)
 def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
                 utc=None, box=True, format=None, exact=True,
                 unit=None, infer_datetime_format=False, origin='unix',
@@ -444,6 +446,12 @@ def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
 
         - If True returns a DatetimeIndex or Index-like object
         - If False returns ndarray of values.
+
+        .. deprecated:: 0.25.0
+            Use :meth:`.to_numpy` or :meth:`Timestamp.to_datetime64`
+            instead to get an ndarray of values or numpy.datetime64,
+            respectively.
+
     format : string, default None
         strftime to parse time, eg "%d/%m/%Y", note that "%f" will parse
         all the way up to nanoseconds.

pandas/core/tools/datetimes.py

@@ -8,13 +8,15 @@
 
 from pandas._libs.tslibs import NaT
 from pandas._libs.tslibs.timedeltas import Timedelta, parse_timedelta_unit
+from pandas.util._decorators import deprecate_kwarg
 
 from pandas.core.dtypes.common import is_list_like
 from pandas.core.dtypes.generic import ABCIndexClass, ABCSeries
 
 from pandas.core.arrays.timedeltas import sequence_to_td64ns
 
 
+@deprecate_kwarg(old_arg_name='box', new_arg_name=None)
 def to_timedelta(arg, unit='ns', box=True, errors='raise'):
     """
     Convert argument to timedelta.
@@ -40,6 +42,12 @@ def to_timedelta(arg, unit='ns', box=True, errors='raise'):
         - If True returns a Timedelta/TimedeltaIndex of the results.
         - If False returns a numpy.timedelta64 or numpy.darray of
           values of dtype timedelta64[ns].
+
+        .. deprecated:: 0.25.0
+            Use :meth:`.to_numpy` or :meth:`Timedelta.to_timedelta64`
+            instead to get an ndarray of values or numpy.timedelta64,
+            respectively.
+
     errors : {'ignore', 'raise', 'coerce'}, default 'raise'
         - If 'raise', then invalid parsing will raise an exception.
         - If 'coerce', then invalid parsing will be set as NaT.

pandas/core/tools/timedeltas.py

@@ -281,6 +281,7 @@ def _recursive_extract(data, path, seen_meta, level=0):
             raise ValueError('Conflicting metadata name {name}, '
                              'need distinguishing prefix '.format(name=k))
 
-        result[k] = np.array(v).repeat(lengths)
+        # forcing dtype to object to avoid the metadata being casted to string
+        result[k] = np.array(v, dtype=object).repeat(lengths)
 
     return result

pandas/io/json/normalize.py

@@ -3164,11 +3164,11 @@ def converter(*date_cols):
                 return tools.to_datetime(
                     ensure_object(strs),
                     utc=None,
-                    box=False,
                     dayfirst=dayfirst,
                     errors='ignore',
                     infer_datetime_format=infer_datetime_format
-                )
+                ).to_numpy()
+
             except ValueError:
                 return tools.to_datetime(
                     parsing.try_parse_dates(strs, dayfirst=dayfirst))

pandas/io/parsers.py

@@ -1,6 +1,4 @@
 # -*- coding: utf-8 -*-
-import sys
-
 import pandas as pd
 from pandas import api
 from pandas.util import testing as tm
@@ -53,10 +51,10 @@ class TestPDApi(Base):
                ]
 
     # these are already deprecated; awaiting removal
-    deprecated_classes = ['TimeGrouper']
+    deprecated_classes = ['TimeGrouper', 'Panel']
 
     # these should be deprecated in the future
-    deprecated_classes_in_future = ['Panel']
+    deprecated_classes_in_future = []
 
     # external modules exposed in pandas namespace
     modules = ['np', 'datetime']
@@ -148,18 +146,3 @@ def test_deprecation_cdaterange(self):
         with tm.assert_produces_warning(FutureWarning,
                                         check_stacklevel=False):
             cdate_range('2017-01-01', '2017-12-31')
-
-
-class TestCategoricalMove(object):
-
-    def test_categorical_move(self):
-        # May have been cached by another import, e.g. pickle tests.
-        sys.modules.pop("pandas.core.categorical", None)
-
-        with tm.assert_produces_warning(FutureWarning):
-            from pandas.core.categorical import Categorical  # noqa
-
-        sys.modules.pop("pandas.core.categorical", None)
-
-        with tm.assert_produces_warning(FutureWarning):
-            from pandas.core.categorical import CategoricalDtype  # noqa

pandas/tests/api/test_api.py

@@ -14,7 +14,7 @@
 from pandas.core.dtypes.common import is_bool, is_list_like, is_scalar
 
 import pandas as pd
-from pandas import DataFrame, Panel, Series, date_range
+from pandas import DataFrame, Series, date_range
 from pandas.core.computation import pytables
 from pandas.core.computation.check import _NUMEXPR_VERSION
 from pandas.core.computation.engines import NumExprClobberingError, _engines
@@ -1112,14 +1112,6 @@ def test_bool_ops_with_constants(self):
             exp = eval(ex)
             assert res == exp
 
-    @pytest.mark.filterwarnings("ignore::FutureWarning")
-    def test_panel_fails(self):
-        x = Panel(randn(3, 4, 5))
-        y = Series(randn(10))
-        with pytest.raises(NotImplementedError):
-            self.eval('x + y',
-                      local_dict={'x': x, 'y': y})
-
     def test_4d_ndarray_fails(self):
         x = randn(3, 4, 5, 6)
         y = Series(randn(10))

pandas/tests/computation/test_eval.py

@@ -11,7 +11,6 @@
 from fractions import Fraction
 from numbers import Number
 import re
-from warnings import catch_warnings, simplefilter
 
 import numpy as np
 import pytest
@@ -30,8 +29,8 @@
 
 import pandas as pd
 from pandas import (
-    Categorical, DataFrame, DateOffset, DatetimeIndex, Index, Interval, Panel,
-    Period, Series, Timedelta, TimedeltaIndex, Timestamp, compat, isna)
+    Categorical, DataFrame, DateOffset, DatetimeIndex, Index, Interval, Period,
+    Series, Timedelta, TimedeltaIndex, Timestamp, compat, isna)
 from pandas.util import testing as tm
 
 
@@ -1305,10 +1304,6 @@ def test_is_scalar_pandas_containers(self):
         assert not is_scalar(Series([1]))
         assert not is_scalar(DataFrame())
         assert not is_scalar(DataFrame([[1]]))
-        with catch_warnings(record=True):
-            simplefilter("ignore", FutureWarning)
-            assert not is_scalar(Panel())
-            assert not is_scalar(Panel([[[1]]]))
         assert not is_scalar(Index([]))
         assert not is_scalar(Index([1]))
 

pandas/tests/dtypes/test_inference.py

@@ -808,11 +808,15 @@ def test_astype_to_incorrect_datetimelike(self, unit):
         other = "m8[{}]".format(unit)
 
         df = DataFrame(np.array([[1, 2, 3]], dtype=dtype))
-        with pytest.raises(TypeError):
+        msg = (r"cannot astype a datetimelike from \[datetime64\[ns\]\] to"
+               r" \[timedelta64\[{}\]\]").format(unit)
+        with pytest.raises(TypeError, match=msg):
             df.astype(other)
 
+        msg = (r"cannot astype a timedelta from \[timedelta64\[ns\]\] to"
+               r" \[datetime64\[{}\]\]").format(unit)
         df = DataFrame(np.array([[1, 2, 3]], dtype=other))
-        with pytest.raises(TypeError):
+        with pytest.raises(TypeError, match=msg):
             df.astype(dtype)
 
     def test_timedeltas(self):