.loc lookup with missing values

[jennolsen84]

Hi everyone, I recently upgraded from 0.16.2 to 0.17.0, and I am wondering if this change was intentional:

0.16.2
(numpy version 1.10)

In [15]: s
Out[15]: 
             0
2001-01-01   2
2001-01-02   5
2001-01-03   8
2001-01-04  11

In [16]: ddd
Out[16]: array(['2001-01-04', '2001-01-02', '2001-01-04', '2001-01-14'], dtype='datetime64[D]')

In [17]: s.loc[ddd]
Out[17]: 
             0
2001-01-04  11
2001-01-02   5
2001-01-04  11
2001-01-14 NaN

pd version 0.17.0
np version 1.10.1

There's a long message, but here is what happens at the end:
In [12]: ddd = np.array(['2001-01-04','2001-01-02','2001-01-04','2001-01-14'], dtype='datetime64')

In [13]: s = pd.DataFrame([2, 5, 8, 11], pd.date_range('2001-01-01', freq='D', periods=4))

In [14]: s.loc[ddd]
ValueError: Inferred frequency None from passed dates does not conform to passed frequency D

I realize that 2001-01-14 is not in the index of series s, and I was hoping that .loc would just fill it in as nan. I can understand if it threw a KeyError, but I am now getting a ValueError. Was this change intentional? I checked the docs, but I didn't see it there.

[jorisvandenbossche]

I can confirm this, and this does not sound as intentional I think.

Using reindex still works as expected:

In [6]:  s.reindex(ddd)
Out[6]:
             0
2001-01-04  11
2001-01-02   5
2001-01-04  11
2001-01-14 NaN

[jennolsen84]

not sure if this is related, but this also looks off. The tests pass when timezones are not used.

In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: d = np.datetime64('2000-01-01T00Z')
In [4]: s = pd.Series([17], index=pd.DatetimeIndex(['2000-01-01'], tz='UTC'))
In [5]: s.ix[d]
Out[5]: 17
In [6]: s.reindex([d])
Out[6]: 
2000-01-01   NaN
dtype: float64
In [7]: s.ix[[d]]
Out[7]: 
2000-01-01   NaN
dtype: float64
In [8]: pd.__version__
Out[8]: '0.17.1'
In [9]: np.__version__
Out[9]: '1.10.2'
In [10]: s
Out[10]: 
2000-01-01 00:00:00+00:00    17
dtype: int64

[mroeschke]

This looks fixed on master, but as the warning notes this will deprecated in the future.

In [1]: In [12]: ddd = np.array(['2001-01-04','2001-01-02','2001-01-04','2001-01-14'], dtype='datetime64')
   ...:
   ...: In [13]: s = pd.DataFrame([2, 5, 8, 11], pd.date_range('2001-01-01', freq='D', periods=4))
   ...:
   ...: In [14]: s.loc[ddd]
/anaconda3/envs/pandas-dev/bin/ipython:5: FutureWarning:
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
  import sys
Out[1]:
               0
2001-01-04  11.0
2001-01-02   5.0
2001-01-04  11.0
2001-01-14   NaN

In [2]: pd.__version__
Out[2]: '0.25.0.dev0+334.g7721f7009'

[anordin95]

Does this still need unit testing? I'm new to working on pandas and I'm not sure if I'd be doing redundant work here.

[mroeschke]

Sure. Feel free to add a unit test for this.