KeyError in datetime pandas when accessing with datetime.date

[Magneto49-3]

I've just upgrade Big Sur 11.0.1 and I'm facing some issue while trying to execute a notebook. Weird because it worked on Catalina. Could anyone help ?

import numpy as np
import pandas as pd
import yfinance as yf


# In[56]:


#parameters
RISKY_ASSET = 'ADBE'
START_DATE = '2017-01-01'
END_DATE = '2020-07-31'


# In[57]:


#Download data:
df = yf.download(RISKY_ASSET, start=START_DATE, end=END_DATE, adjusted=True)


# In[58]:


#daily return
adj_close = df['Adj Close']
returns = adj_close.pct_change().dropna()
print(f'Average return: {100 * returns.mean():.2f}%')
returns.plot(title=f'{RISKY_ASSET} returns: {START_DATE} - {END_DATE}')


# In[59]:


#Split the data into training and test sets:
train = returns['2017-01-01':'2020-06-30']
test = returns['2020-07-01':'2020-07-31']


# In[60]:


#parameters of the simulation:
T = len(test)
N = len(test)
S_0 = adj_close[train.index[-1].date()]

N_SIM = 1000
mu = train.mean()
sigma = train.std()

And here the errors:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-9316c025c623> in <module>
      2 T = len(test)
      3 N = len(test)
----> 4 S_0 = adj_close[train.index[-1].date()]
      5 N_SIM = 1000
      6 mu = train.mean()

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/series.py in __getitem__(self, key)
    880 
    881         elif key_is_scalar:
--> 882             return self._get_value(key)
    883 
    884         if is_hashable(key):

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/series.py in _get_value(self, label, takeable)
    987 
    988         # Similar to Index.get_value, but we do not fall back to positional
--> 989         loc = self.index.get_loc(label)
    990         return self.index._get_values_for_loc(self, loc, label)
    991 

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)
    620         else:
    621             # unrecognized type
--> 622             raise KeyError(key)
    623 
    624         try:

KeyError: datetime.date(2019, 12, 31) `

[jbrockmendel]

Can you post a copy/paste-able example https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

Best guess is that it isn't Big Sur, but that you recently updated to a newer version of pandas. There have been a few issues involving datetime.date objects recently. You're safest bet is to always use datetime.datetime or pd.Timestamp objects instead.

[Magneto49-3]

Can you post a copy/paste-able example https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

Best guess is that it isn't Big Sur, but that you recently updated to a newer version of pandas. There have been a few issues involving datetime.date objects recently. You're safest bet is to always use datetime.datetime or pd.Timestamp objects instead.

Exactly, I have downgraded it to 1.0.5 and everything works know ! Thank you mate !

[MarcoGorelli]

I have downgraded it to 1.0.5 and everything works know !

Wouldn't S_0 = adj_close[train.index[-1]] also have worked?

Simpler reproducer:

>>> _ = pd.Series([0], index=[pd.Timestamp('2020-01-01')])[pd.Timestamp('2020-01-01')]  # works
>>> _ = pd.Series([0], index=[pd.Timestamp('2020-01-01')])[pd.Timestamp('2020-01-01').date()]  # error
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marco/tmp/venv/lib/python3.8/site-packages/pandas/core/series.py", line 851, in __getitem__
    return self._get_value(key)
  File "/home/marco/tmp/venv/lib/python3.8/site-packages/pandas/core/series.py", line 959, in _get_value
    loc = self.index.get_loc(label)
  File "/home/marco/tmp/venv/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 683, in get_loc
    raise KeyError(key)
KeyError: datetime.date(2020, 1, 1)

As was suggested above:

You're safest bet is to always use datetime.datetime or pd.Timestamp objects instead

---

Anyway, from git bisect:

9b0ef5d07fb218df4e36e133d69b1ea4c6be43bd is the first bad commit
commit 9b0ef5d07fb218df4e36e133d69b1ea4c6be43bd
Author: jbrockmendel <[email protected]>
Date:   Tue Jan 14 19:10:24 2020 -0800

    refactor DTI.get_loc (#31023)

 pandas/core/indexes/datetimes.py | 77 ++++++++++++++++++----------------------
 1 file changed, 34 insertions(+), 43 deletions(-)

#31023

[jbrockmendel]

@KylePoe you're welcome to make a PR

[diprajkadlag]

Can you post a copy/paste-able example https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

Best guess is that it isn't Big Sur, but that you recently updated to a newer version of pandas. There have been a few issues involving datetime.date objects recently. You're safest bet is to always use datetime.datetime or pd.Timestamp objects instead.

datetime.datetime(2016,1,1) worked for me, while I was trying datetime.date(2016,1,1). My pandas version is 1.4.1. As soon I updated my pandas library this error started showing up. New versions of pandas accept datetime.datetime rather than datetime.date!