Create Holiday Calendar for NYSE Stock Market

[cancan101]

Using new code in #6719

See: http://www.nyx.com/holidays-and-hours/nyse
and with some guidlines: http://www.investmentcontrarians.com/nyse-2013-stock-market-holidays/

[rockg]

I believe @jreback was inclined to not have any calendars specified in pandas. I would see a benefit to having widely used calendars in pandas so work isn't reproduced and the community can use a blessed calendar definition. Now, the trick is to determine what is 'widely used.'

[jreback]

I suppose something like: https://github.com/pydata/pandas/blob/master/pandas/io/date_converters.py

would be ok, sort of a community contributed 'list' of holiday calendars; would not be imported into any pandas namespace (but would need some minimal tests).

[rockg]

@cancan101, as we discussed awhile back when we were creating the calendars how should the half-day holidays be represented? Should there be hourly granularity or simply knowing the day? Or should there be two calendars...one with the holidays as full days and one with half. How would you use this in practice which I guess will make the choice obvious?

[cancan101]

To start with I think the most useful calendar is full market holidays (ie
when the market is closed for the entire day).

Half days can be deferred.
On May 9, 2014 8:35 AM, "rockg" [email protected] wrote:

@cancan101 https://github.com/cancan101, as we discussed awhile back
when we were creating the calendars how should the half-day holidays be
represented? Should there be hourly granularity or simply knowing the day?
Or should there be two calendars...one with the holidays as full days and
one with half. How would you use this in practice which I guess will make
the choice obvious?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7070#issuecomment-42660918
.

[MichaelWS]

I am not sure if this should be implemented in pandas. There will be constant pull requests for changes and zipline has it already

[cancan101]

Why do you think that there will be constant PRs for changes?
Is the Zipline implementation usable as a Pandas Custom BDay offset?

[MichaelWS]

Well, there would be a PR request for every new holiday or exchange closing that was not scheduled.
I am not sure where this should be. If anyone wants to work with me, I would open source a trading calendar. Where would we put it?
Here's zipline's implementation
https://github.com/quantopian/zipline/blob/master/zipline/utils/tradingcalendar.py

[MichaelWS]

I think half days should be separate, i.e. list of half days and list of holidays

[cancan101]

I actually have my reservation about combining the lists of scheduled and unscheduled holidays: quantopian/zipline#334.

I think adding rules for scheduled holidays (which change infrequently) is reasonable.

[jreback]

You have to remember, that a user can always sub-class the Holiday Calenders and add/modify as needed.

[rockg]

I find the current holiday implementation in pandas to be very concise and effective (instead of >400 lines, it can probably be done in about 20 as all the rrule stuff is abstracted away and the scheduled holidays would be 9 lines, equal to the number of holidays). @MichaelWS, are you suggesting making a Holiday package? I think that would be fine, but I would strongly suggest we use HolidayCalendar in pandas as the basis.

[MichaelWS]

I just don't like the concept that users would get a NYSE calendar and not actually have all the holidays.
@rockg I am suggesting making a Holiday package using HolidayCalendar.

[ehebert]

I can confer with the rest of the maintaners of Zipline about breaking out the calendar rules, hich use pandas CDay under the hood, into a separate project that updates, when appropriate, to include unscheduled holidays, ala the updates done to the pytz library to keep up-to-date.

[rockg]

@ehebert Any updates?

[ehebert]

@rockg https://github.com/quantopian/tradingcalendar now contains the extracted tradingcalendar modules from https://github.com/quantopian/zipline

That project could either be developed as a standalone library which is updated when new holiday exceptions come out or bugs are discover rules, much like the release cycle of pytz. (Though timezones are much more volatile than the trading calendar.) I may continue some work on that path in the short term, since I currently have a few scripts that do not need all of Zipline, but do need the calendar with support for half days. If there is interest there, conversation can be carried over to the tradingcalendar GitHub repo.

Or those libraries could be used as a base to upstream a Holiday calendar to pandas, to that end I can help assign copyright and figure out re-licensing, and contribute some coding it looks like https://github.com/pydata/pandas/blob/master/pandas/tseries/holiday.py with a NYSE version of USFederalHolidayCalendar, but for our code to use it, we would need to add half day support. The NYSE calendar would be easiest since it's work was done wholly by Quantopian employees, whereas non-NYSE calendars, like the Toronto exchange were provided by community members.

Apologies to pandas maintainers if this discussion is trampling too much, and let me know if we should move the discussion off list, but hoping that sharing the rules we've been using is beneficial to everyone.

[MichaelWS]

ok, can we convert this to pandas.

i agree with @cancan101 on the scheduled vs unscheduled. I would include in separate list and have a way of combining it.

How would would have a holiday calendar that changes? For example, NYSE has changed holidays through time.

do we have any opinion where this should live?

[rockg]

@MichaelWS Holiday has start_date/end_date to support changed holidays through time.

[nehalecky]

Hey all, this is great stuff.

While I understand that pandas development began out of applications in the US financial domain, I know it's used in many other areas. In this regard, there are likely other types of holiday calendars that will be useful for different users. Holiday calendars are important for time series data associated with many types of human activities.

On that note, I'm interested to understand how will pandas manage a possibly large number of calendars across a spectrum of different domains?

[MichaelWS]

this part of my concern before we all work on this. I want to see how we can have this work going forward and have people add any major reasonable calendar.

Also, What would tests look like?

[jreback]

I think a nice way to do this would be to have separate repo / repos which maintain the actual holiday calendars. I would maintain these prob using .py files (with tests!) in an organized way, registering to the pandas holiday module on import (its possible pandas can provide an easier/better method for this).

Easy enough for people to clone the repo and just import these files.

Then this can be updated pretty easily by multiple groups and it is not subject to pandas release schedule.

[MichaelWS]

what would the tests look like? Have a few dates hard coded and then verified?

[jreback]

@MichaelWS yep (mainly to verify that changes to the file/s are not corrupting it, e.g. typos and such).

[nehalecky]

@MichaelWS, right, and I should have noted your previous comments regarding those concerns, we're totally on the same page here. :)

@jreback, grand. This is exactly how I was envisioning the implementation! A separate repo with independent release cycle is ideal.

I like the discussion on tests and all, but what about a name? :)

[MichaelWS]

also, how would it be included in the pydata tool chain?

[jreback]

why don't u start it as a reg repo
can prob add to the pydata org at some point

[MichaelWS]

easy enough.

One question though. How would I add holidays that are one off?
for example President Nixon died on 4/27/1994 and this was an exchange holiday.

[rockg]

Just define a holiday with year, month, and day and it will only return that day. Holiday('President Nixon Death', year=1994, month=4, day=27).

[MichaelWS]

am I ok to open up pr's for next_workday and previous_workday for pandas for this?

[rockg]

What do you mean by next_workday and previous_workday? All regular DateOffsets are supported by this. Do you have an example? In general, I think any new rules would go under a pandas PR and any holidays/calendars would go under the other repository.

[jreback]

https://github.com/pydata/pandas/blob/master/pandas/tseries/holiday.py#L49

[MichaelWS]

i saw nearest_workday.

Take for example the July 4th holiday. It is observed using nearest_workday, but previous_workday is a half day.

[rockg]

Wouldn't that be simply a -BDay() offset? I don't know how we want to represent half days, but we could do -BDay() + Hour(13) or something like that to denote when the holiday starts.

[MichaelWS]

you would need an observance rule for that though, i.e. nearest workday - BDAY

I was thinking of having a numpy array of timestamps for half days

[rockg]

I don't see how a numpy array of timstamps would work. Holiday calendars are an aggregation of Holidays which define rules or specific days. Can you provide a more fleshed out example of what you are thinking for half days?

[MichaelWS]

sure, I was thinking of having a financial calendar where we can ask for any date what time the market is open on a given day, early_closes, and holidays.

the class US_Stocks or whatever we wanted to call it would have holidays, early closes and rules around market_times.

I will post a more fleshed out example in a day or two.

[MichaelWS]

Also, for trading times, how would we handle it? Should it return a dict? I am trying to think about countries with lunches as well.

[native-api]

IMO country-specific stuff belongs in optional addons unless it supports (or at least, intends to support) all countries on the same level.

[rsheftel]

I had a need for market and exchange calendars and was lead to this thread. Inspired by it I made a fork of the Zipline project to just extract the trading calendar information and put it into a new project that creates pandas objects for holidays, date_range and DatetimeIndex. I hope people find it useful:

https://pypi.python.org/pypi/pandas-market-calendars

https://github.com/rsheftel/pandas_market_calendars

[jreback]

closing. this could be an external package.