Add more Safegraph indicators

[huisaddison]

Safegraph offers other datasets, like bar and restaurant... much larger dataset, harder to wrangle with, but would be useful for forecasting, especially causal stuff being done by Larry et al.

@capnrefsmmat

[capnrefsmmat]

Brief context to aid prioritization: The current indicators hardly seem to change over time, even as cases increase dramatically. We expect data on bar and restaurant opening or use to be more informative, since it seems a lot of spread comes that way. It would also change much more as restrictions are imposed and lifted.

The Facebook survey revision will have items on restaurant/bar patronage, but SafeGraph has it historically.

[jingjtang]

Datasets related:

• Neighborhood Patterns

  • Monthly
  • Census block group Level
  • Potential interesting features:
    • weekday_device_home_areas
    • weekend_device_home_areas
    • breakfast_device_home_areas
    • lunch_device_home_areas
    • afternoon_tea_device_home_areas
    • dinner_device_home_areas
    • nightlife_device_home_areas
    • others

• Weekly Pattern

  • Weekly
  • 5 digit zip code level

[capnrefsmmat]

Larry says, about additional SafeGraph data, "it's not urgent but I do need it eventually." Not sure what priority that gives it...

[krivard]

The bar and restaurant signals are part of the neighborhood dataset, which is only available my month (with a by-weekday breakdown, but not a by-day breakdown). Is that useful?

There are individual business addresses available by day (by hour even) in a weekly rollup, but we'd need Larry to tell us which NAICS codes he's interested in.

[capnrefsmmat]

Got it; will ask modeling to make a list of codes they're interested in.

[jingjtang]

core_poi.csv in Core Places :
lists all of the points of interest with location name, address, category, and brand association (> 1 GB).
It will be updated each month.

Patterns.csv in Monthly Places Patterns (from 01/2018 to present):
has daily number of visits, each csv file represents a dataset for a month.
Each row represents a place with Safegraph_id, location name, address including zip code etc.
Each cell in the visit_by_day column is a list of length 30-31 (number of visit for each day in this month).
(Similar in Weekly Patterns)

neighborhood_patterns
Each row is a Census block group.

• stops_by_day
• stops_by_each_hour
• popularity_by_hour_Monday/Tuesday/...

[capnrefsmmat]

Larry would like these NAICS codes: bars (722410) and restaurants (722511).

[huisaddison]

Hi all - spoke briefly with @jingjtang about this. Seems like some of the datasets available monthly actually have data on a daily resolution?

Some of my thoughts:

• If the dataset is updated monthly (even if it is available at a daily resolution), we may want to prioritize data that is updated weekly (because forecasting would need "as real time as possible" data to produce predictions for the future). However, this may be less important of a distinction for Larry and others doing causal modeling, since (to my understanding) they are trying to fit causal models to historical data to understand causal effects before incorporating them into a forward-looking model (at which point the immediate-availability of data becomes more important).
• The full list of Safegraph-provided datasets is here: https://docs.google.com/spreadsheets/d/1UNWvPzkUTTlXBZ6M6iGhM_7sr8h-MxsZdE7iOszkAmk/edit#gid=0 One dataset that we may have overlooked is row 9: Transaction Data (Facteus). It is more thoroughly described here: https://docs.google.com/document/d/1cInmNDf59BCLl1amm09dIH1GM5vYbJf80TGDbraiX30/edit#heading=h.kzfy1kkxiyze
  • I think that Cut 2 may be particularly useful for us (Total spend per Merchent Category Code (MCC) per zip code). This would allow us to obtain zip-code level amounts for total amount spent over m transactions on n cards for each MCC. MCCs of interest may include: 5812 (Eating Places and Restaurants), 5813 (Drinking places, e.g. bars), 5814 (Fast Food Restaurants). Full list of MCCs here: https://usa.visa.com/content/dam/VCOM/download/merchants/visa-merchant-data-standards-manual.pdf
• Finally, (and I hope I'm not overstepping by suggesting this), since these new signals are being requested for a very specific purpose by Larry (and other causal modelers at CMU but outside Delphi), I would encourage Jingjing and others to set up a Zoom call with Larry once we know all the data that's available, so that design decisions (e.g., normalize by population?) can be made with the end-user in mind.

[huisaddison]

I had a short chat with Larry today - his team is interested in "how many people are going to bars (722410) and restaurants (722511) in each state (Rob also wants county) at a daily resolution".

He suggested we also normalize by population, since the existing signals are already normalized by population.

Since @jingjtang may already have her hands full with other work, I offered to take care of this, since Larry's team needs at least a WIP version of this very soon. I will attend tomorrow's eng meeting to discuss, since I know a lot of the eng working procedures have changed since I last wrote a pipeline. @jsharpna @krivard .

[jingjtang]

Problems encountered when trying to switch to utility @jsharpna:

1. No zip_to_msa or zip_to_state
2. function zip_to_countyand zip_to_hrr met with NotImplementedError: Can't perform this operation for loaders without 'get_data()'
3. Didn't find a function for getting population information, still need a population file in ./static

[krivard]

#225 merged. Next:

• correlations review
• visual review -- either with wip or staging
• signal name approval
• nominate for public release