Missing data in GHT

[huisaddison]

Actual Behavior:

Data missing for many MSA / States on July 11, 2020; August 18 - 20, 2020

Expected behavior

Data is there for those days.

Context

GHT should not have "missing data" - the usual behavior of the API is to report 0 when seearches are below a minimum threshold. This suggests to me that there was a glitch in our pipeline automation. We should try rerunning the pipeline for those days to see if this fixes the issue

cc @RoniRos who pointed this out
cc @jingjtang who I b elieve is the current pipeline maintainer

[huisaddison]

@jingjtang in response to the pipeline being automated (which you commented on another issue) Is there any chance that the GHT API did not report data for a really long time, and somehow our automation didn't "look far back" enough when it finally updated?

I can look into this by directly querying the GHT API for those days data tomorrow (Th/F Delphi days) but if somebody else beats me to it they should post first

[jingjtang]

@huisaddison It could be possible since initially the automation just uploaded the latest report to the API. But later we have added code for archiving to solve the backfill problem. @korlaxxalrok should know more details about what happened. I will look into the data for those days too.