pipeline for Quidel covidtest

[jingjtang]

• Pull data from the email attachment
• The pulling step is very time consuming especially when we have much more dates to consider, so now:
  • store the intermediate file (aggregated by zip with number of total tests, number of positive tests) as cache
  • only update new data when run next time
• ~1.97% invalid tests, Now treat them as negative tests
• Geographical poolling is added since the data volume is now enough. The minimum sample size is 50. If there is no data report for a state/hrr/msa/county, it is filtered out because of this.
• The data start from 5/9 but has only extremely low volume. So, temporarily set the export_start_date to be 5/26.
• As for the time stamp:
  • Use test date as time stamp for normal cases
  • If storage date is earlier than test date which is abnormal, remove those cases. (0.10%)
  • If storage date - test date > 90 days. (0.01%)
• Report data through today (GMT) minus 5 days due to backfill
• Do we want to report county level data? (did not report county level for flu tests due to the data volume problem, there is still the same problem for COVID Test)

[krivard]

Looks good! This will update from 5/26 to (today minus 5 days) each day it is run, right? I believe that's what we want.

Other todos:

• API documentation
• Missingness documentation
• Some kind of pipeline summary/developer documentation in the README, a la this outline in the facebook package

[eujing]

Looks good to me!