Merged quidel_covidtest and quidel_flutest

quidel/DETAILS.md

@@ -0,0 +1,75 @@
+# Quidel Test Data
+
+### Background
+Starting from 2014-08, we began getting flu test data from Quidel. The data contains a number of features for every test, including localization at 5-digit Zip Code level, a TestDate and StorageDate, patient age, and several identifiers that uniquely identify the device on which the test was performed (SofiaSerNum, the individual test (FluTestNum), and the result (ResultID). Multiple tests are stored on each device and we suspect that the device identifiers could potentially be useful to normalize the results over time.
+
+Starting from 2020-05-09, we began getting Quidel COVID Test data and started reporting it from May 26, 2020 due to limitation in the data volume. The data contains a number of features for every test, including localization at 5-digit Zip Code level, a TestDate and StorageDate, patient age, and several identifiers that uniquely identify the device on which the test was performed (SofiaSerNum, the individual test (FluTestNum), and the result (ResultID). Multiple tests are stored on each device. The present Quidel COVID Test sensor concerns the positive rate in the test result.
+
+### Signal names
+- covid_ag_raw_pct_positive: percent of tests returning positive that day
+- covid_ag_smoothed_pct_positive: same as above, but for the moving average of the most recent 7 days
+- flu_ag_raw_pct_positive: percent of tests returning positive that day
+- flu_ag_smoothed_pct_positive : same as above, but for the moving average of the most recent 7 days
+- flu_ag_raw_tests_per_device: average number of tests per active testing device (= device that performed at least one test this period) that day
+- flu_ag_smoothed_tests_per_device: same as above, but for the moving average of the most recent 7 days
+
+### Estimating percent positive test proportion
+Let n be the number of total tests taken over a given time period and a given location (the test result can be negative/positive/invalid). Let x be the number of tests taken with positive results in this location over the given time period. We are interested in estimating the percentage of positive tests which is defined as:
+```
+p = 100 * x / n 
+```
+We estimate p across 3 temporal-spatial aggregation schemes:
+- daily, at the MSA (metropolitan statistical area) level;
+- daily, at the HRR (hospital referral region) level;
+- daily, at the state level.
+We are able to make these aggregations accurately because each test is reported with its 5-digit ZIP code. We do not report estimates for individual counties, as typically each county has too few tests to make the estimated value statistically meaningful.
+
+**MSA and HRR levels**: In a given MSA or HRR, suppose N flu tests are taken in a certain time period, X is the number of tests taken with positive results. 
+If N >= 50, we simply use:
+```
+p = 100 * X / N 
+```
+If 30 <= N < 50, we lend 50 - N  fake samples from its home state to shrink the estimate to the state's mean, which means:
+```
+p = 100 * [ N /50 * X/N + (50 - N)/50  * Xs /Ns ] 
+```
+where Ns, Xs are the number of flu tests and the number of flu tests taken with positive results taken in its home state in the same time period.
+If N < 30, nothing will be reported.
+
+**State level**:  the states with sample sizes smaller than a certain threshold are discarded. (The threshold is set to be 50 temporarily). For the rest of the states with big enough sample sizes,
+```
+p = 100 * X / N
+```
+
+The estimated standard error is simply:
+```
+se = 100 * sqrt{ p/100 *(1-p/100)/N } 
+```
+where we assume for each time point, the estimates follow a binomial distribution.
+
+
+### Temporal and Spatial Pooling
+We conduct temporal and spatial pooling for the smoothed signal. The spatial pooling is described in the previous section where we shrink the estimates to the state's mean if the total test number is smaller than 50 for a certain location on a certain day. Additionally, as with the Quidel COVID Test signal, we consider smoothed estimates formed by pooling data over time. That is, daily, for each location, we first pool all data available in that location over the last 7 days, and we then recompute everything described in the last two subsections. Pooling in this data makes estimates available in more geographic areas.
+
+### Exceptions
+#### Quidel COVID Test
+There are 9 special zip codes that are included in Quidel COVID raw data but are not included in our reports temporarily since we do not have enough mapping information for them. 
+
+|zip       |State| Number of Tests|
+|---|-------|------|
+|78086    |TX|98|
+|20174    | VA|17|
+|48824    |MI|14|
+|32313     |FL|37|
+|29486    |SC|69|
+|75033    |TX|2318|
+|79430    |TX|43|
+|44325 |OH|56|
+|75072    |TX|63|
+
+* Number of tests calculated until 08-05-2020
+* Until 08-05-2020, only 2,715 tests out of 942,293 tests for those zip codes. 
+
+#### Quidel Flu Test
+There are 89 special zip codes that are included in Quidel Flu raw data but are not included in our reports temporarily since we do not have enough mapping information for them. 
+* Until 08-05-2020, 133,000 tests out of 7,519,726 tests for those zip codes. 

quidel/README.md

@@ -0,0 +1,70 @@
+# Quidel Test Indicators
+
+## Running the Indicator
+
+The indicator is run by directly executing the Python module contained in this
+directory. The safest way to do this is to create a virtual environment,
+installed the common DELPHI tools, and then install the module and its
+dependencies. To do this, run the following code from this directory:
+
+```
+python -m venv env
+source env/bin/activate
+pip install ../_delphi_utils_python/.
+pip install .
+```
+
+All of the user-changable parameters are stored in `params.json`. A template is
+included as `params.json.template`. At a minimum, you will need to include a
+password for the datadrop email account and the email address of the data sender. 
+Note that setting `export_end_date` to an empty string will export data through 
+today (GMT) minus 5 days for COVID Antigen Tests. (It has not been settled for 
+Flu Antigen Tests) Setting `pull_end_date` to an empty string will pull data 
+through today (GMT).
+
+Quidel COVID test datasets are received by email from the first available date.
+However, the earliest part of Quidel Flu test datasets are stored in MIDAS. The 
+default of `pull_start_date` for Quidel Flu test is set to be `2020-05-08`, which 
+is the first valid date to pull the data from email. When officially running 
+this pipeline to get all the historical data for Quidel Flu test, this pipeline 
+needs to be run on MIDAS whith `pull_start_date` for Quidel Flu test set to be 
+an arbitrary date earlier than `2020-05-08`.
+
+To execute the module and produce the output datasets (by default, in
+`receiving`), run the following:
+
+```
+env/bin/python -m delphi_quidel
+```
+
+Once you are finished with the code, you can deactivate the virtual environment
+and (optionally) remove the environment itself.
+
+```
+deactivate
+rm -r env
+```
+
+## Testing the code
+
+To do a static test of the code style, it is recommended to run **pylint** on
+the module. To do this, run the following from the main module directory:
+
+```
+env/bin/pylint delphi_quidel
+```
+
+The most aggressive checks are turned off; only relatively important issues
+should be raised and they should be manually checked (or better, fixed).
+
+Unit tests are also included in the module. To execute these, run the following
+command from this directory:
+
+```
+(cd tests && ../env/bin/pytest --cov=delphi_quidel --cov-report=term-missing)
+```
+
+The output will show the number of unit tests that passed and failed, along
+with the percentage of code covered by the tests. None of the tests should
+fail and the code lines that are not covered by unit tests should be small and
+should not include critical sub-routines.

quidel/REVIEW.md

@@ -0,0 +1,39 @@
+## Code Review (Python)
+
+A code review of this module should include a careful look at the code and the
+output. To assist in the process, but certainly not in replace of it, please
+check the following items.
+
+**Documentation**
+
+- [ ] the README.md file template is filled out and currently accurate; it is
+possible to load and test the code using only the instructions given
+- [ ] minimal docstrings (one line describing what the function does) are
+included for all functions; full docstrings describing the inputs and expected
+outputs should be given for non-trivial functions
+
+**Structure**
+
+- [ ] code should use 4 spaces for indentation; other style decisions are
+flexible, but be consistent within a module
+- [ ] any required metadata files are checked into the repository and placed
+within the directory `static`
+- [ ] any intermediate files that are created and stored by the module should
+be placed in the directory `cache`
+- [ ] final expected output files to be uploaded to the API are placed in the
+`receiving` directory; output files should not be committed to the respository
+- [ ] all options and API keys are passed through the file `params.json`
+- [ ] template parameter file (`params.json.template`) is checked into the
+code; no personal (i.e., usernames) or private (i.e., API keys) information is
+included in this template file
+
+**Testing**
+
+- [ ] module can be installed in a new virtual environment
+- [ ] pylint with the default `.pylint` settings run over the module produces
+minimal warnings; warnings that do exist have been confirmed as false positives
+- [ ] reasonably high level of unit test coverage covering all of the main logic
+of the code (e.g., missing coverage for raised errors that do not currently seem
+possible to reach are okay; missing coverage for options that will be needed are
+not)
+- [ ] all unit tests run without errors

quidel/cache/.gitignore

@@ -0,0 +1 @@
+*.csv

quidel/delphi_quidel/__init__.py

@@ -0,0 +1,16 @@
+# -*- coding: utf-8 -*-
+"""Module to pull and clean indicators from the Quidel COVID Test.
+
+This file defines the functions that are made public by the module. As the
+module is intended to be executed though the main method, these are primarily
+for testing.
+"""
+
+from __future__ import absolute_import
+
+from . import geo_maps
+from . import data_tools
+from . import generate_sensor
+from . import export
+from . import pull
+from . import run

quidel/delphi_quidel/__main__.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+"""Call the function run_module when executed.
+
+This file indicates that calling the module (`python -m MODULE_NAME`) will
+call the function `run_module` found within the run.py file. There should be
+no need to change this template.
+"""
+
+from .run import run_module  # pragma: no cover
+
+run_module()  # pragma: no cover

quidel/delphi_quidel/constants.py

@@ -0,0 +1,23 @@
+"""Registry for constants"""
+# global constants
+MIN_OBS = 50  # minimum number of observations in order to compute a proportion.
+MAX_BORROW_OBS = 20 # maximum number of observations can be borrowed in geographical pooling
+POOL_DAYS = 7  # number of days in the past (including today) to pool over
+END_FROM_TODAY_MINUS = 5 # report data until - X days
+EXPORT_DAY_RANGE = 40 # Number of dates to report
+# Signal names
+SENSORS = {
+    "covid_ag_smoothed_pct_positive": (False, True),
+    "covid_ag_raw_pct_positive": (False, False),
+#    "covid_ag_smoothed_test_per_device": (True, True),
+#    "covid_ag_raw_test_per_device": (True, False),
+#    "flu_ag_smoothed_pct_positive": (False, True),
+#    "flu_ag_raw_pct_positive": (False, False),
+#    "flu_ag_smoothed_test_per_device": (True, True),
+#    "flu_ag_raw_test_per_device": (True, False)
+}
+GEO_RESOLUTIONS = [
+    "county",
+    "msa",
+    "hrr"
+]