Merge pull request #344 from cmu-delphi/run-google-symptoms

Pipeline for the new google symptoms indicator

Author: krivard

google_symptoms/.gitignore

@@ -0,0 +1,121 @@
+# You should hard commit a prototype for this file, but we
+# want to avoid accidental adding of API tokens and other
+# private data parameters
+params.json
+
+# Do not commit output files
+receiving/*.csv
+tests/receiving/*.csv
+
+# Remove macOS files
+.DS_Store
+
+# virtual environment
+dview/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+coverage.xml
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+.static_storage/
+.media/
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/

google_symptoms/.pylintrc

@@ -0,0 +1,8 @@
+[DESIGN]
+
+min-public-methods=1
+
+
+[MESSAGES CONTROL]
+
+disable=R0801, C0330, E1101, E0611, C0114, C0116, C0103, R0913, R0914, W0702, W0707

google_symptoms/DETAILS.md

@@ -0,0 +1,30 @@
+# Google Symptoms
+
+We import the normalized symptom search term popularity data from the Google 
+Research's Open COVID-19 Data project and export the county-level and state-level 
+data as-is.  We also aggregate the data to the MSA and HRR levels.
+
+## Geographical Levels (`geo`)
+* `county`: reported using zero-padded FIPS codes.  The county level data is derived 
+from `/subregions/state/2020_US_state_daily_symptoms_dataset.csv`.
+* `msa`: reported using cbsa (consistent with all other COVIDcast sensors). The msa
+level data is derived from county level data using population weighted average.
+* `hrr`: reported using HRR number (consistent with all other COVIDcast sensors). The 
+hrr level data is derived from county level data using population weighted average.
+* `state`: reported using two-letter postal code. The state level data is derived from
+`2020_US_daily_symptoms_dataset.csv` which includes data for District of Columbia.
+
+## Metrics, Level 1 (`m1`)
+* `Anosmia`: Google search volume for Anosmia-related searches
+* `Ageusia`: Google search volume for Ageusia-related searches
+
+## Metrics, Level 2 (`m2`)
+* `raw_search`:  Google search volume reported as-is
+* `smoothed_search`:  Google search volume using 7-day moving average
+
+This data reflects the volume of Google searches mapped to symptoms such Anosmia
+and Ageusia. The resulting daily dataset for each region showing the relative frequency
+of searches for each symptom.  This signal is measured in arbitrary units that are normalized
+for population and for the most popular symptom search term within a geographic region. Thus, 
+values are not comparable between geographic regions. Larger numbers represent higher 
+numbers of symptom-related searches.

google_symptoms/README.md

@@ -0,0 +1,61 @@
+# Google Symptoms
+
+We import the normalized symptom search term popularity data from the Google 
+Research's Open COVID-19 Data project and export the county-level and state-level 
+data as-is.  We also aggregate the data to the MSA and HRR levels. For detailed 
+information see the files `DETAILS.md` contained in this directory.
+
+## Running the Indicator
+
+The indicator is run by directly executing the Python module contained in this
+directory. The safest way to do this is to create a virtual environment,
+installed the common DELPHI tools, and then install the module and its
+dependencies. To do this, run the following code from this directory:
+
+```
+python -m venv env
+source env/bin/activate
+pip install ../_delphi_utils_python/.
+pip install .
+```
+
+All of the user-changable parameters are stored in `params.json`. To execute the module
+and produce the output datasets (by default, in `receiving`), run the following.
+
+```
+env/bin/python -m delphi_google_symptoms
+```
+
+Once you are finished with the code, you can deactivate the virtual environment
+and (optionally) remove the environment itself.
+
+```
+deactivate
+rm -r env
+```
+
+## Testing the code
+
+To do a static test of the code style, it is recommended to run **pylint** on
+the module. To do this, run the following from the main module directory:
+
+```
+env/bin/pylint delphi_google_symptoms
+```
+
+The most aggressive checks are turned off; only relatively important issues
+should be raised and they should be manually checked (or better, fixed).
+
+Unit tests are also included in the module. To execute these, run the following
+command from this directory:
+
+```
+(cd tests && ../env/bin/pytest --cov=delphi_google_symptoms --cov-report=term-missing)
+```
+
+The output will show the number of unit tests that passed and failed, along
+with the percentage of code covered by the tests. None of the tests should
+fail and the code lines that are not covered by unit tests should be small and
+should not include critical sub-routines.
+
+- Jenkins test #1

google_symptoms/REVIEW.md

@@ -0,0 +1,39 @@
+## Code Review (Python)
+
+A code review of this module should include a careful look at the code and the
+output. To assist in the process, but certainly not in replace of it, please
+check the following items.
+
+**Documentation**
+
+- [ ] the README.md file template is filled out and currently accurate; it is
+possible to load and test the code using only the instructions given
+- [ ] minimal docstrings (one line describing what the function does) are
+included for all functions; full docstrings describing the inputs and expected
+outputs should be given for non-trivial functions
+
+**Structure**
+
+- [ ] code should use 4 spaces for indentation; other style decisions are
+flexible, but be consistent within a module
+- [ ] any required metadata files are checked into the repository and placed
+within the directory `static`
+- [ ] any intermediate files that are created and stored by the module should
+be placed in the directory `cache`
+- [ ] final expected output files to be uploaded to the API are placed in the
+`receiving` directory; output files should not be committed to the respository
+- [ ] all options and API keys are passed through the file `params.json`
+- [ ] template parameter file (`params.json.template`) is checked into the
+code; no personal (i.e., usernames) or private (i.e., API keys) information is
+included in this template file
+
+**Testing**
+
+- [ ] module can be installed in a new virtual environment
+- [ ] pylint with the default `.pylint` settings run over the module produces
+minimal warnings; warnings that do exist have been confirmed as false positives
+- [ ] reasonably high level of unit test coverage covering all of the main logic
+of the code (e.g., missing coverage for raised errors that do not currently seem
+possible to reach are okay; missing coverage for options that will be needed are
+not)
+- [ ] all unit tests run without errors

google_symptoms/cache/.gitignore

@@ -0,0 +1,12 @@
+# -*- coding: utf-8 -*-
+"""Module to pull and clean indicators from the Google Research's Open
+   COVID-19 Data project.
+This file defines the functions that are made public by the module. As the
+module is intended to be executed though the main method, these are primarily
+for testing.
+"""
+
+from __future__ import absolute_import
+
+from . import pull
+from . import run

google_symptoms/delphi_google_symptoms/__init__.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+"""Call the function run_module when executed.
+
+This file indicates that calling the module (`python -m delphi_google_symptoms`) will
+call the function `run_module` found within the run.py file. There should be
+no need to change this template.
+"""
+
+from .run import run_module  # pragma: no cover
+
+run_module()  # pragma: no cover

google_symptoms/delphi_google_symptoms/__main__.py

@@ -0,0 +1,83 @@
+"""Registry for constants"""
+from functools import partial
+from datetime import timedelta
+
+from .smooth import (
+    identity,
+    kday_moving_average,
+)
+
+# global constants
+METRICS = ["Anosmia", "Ageusia"]
+SMOOTHERS = ["raw", "smoothed"]
+GEO_RESOLUTIONS = [
+        "state",
+        "county",
+        "msa",
+        "hrr"
+]
+
+seven_day_moving_average = partial(kday_moving_average, k=7)
+SMOOTHERS_MAP = {
+    "raw":           (identity, lambda d: d - timedelta(days=7)),
+    "smoothed":      (seven_day_moving_average, lambda d: d),
+}
+
+STATE_TO_ABBREV = {'Alabama':'al',
+                   'Alaska': 'ak',
+#                   'American Samoa': 'as',
+                   'Arizona': 'az',
+                   'Arkansas': 'ar',
+                   'California': 'ca',
+                   'Colorado': 'co',
+                   'Connecticut': 'ct',
+                   'Delaware': 'de',
+#                   'District of Columbia': 'dc',
+                   'Florida': 'fl',
+                   'Georgia': 'ga',
+#                   'Guam': 'gu',
+                   'Hawaii': 'hi',
+                   'Idaho': 'id',
+                   'Illinois': 'il',
+                   'Indiana': 'in',
+                   'Iowa': 'ia',
+                   'Kansas': 'ks',
+                   'Kentucky': 'ky',
+                   'Louisiana': 'la',
+                   'Maine': 'me',
+                   'Maryland': 'md',
+                   'Massachusetts': 'ma',
+                   'Michigan': 'mi',
+                   'Minnesota': 'mn',
+                   'Mississippi': 'ms',
+                   'Missouri': 'mo',
+                   'Montana': 'mt',
+                   'Nebraska': 'ne',
+                   'Nevada': 'nv',
+                   'New_Hampshire': 'nh',
+                   'New_Jersey': 'nj',
+                   'New_Mexico':'nm',
+                   'New_York': 'ny',
+                   'North_Carolina': 'nc',
+                   'North_Dakota': 'nd',
+#                   'Northern Mariana Islands': 'mp',
+                   'Ohio': 'oh',
+                   'Oklahoma': 'ok',
+                   'Oregon': 'or',
+                   'Pennsylvania': 'pa',
+#                   'Puerto Rico': 'pr',
+                   'Rhode_Island': 'ri',
+                   'South_Carolina': 'sc',
+                   'South_Dakota': 'sd',
+                   'Tennessee': 'tn',
+                   'Texas': 'tx',
+                   'Utah': 'ut',
+                   'Vermont': 'vt',
+#                   'Virgin Islands': 'vi',
+                   'Virginia': 'va',
+                   'Washington': 'wa',
+                   'West_Virginia': 'wv',
+                   'Wisconsin': 'wi',
+                   'Wyoming': 'wy'}
+
+DC_FIPS = "11001"