Merge pull request #76 from cmu-delphi/dv-package

Doctor's visits package

Author: krivard

doctor_visits/.gitignore

@@ -0,0 +1,120 @@
+# You should hard commit a prototype for this file, but we
+# want to avoid accidental adding of API tokens and other
+# private data parameters
+params.json
+
+# Do not commit output files
+receiving/*.csv
+
+# Remove macOS files
+.DS_Store
+
+# virtual environment
+dview/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+coverage.xml
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+.static_storage/
+.media/
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/

doctor_visits/.pylintrc

@@ -0,0 +1,8 @@
+[DESIGN]
+
+min-public-methods=0
+
+
+[MESSAGES CONTROL]
+
+disable=R0801, C0200, C0330, E1101, E0611, E1136, C0114, C0116, C0103, R0913, R0914, R0915, W1401, W1202, W1203, W0702

doctor_visits/README.md

@@ -0,0 +1,55 @@
+# Doctor Visits Indicator
+
+## Running the Indicator
+
+The indicator is run by directly executing the Python module contained in this
+directory. The safest way to do this is to create a virtual environment,
+installed the common DELPHI tools, and then install the module and its
+dependencies. To do this, run the following code from this directory:
+
+```
+python -m venv env
+source env/bin/activate
+pip install ../_delphi_utils_python/.
+pip install .
+```
+
+All of the user-changable parameters are stored in `params.json`. To execute
+the module and produce the output datasets (by default, in `receiving`), run
+the following:
+
+```
+env/bin/python -m delphi_doctor_visits
+```
+
+Once you are finished with the code, you can deactivate the virtual environment
+and (optionally) remove the environment itself.
+
+```
+deactivate
+rm -r env
+```
+
+## Testing the code
+
+To do a static test of the code style, it is recommended to run **pylint** on
+the module. To do this, run the following from the main module directory:
+
+```
+env/bin/pylint delphi_doctor_visits
+```
+
+The most aggressive checks are turned off; only relatively important issues
+should be raised and they should be manually checked (or better, fixed).
+
+Unit tests are also included in the module. To execute these, run the following
+command from this directory:
+
+```
+(cd tests && ../env/bin/pytest --cov=delphi_doctor_visits --cov-report=term-missing)
+```
+
+The output will show the number of unit tests that passed and failed, along
+with the percentage of code covered by the tests. None of the tests should
+fail and the code lines that are not covered by unit tests should be small and
+should not include critical sub-routines.

doctor_visits/REVIEW.md

@@ -0,0 +1,39 @@
+## Code Review (Python)
+
+A code review of this module should include a careful look at the code and the
+output. To assist in the process, but certainly not in replace of it, please
+check the following items.
+
+**Documentation**
+
+- [ ] the README.md file template is filled out and currently accurate; it is
+possible to load and test the code using only the instructions given
+- [ ] minimal docstrings (one line describing what the function does) are
+included for all functions; full docstrings describing the inputs and expected
+outputs should be given for non-trivial functions
+
+**Structure**
+
+- [ ] code should use 4 spaces for indentation; other style decisions are
+flexible, but be consistent within a module
+- [ ] any required metadata files are checked into the repository and placed
+within the directory `static`
+- [ ] any intermediate files that are created and stored by the module should
+be placed in the directory `cache`
+- [ ] final expected output files to be uploaded to the API are placed in the
+`receiving` directory; output files should not be committed to the respository
+- [ ] all options and API keys are passed through the file `params.json`
+- [ ] template parameter file (`params.json.template`) is checked into the
+code; no personal (i.e., usernames) or private (i.e., API keys) information is
+included in this template file
+
+**Testing**
+
+- [ ] module can be installed in a new virtual environment
+- [ ] pylint with the default `.pylint` settings run over the module produces
+minimal warnings; warnings that do exist have been confirmed as false positives
+- [ ] reasonably high level of unit test coverage covering all of the main logic
+of the code (e.g., missing coverage for raised errors that do not currently seem
+possible to reach are okay; missing coverage for options that will be needed are
+not)
+- [ ] all unit tests run without errors

doctor_visits/cache/.gitignore

@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+"""Module to pull and clean indicators from the Doctor's Visits source.
+
+This file defines the functions that are made public by the module. As the
+module is intended to be executed though the main method, these are primarily
+for testing.
+"""
+
+from __future__ import absolute_import
+
+from . import config
+from . import direction
+from . import geo_maps
+from . import run
+from . import sensor
+from . import smooth
+from . import update_sensor
+from . import weekday
+
+__version__ = "0.1.0"

doctor_visits/delphi_doctor_visits/__init__.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+"""Call the function run_module when executed.
+
+This file indicates that calling the module (`python -m delphi_doctor_visits`) will
+call the function `run_module` found within the run.py file. There should be
+no need to change this template.
+"""
+
+from .run import run_module  # pragma: no cover
+
+run_module()  # pragma: no cover

doctor_visits/delphi_doctor_visits/__main__.py

@@ -0,0 +1,41 @@
+"""
+This file contains configuration variables used to generate the doctor visits signal.
+
+Author: Maria
+Created: 2020-04-16
+Last modified: 2020-06-17
+"""
+
+from datetime import datetime, timedelta
+
+
+class Config:
+    """Static configuration variables.
+    """
+
+    # dates
+    FIRST_DATA_DATE = datetime(2020, 1, 1)
+    DAY_SHIFT = timedelta(days=1)  # shift dates forward for labeling purposes
+
+    # data columns
+    CLI_COLS = ["Covid_like", "Flu_like", "Mixed"]
+    FLU1_COL = ["Flu1"]
+    COUNT_COLS = CLI_COLS + FLU1_COL + ["Denominator"]
+    DATE_COL = "ServiceDate"
+    GEO_COL = "PatCountyFIPS"
+    AGE_COL = "PatAgeGroup"
+    HRR_COLS = ["Pat HRR Name", "Pat HRR ID"]
+    ID_COLS = [DATE_COL] + [GEO_COL] + [AGE_COL] + HRR_COLS
+    FILT_COLS = ID_COLS + COUNT_COLS
+    DTYPES = {"ServiceDate": str, "PatCountyFIPS": str,
+              "Denominator": int, "Flu1": int,
+              "Covid_like": int, "Flu_like": int,
+              "Mixed": int, "PatAgeGroup": str,
+              "Pat HRR Name": str, "Pat HRR ID": float}
+
+    SMOOTHER_BANDWIDTH = 100  # bandwidth for the linear left Gaussian filter
+    MAX_BACKFILL_WINDOW = 7  # maximum number of days used to average a backfill correction
+    MIN_CUM_VISITS = 500  # need to observe at least 500 counts before averaging
+    RECENT_LENGTH = 7  # number of days to sum over for sparsity threshold
+    MIN_RECENT_VISITS = 100  # min numbers of visits needed to include estimate
+    MIN_RECENT_OBS = 3  # minimum days needed to produce an estimate for latest time

doctor_visits/delphi_doctor_visits/config.py

@@ -0,0 +1,53 @@
+"""
+Functions used to calculate direction. (Thanks to Addison Hu)
+
+Author: Maria Jahja
+Created: 2020-04-17
+
+"""
+
+import numpy as np
+
+
+def running_mean(s):
+    """Compute running mean."""
+    return np.cumsum(s) / np.arange(1, len(s) + 1)
+
+
+def running_sd(s, mu=None):
+    """
+    Compute running standard deviation.    Running mean can be pre-supplied
+    to save on computation.
+    """
+    if mu is None:
+        mu = running_mean(s)
+    sqmu = running_mean(s ** 2)
+    sd = np.sqrt(sqmu - mu ** 2)
+    return sd
+
+
+def first_difference_direction(s):
+    """
+    Code taken from Addison Hu. Modified to return directional strings.
+    Declares "notable" increases and decreases based on the distribution
+    of past first differences.
+
+    Args:
+        s: input data
+
+    Returns: Directions in "-1", "0", "+1", or "NA" for first 3 values
+    """
+    T = s[1:] - s[:-1]
+    mu = running_mean(T)
+    sd = running_sd(T, mu=mu)
+    d = np.full(s.shape, "NA")
+
+    for idx in range(2, len(T)):
+        if T[idx] < min(mu[idx - 1] - sd[idx - 1], 0):
+            d[idx + 1] = "-1"
+        elif T[idx] > max(mu[idx - 1] + sd[idx - 1], 0):
+            d[idx + 1] = "+1"
+        else:
+            d[idx + 1] = "0"
+
+    return d