implement claims-only hospitalization indicator

claims_hosp/README.md

@@ -0,0 +1,72 @@
+# Claims Hospitalizations Indicator
+
+COVID-19 indicator using hospitalizations from claims records.
+Makes appropriate date shifts, pads for small counts, and smooths estimates.
+Results are written to CSV.
+
+## Running the Indicator
+
+The indicator is run by directly executing the Python module contained in this
+directory. The safest way to do this is to create a virtual environment,
+installed the common DELPHI tools, and then install the module and its
+dependencies. To do this, run the following code from this directory:
+
+```
+python -m venv env
+source env/bin/activate
+pip install ../_delphi_utils_python/.
+pip install .
+```
+
+All of the user-changable parameters are stored in `params.json`. To execute
+the module and produce the output datasets (by default, in `receiving`), run
+the following:
+
+```
+env/bin/python -m delphi_claims_hosp
+```
+
+Once you are finished with the code, you can deactivate the virtual environment
+and (optionally) remove the environment itself.
+
+```
+deactivate
+rm -r env
+```
+
+## Testing the code
+
+To do a static test of the code style, it is recommended to run **pylint** on
+the module. To do this, run the following from the main module directory:
+
+```
+env/bin/pylint delphi_claims_hosp
+```
+
+The most aggressive checks are turned off; only relatively important issues
+should be raised and they should be manually checked (or better, fixed).
+
+Unit tests are also included in the module. To execute these, run the following
+command from this directory:
+
+```
+(cd tests && ../env/bin/pytest --cov=delphi_claims_hosp --cov-report=term-missing)
+```
+
+The output will show the number of unit tests that passed and failed, along
+with the percentage of code covered by the tests. None of the tests should
+fail and the code lines that are not covered by unit tests should be small and
+should not include critical sub-routines.
+
+
+## Code tour
+
+- run.py: reads params.json to generated updated signal, run daily
+- update_indicator.py: 
+    ClaimsHospIndicatorUpdater: reads the data, makes transformations, writes output
+- indicator.py:
+    ClaimsHospIndicator: methods for transforming data, including padding and smoothing
+- load_data.py: methods for loading claims data
+- smooth.py: left smoother for signal
+- weekday.py: poisson model for handling day-of-week effects
+- config.py: configuration variables

claims_hosp/REVIEW.md

@@ -0,0 +1,82 @@
+## Code Review (Python)
+
+A code review of this module should include a careful look at the code and the
+output. To assist in the process, but certainly not in replace of it, please
+check the following items.
+
+### Internal check by author:
+
+**Documentation**
+
+- [x] the README.md file template is filled out and currently accurate; it is
+possible to load and test the code using only the instructions given
+- [x] minimal docstrings (one line describing what the function does) are
+included for all functions; full docstrings describing the inputs and expected
+outputs should be given for non-trivial functions
+
+**Structure**
+
+- [x] code should use 4 spaces for indentation; other style decisions are
+flexible, but be consistent within a module
+- [x] any required metadata files are checked into the repository and placed
+within the directory `static`
+- [x] any intermediate files that are created and stored by the module should
+be placed in the directory `cache`
+- [x] final expected output files to be uploaded to the API are placed in the
+`receiving` directory; output files should not be committed to the respository
+- [x] all options and API keys are passed through the file `params.json`
+- [x] template parameter file (`params.json.template`) is checked into the
+code; no personal (i.e., usernames) or private (i.e., API keys) information is
+included in this template file
+
+**Testing**
+
+- [x] module can be installed in a new virtual environment
+- [x] pylint with the default `.pylint` settings run over the module produces
+minimal warnings; warnings that do exist have been confirmed as false positives.
+_**(Small warnings appear, these have been checked as non-error messages. Code is rated 
+at 9.50/10.)**_
+- [x] reasonably high level of unit test coverage covering all of the main logic
+of the code (e.g., missing coverage for raised errors that do not currently seem
+possible to reach are okay; missing coverage for options that will be needed are
+not)
+_**(Code coverage is 72%. Little coverage for weekday model (developed externally), 
+and meta-script.)**_
+- [x] all unit tests run without errors
+
+
+### External check:
+
+**Documentation**
+
+- [ ] the README.md file template is filled out and currently accurate; it is
+possible to load and test the code using only the instructions given
+- [ ] minimal docstrings (one line describing what the function does) are
+included for all functions; full docstrings describing the inputs and expected
+outputs should be given for non-trivial functions
+
+**Structure**
+
+- [ ] code should use 4 spaces for indentation; other style decisions are
+flexible, but be consistent within a module
+- [ ] any required metadata files are checked into the repository and placed
+within the directory `static`
+- [ ] any intermediate files that are created and stored by the module should
+be placed in the directory `cache`
+- [ ] final expected output files to be uploaded to the API are placed in the
+`receiving` directory; output files should not be committed to the respository
+- [ ] all options and API keys are passed through the file `params.json`
+- [ ] template parameter file (`params.json.template`) is checked into the
+code; no personal (i.e., usernames) or private (i.e., API keys) information is
+included in this template file
+
+**Testing**
+
+- [ ] module can be installed in a new virtual environment
+- [ ] pylint with the default `.pylint` settings run over the module produces
+minimal warnings; warnings that do exist have been confirmed as false positives
+- [ ] reasonably high level of unit test coverage covering all of the main logic
+of the code (e.g., missing coverage for raised errors that do not currently seem
+possible to reach are okay; missing coverage for options that will be needed are
+not)
+- [ ] all unit tests run without errors

claims_hosp/delphi_claims_hosp/__init__.py

@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+"""Module to pull and clean indicators from the Claims Hospitalization source.
+
+This file defines the functions that are made public by the module. As the
+module is intended to be executed though the main method, these are primarily
+for testing.
+"""
+
+from __future__ import absolute_import
+
+from . import config
+from . import indicator
+from . import load_data
+from . import run
+from . import smooth
+from . import update_indicator
+from . import weekday
+
+__version__ = "0.1.0"

claims_hosp/delphi_claims_hosp/__main__.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+"""Call the function run_module when executed.
+
+This file indicates that calling the module (`python -m MODULE_NAME`) will
+call the function `run_module` found within the run.py file. There should be
+no need to change this template.
+"""
+
+from .run import run_module  # pragma: no cover
+
+run_module()  # pragma: no cover

claims_hosp/delphi_claims_hosp/config.py

@@ -0,0 +1,75 @@
+"""
+This file contains configuration variables used to generate the claims-based Hosp. signal.
+
+Author: Maria Jahja
+Created: 2020-06-01
+Modified: 2020-09-27
+
+"""
+
+from datetime import datetime, timedelta
+
+
+class Config:
+    """
+    Static configuration variables.
+    """
+
+    signal_name = "smoothed_covid19"
+    signal_weekday_name = "smoothed_adj_covid19"
+
+    # max number of CPUs available for pool
+    MAX_CPU_POOL = 10
+
+    # first date we consider in training data
+    FIRST_DATA_DATE = datetime(2020, 1, 1)
+
+    # number of days training needs to produce estimate
+    # (one day needed for smoother to produce values)
+    BURN_IN_PERIOD = timedelta(days=1)
+
+    # shift dates forward for labeling purposes
+    DAY_SHIFT = timedelta(days=1)
+
+    # data columns
+    CLAIMS_COUNT_COLS = ["Denominator", "Covid_like"]
+    CLAIMS_DATE_COL = "ServiceDate"
+    CLAIMS_RENAME_COLS = {"Pat HRR ID": "hrr", "ServiceDate": "date",
+                          "PatCountyFIPS": "fips", "PatAgeGroup": "age_group"}
+    CLAIMS_DTYPES = {
+        "ServiceDate": str,
+        "PatCountyFIPS": str,
+        "Denominator": float,
+        "Covid_like": float,
+        "PatAgeGroup": str,
+        "Pat HRR ID": str,
+    }
+
+    FIPS_COL = "fips"
+    DATE_COL = "date"
+    AGE_COL = "age_group"
+    HRR_COL = "hrr"
+
+    SMOOTHER_BANDWIDTH = 100  # bandwidth for the linear left Gaussian filter
+    MIN_DEN = 100  # number of total visits needed to produce a sensor
+    MAX_BACKWARDS_PAD_LENGTH = (
+        7  # maximum number of days used to average a backwards padding
+    )
+    MIN_CUM_VISITS = 500  # need to observe at least 500 counts before averaging
+
+
+class GeoConstants:
+    """
+    Constant geographical variables.
+    """
+
+    # number of counties in usa, including megacounties
+    NUM_COUNTIES = 3141 + 52
+    NUM_HRRS = 308
+    NUM_MSAS = 392 + 52  # MSA + States
+    NUM_STATES = 52  # including DC and PR
+
+    MAX_GEO = {"county": NUM_COUNTIES,
+               "hrr": NUM_HRRS,
+               "msa": NUM_MSAS,
+               "state": NUM_STATES}