Merge pull request #184 from cmu-delphi/refactor/combo_cases_and_deaths

Combo cases and deaths refactor

Author: krivard

combo_cases_and_deaths/README.md

@@ -0,0 +1,57 @@
+# MODULE NAME
+
+
+
+## Running the Indicator
+
+The indicator is run by directly executing the Python module contained in this
+directory. The safest way to do this is to create a virtual environment,
+installed the common DELPHI tools, and then install the module and its
+dependencies. To do this, run the following code from this directory:
+
+```
+python -m venv env
+source env/bin/activate
+pip install ../_delphi_utils_python/.
+pip install .
+```
+
+All of the user-changable parameters are stored in `params.json`. To execute
+the module and produce the output datasets (by default, in `receiving`), run
+the following:
+
+```
+env/bin/python -m delphi_combo_cases_and_deaths
+```
+
+Once you are finished with the code, you can deactivate the virtual environment
+and (optionally) remove the environment itself.
+
+```
+deactivate
+rm -r env
+```
+
+## Testing the code
+
+To do a static test of the code style, it is recommended to run **pylint** on
+the module. To do this, run the following from the main module directory:
+
+```
+env/bin/pylint delphi_combo_cases_and_deaths
+```
+
+The most aggressive checks are turned off; only relatively important issues
+should be raised and they should be manually checked (or better, fixed).
+
+Unit tests are also included in the module. To execute these, run the following
+command from this directory:
+
+```
+(cd tests && ../env/bin/pytest --cov=delphi_combo_cases_and_deaths --cov-report=term-missing)
+```
+
+The output will show the number of unit tests that passed and failed, along
+with the percentage of code covered by the tests. None of the tests should
+fail and the code lines that are not covered by unit tests should be small and
+should not include critical sub-routines.

combo_cases_and_deaths/REVIEW.md

@@ -0,0 +1,39 @@
+## Code Review (Python)
+
+A code review of this module should include a careful look at the code and the
+output. To assist in the process, but certainly not in replace of it, please
+check the following items.
+
+**Documentation**
+
+- [ ] the README.md file template is filled out and currently accurate; it is
+possible to load and test the code using only the instructions given
+- [ ] minimal docstrings (one line describing what the function does) are
+included for all functions; full docstrings describing the inputs and expected
+outputs should be given for non-trivial functions
+
+**Structure**
+
+- [ ] code should use 4 spaces for indentation; other style decisions are
+flexible, but be consistent within a module
+- [ ] any required metadata files are checked into the repository and placed
+within the directory `static`
+- [ ] any intermediate files that are created and stored by the module should
+be placed in the directory `cache`
+- [ ] final expected output files to be uploaded to the API are placed in the
+`receiving` directory; output files should not be committed to the respository
+- [ ] all options and API keys are passed through the file `params.json`
+- [ ] template parameter file (`params.json.template`) is checked into the
+code; no personal (i.e., usernames) or private (i.e., API keys) information is
+included in this template file
+
+**Testing**
+
+- [ ] module can be installed in a new virtual environment
+- [ ] pylint with the default `.pylint` settings run over the module produces
+minimal warnings; warnings that do exist have been confirmed as false positives
+- [ ] reasonably high level of unit test coverage covering all of the main logic
+of the code (e.g., missing coverage for raised errors that do not currently seem
+possible to reach are okay; missing coverage for options that will be needed are
+not)
+- [ ] all unit tests run without errors

combo_cases_and_deaths/cache/.gitignore

@@ -0,0 +1,13 @@
+# -*- coding: utf-8 -*-
+"""Module to combine the JHU and USA Facts indicators.
+
+This file defines the functions that are made public by the module. As the
+module is intended to be executed though the main method, these are primarily
+for testing.
+"""
+
+from __future__ import absolute_import
+
+from . import run
+
+__version__ = "0.1.0"

deploy_cases_and_deaths/README.md renamed to combo_cases_and_deaths/delphi_combo_cases_and_deaths/README.md

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+"""Call the function run_module when executed.
+
+This file indicates that calling the module (`python -m delphi_combo_cases_and_deaths`) will
+call the function `run_module` found within the run.py file. There should be
+no need to change this template.
+"""
+
+from .run import run_module  # pragma: no cover
+
+run_module()  # pragma: no cover

combo_cases_and_deaths/delphi_combo_cases_and_deaths/__init__.py

@@ -0,0 +1,165 @@
+# -*- coding: utf-8 -*-
+"""Functions to call when running the function.
+
+This module should contain a function called `run_module`, that is executed when
+the module is run with `python -m delphi_combo_cases_and_deaths`.
+
+This module produces a combined signal for jhu-csse and usa-facts.  This signal
+is only used for visualization.  It sources Puerto Rico from jhu-csse and
+everything else from usa-facts.
+
+"""
+from datetime import date, timedelta, datetime
+from itertools import product
+import re
+import sys
+
+import covidcast
+import pandas as pd
+
+from delphi_utils import read_params, create_export_csv
+
+
+METRICS = [
+    "confirmed",
+    "deaths",
+]
+SMOOTH_TYPES = [
+    "",
+    "7dav",
+]
+SENSORS = [
+    "incidence_num",
+    "cumulative_num",
+    "incidence_prop",
+    "cumulative_prop",
+]
+GEO_RESOLUTIONS = [
+    "county",
+    "state",
+    "msa",
+    "hrr",
+]
+
+def check_not_none(data_frame, label, date_range):
+    """Exit gracefully if a data frame we attempted to retrieve is empty"""
+    if data_frame is None:
+        print(f"{label} not available in range {date_range}")
+        sys.exit(1)
+
+def combine_usafacts_and_jhu(signal, geo, date_range):
+    """
+    Add rows for PR from JHU signals to USA-FACTS signals
+    """
+    usafacts_df = covidcast.signal("usa-facts", signal, date_range[0], date_range[1], geo)
+    jhu_df = covidcast.signal("jhu-csse", signal, date_range[0], date_range[1], geo)
+    check_not_none(usafacts_df, "USA-FACTS", date_range)
+    check_not_none(jhu_df, "JHU", date_range)
+
+    # State level
+    if geo == 'state':
+        combined_df = usafacts_df.append(jhu_df[jhu_df["geo_value"] == 'pr'])
+    # County level
+    elif geo == 'county':
+        combined_df = usafacts_df.append(jhu_df[jhu_df["geo_value"] == '72000'])
+    # For MSA and HRR level, they are the same
+    else:
+        combined_df = usafacts_df
+
+    combined_df = combined_df.drop(["direction"], axis=1)
+    combined_df = combined_df.rename({"time_value": "timestamp",
+                                      "geo_value": "geo_id",
+                                      "value": "val",
+                                      "stderr": "se"},
+                                     axis=1)
+    return combined_df
+
+def extend_raw_date_range(params, sensor_name):
+    """A complete issue includes smoothed signals as well as all raw data
+    that contributed to the smoothed values, so that it's possible to use
+    the raw values in the API to reconstruct the smoothed signal at will.
+
+    The smoother we're currently using incorporates the previous 7
+    days of data, so we must extend the date range of the raw data
+    backwards by 7 days.
+    """
+    if sensor_name.find("7dav") < 0:
+        return [
+            params['date_range'][0] - timedelta(days=7),
+            params['date_range'][-1]
+            ]
+    return params['date_range']
+
+def next_missing_day(source, signals):
+    """Fetch the first day for which we want to generate new data."""
+    meta_df = covidcast.metadata()
+    meta_df = meta_df[meta_df["data_source"] == source]
+    meta_df = meta_df[meta_df["signal"].isin(signals)]
+    # min: use the max_time of the most lagged signal, in case they differ
+    # +timedelta: the subsequent day is the first day of new data to generate
+    day = min(meta_df["max_time"]) + timedelta(days=1)
+    return day
+
+def sensor_signal(metric, sensor, smoother):
+    """Generate the signal name for a particular configuration"""
+    if smoother == "7dav":
+        sensor_name = "_".join([smoother, sensor])
+    else:
+        sensor_name = sensor
+    signal = "_".join([metric, sensor_name])
+    return sensor_name, signal
+
+def run_module():
+    """Produce a combined cases and deaths signal using data from JHU and USA Facts"""
+    variants = [tuple((metric, geo_res)+sensor_signal(metric, sensor, smoother))
+                for (metric, geo_res, sensor, smoother) in
+                product(METRICS, GEO_RESOLUTIONS, SENSORS, SMOOTH_TYPES)]
+
+    params = read_params()
+    params['export_start_date'] = date(*params['export_start_date'])
+    yesterday = date.today() - timedelta(days=1)
+    if params['date_range'] == 'new':
+        # only create combined file for the newest update
+        # (usually for yesterday, but check just in case)
+        params['date_range'] = [
+            min(
+                yesterday,
+                next_missing_day(
+                    params["source"],
+                    set(signal[-1] for signal in variants)
+                )
+            ),
+            yesterday
+        ]
+    elif params['date_range'] == 'all':
+        # create combined files for all of the historical reports
+        params['date_range'] = [params['export_start_date'], yesterday]
+    else:
+        pattern = re.compile(r'^\d{8}-\d{8}$')
+        match_res = re.findall(pattern, params['date_range'])
+        if len(match_res) == 0:
+            raise ValueError(
+                "Invalid date_range parameter. Please choose from (new, all, yyyymmdd-yyyymmdd).")
+        try:
+            date1 = datetime.strptime(params['date_range'][:8], '%Y%m%d').date()
+        except ValueError:
+            raise ValueError("Invalid date_range parameter. Please check the first date.")
+        try:
+            date2 = datetime.strptime(params['date_range'][-8:], '%Y%m%d').date()
+        except ValueError:
+            raise ValueError("Invalid date_range parameter. Please check the second date.")
+
+        #The the valid start date
+        if date1 < params['export_start_date']:
+            date1 = params['export_start_date']
+        params['date_range'] = [date1, date2]
+
+    for metric, geo_res, sensor_name, signal in variants:
+        create_export_csv(
+            combine_usafacts_and_jhu(signal, geo_res, extend_raw_date_range(params, sensor_name)),
+            export_dir=params['export_dir'],
+            start_date=pd.to_datetime(params['export_start_date']),
+            metric=metric,
+            geo_res=geo_res,
+            sensor=sensor_name,
+        )

combo_cases_and_deaths/delphi_combo_cases_and_deaths/__main__.py

@@ -0,0 +1,8 @@
+{
+  "static_file_dir": "./static",
+  "export_dir": "./receiving",
+  "cache_dir": "./cache",
+  "export_start_date":[2020,4,1],
+  "date_range":"new",
+  "source":"indicator-combination"
+}

combo_cases_and_deaths/delphi_combo_cases_and_deaths/run.py

@@ -0,0 +1 @@
+*.csv

combo_cases_and_deaths/params.json.template

@@ -0,0 +1,27 @@
+from setuptools import setup
+from setuptools import find_packages
+
+required = [
+    "pandas",
+    "pytest",
+    "pytest-cov",
+    "pylint",
+    "delphi-utils",
+    "covidcast"
+]
+
+setup(
+    name="delphi_combo_cases_and_deaths",
+    version="0.1.0",
+    description="A combined signal for cases and deaths using JHU for Puerto Rico and USA Facts everywhere else",
+    author="Jingjing Tang, Kathryn Mazaitis",
+    author_email="[email protected]",
+    url="https://github.com/cmu-delphi/covidcast-indicators",
+    install_requires=required,
+    classifiers=[
+        "Development Status :: 5 - Production/Stable",
+        "Intended Audience :: Developers",
+        "Programming Language :: Python :: 3.7",
+    ],
+    packages=find_packages(),
+)

combo_cases_and_deaths/receiving/.gitignore

@@ -0,0 +1,5 @@
+{
+  "static_file_dir": "../static",
+  "export_dir": "./receiving",
+  "cache_dir": "./cache"
+}

combo_cases_and_deaths/setup.py

@@ -0,0 +1,23 @@
+from datetime import date
+from itertools import product
+import pytest
+
+from delphi_combo_cases_and_deaths.run import extend_raw_date_range,sensor_signal,METRICS,SENSORS,SMOOTH_TYPES
+
+def test_issue_dates():
+    reference_dr = [date.today(),date.today()]
+    params = {'date_range': reference_dr}
+    n_changed = 0
+    variants = [sensor_signal(metric, sensor, smoother) for
+                metric, sensor, smoother in
+                product(METRICS,SENSORS,SMOOTH_TYPES)]
+    variants_changed = []
+    for sensor_name,signal in variants:
+        dr = extend_raw_date_range(params, sensor_name)
+        if dr[0] != reference_dr[0]:
+            n_changed += 1
+            variants_changed.append(sensor_name)
+    assert n_changed == len(variants) / 2, f"""Raw variants should post more days than smoothed.
+All variants: {variants}
+Date-extended variants: {variants_changed}
+"""