cmu-delphi · korlaxxalrok · Jun 24, 2020 · Jun 2, 2020 · Jun 8, 2020 · Jun 8, 2020
diff --git a/.gitignore b/.gitignore
@@ -118,3 +118,8 @@ venv.bak/
 
 # mypy
 .mypy_cache/
+
+# Ansible
+.retry
+.indicators-ansible-vault-pass
+indicators-ansible-vault-pass
diff --git a/README.md b/README.md
@@ -0,0 +1,83 @@
+# Covidcast Indicators
+
+Pipeline code and supporting libraries for the **Real-time COVID-19 Indicators** used in the Delphi Group's [**COVIDcast** map](https://covidcast.cmu.edu).
+
+## The indicators
+
+Each subdirectory contained here that is named after an indicator has specific documentation. Please review as necessary!
+
+## General workflow for indicators creation and deployment
+
+**tl;dr**
+
+- Create your new indicator branch from `main`.
+- Build it using the appropriate template, following the guidelines in the included README.md and REVIEW.md files.
+- Make some stuff!
+- When your stuff works, push your `dev-*` branch to remote for review.
+- Consult with a platform engineer for the remaining production setup needs. They will create a branch called `deploy-*` for your indicator.
+- Initiate a pull request against this new branch.
+- If your peers like it and Jenkins approves, deploy your changes by merging the PR.
+- Rejoice!
+
+### Starting out
+
+The `main` branch should contain up-to-date code and supporting libraries. This should be your starting point when creating a new indicator.
+
+```shell
+# Hint
+#
+git checkout main
+git checkout -b dev-my-feature-branch
+```
+
+### Creating your indicator
+
+Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach.
+
+- Consult your peers with questions! :handshake:
+
+Once you have something that runs locally and passes tests you set up your remote branch eventual review and production deployment.
+
+```shell
+# Hint
+#
+git push -u origin dev-my-feature-branch
+```
+
+### Setting up for review and deployment
+
+Once you have your branch set up you should get in touch with a platform engineer to pair up on the remaining production needs. These include:
+
+- Creating the corresponding `deploy-*` branch in the repo.
+- Adding the necessary Jenkins scripts for your indicator.
+- Preparing the runtime host with any Automation configuration necessities.
+- Reviewing the workflow to make sure it meets the general guidelines and will run as expected on the runtime host.
+
+Once all the last mile configuration is in place you can create a pull request against the correct `deploy-*` branch to initiate the CI/CD pipeline which will build, test, and package your indicator for deployment.
+
+If everything looks ok, platform engineering has validated the last mile, and the pull request is accepted, you can merge the PR. Deployment will start automatically.
+
+Hopefully it'll be a full on :tada:, after that :crossed_fingers:
+
+If not, circle back and try again.
+
+## Production overview
+
+### Running production code
+
+Currently, the production indicators all live and run on the venerable and perennially useful Delphi primary server (also known generically as "the runtime host").
+
+- This is a virtual machine running RHEL 7.5 and living in CMU's Campus Cloud vSphere-based infrastructure environemnt.
+
+### Delivering an indicator to the production environment
+
+We use a branch-based git workflow coupled with [Jenkins](https://www.jenkins.io/) and [Ansible](https://www.ansible.com/) to build, test, package, and deploy each indicator individually to the runtime host.
+
+- Jenkins dutifully manages the whole process for us by executing several "stages" in the context of a [CI/CD pipeline](https://dzone.com/articles/learn-how-to-setup-a-cicd-pipeline-from-scratch). Each stage does something unique, building on the previous stage. The stages are:
+  - Environment - Sets up some environment-specific needs that the other stages depend on.
+  - Build - Create the Python venv on the Jenkins host.
+  - Test - Run linting and unit tests.
+  - Package - Tar and gzip the built environment.
+  - Deploy - Trigger an Ansible playbook to place the built package onto the runtime host, place any necessary production configuration, and adjust the runtime envirnemnt (if necessary).
+
+There are several additional Jenkins-specific files that will need to be created for each indicator, as well as some configuration additions to the runtime host. It will be important to pair with a platform engineer to prepare the necessary production environment needs, test the workflow, validate on production, and ultimately sign off on a production release.
diff --git a/google_health/cache/Data_500_anosmia_ms.csv b/google_health/cache/Data_500_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 500,2020-05-13,0.00000000
 500,2020-05-14,0.00000000
 500,2020-05-15,0.00000000
+500,2020-05-16,0.00000000
+500,2020-05-17,0.00000000
+500,2020-05-18,0.00000000
+500,2020-05-19,0.00000000
+500,2020-05-20,0.00000000
+500,2020-05-21,0.00000000
+500,2020-05-22,0.00000000
+500,2020-05-23,0.00000000
+500,2020-05-24,2200.70422535
+500,2020-05-25,1973.16495659
+500,2020-05-26,0.00000000
+500,2020-05-27,0.00000000
+500,2020-05-28,0.00000000
+500,2020-05-29,0.00000000
+500,2020-05-30,0.00000000
+500,2020-05-31,0.00000000
+500,2020-06-01,0.00000000
+500,2020-06-02,0.00000000
+500,2020-06-03,0.00000000
+500,2020-06-04,0.00000000
+500,2020-06-05,0.00000000
diff --git a/google_health/cache/Data_501_anosmia_ms.csv b/google_health/cache/Data_501_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 501,2020-05-13,392.32625361
 501,2020-05-14,324.14725935
 501,2020-05-15,247.25216534
+501,2020-05-16,282.22614066
+501,2020-05-17,289.84124969
+501,2020-05-18,390.53549826
+501,2020-05-19,540.29622067
+501,2020-05-20,241.80713566
+501,2020-05-21,376.88500026
+501,2020-05-22,356.14953551
+501,2020-05-23,320.52164769
+501,2020-05-24,256.10714479
+501,2020-05-25,328.92618894
+501,2020-05-26,142.09376105
+501,2020-05-27,285.08656631
+501,2020-05-28,145.95659513
+501,2020-05-29,203.30545511
+501,2020-05-30,249.98138165
+501,2020-05-31,142.33883376
+501,2020-06-01,134.02327689
+501,2020-06-02,328.02073255
+501,2020-06-03,268.44266113
+501,2020-06-04,403.02506167
+501,2020-06-05,203.98113312
diff --git a/google_health/cache/Data_502_anosmia_ms.csv b/google_health/cache/Data_502_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 502,2020-05-13,0.00000000
 502,2020-05-14,0.00000000
 502,2020-05-15,0.00000000
+502,2020-05-16,0.00000000
+502,2020-05-17,0.00000000
+502,2020-05-18,0.00000000
+502,2020-05-19,0.00000000
+502,2020-05-20,0.00000000
+502,2020-05-21,0.00000000
+502,2020-05-22,0.00000000
+502,2020-05-23,0.00000000
+502,2020-05-24,0.00000000
+502,2020-05-25,0.00000000
+502,2020-05-26,0.00000000
+502,2020-05-27,0.00000000
+502,2020-05-28,0.00000000
+502,2020-05-29,0.00000000
+502,2020-05-30,0.00000000
+502,2020-05-31,0.00000000
+502,2020-06-01,0.00000000
+502,2020-06-02,0.00000000
+502,2020-06-03,0.00000000
+502,2020-06-04,0.00000000
+502,2020-06-05,0.00000000
diff --git a/google_health/cache/Data_503_anosmia_ms.csv b/google_health/cache/Data_503_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 503,2020-05-13,0.00000000
 503,2020-05-14,0.00000000
 503,2020-05-15,0.00000000
+503,2020-05-16,0.00000000
+503,2020-05-17,0.00000000
+503,2020-05-18,0.00000000
+503,2020-05-19,0.00000000
+503,2020-05-20,3672.42012486
+503,2020-05-21,0.00000000
+503,2020-05-22,0.00000000
+503,2020-05-23,0.00000000
+503,2020-05-24,0.00000000
+503,2020-05-25,0.00000000
+503,2020-05-26,0.00000000
+503,2020-05-27,0.00000000
+503,2020-05-28,0.00000000
+503,2020-05-29,0.00000000
+503,2020-05-30,0.00000000
+503,2020-05-31,0.00000000
+503,2020-06-01,0.00000000
+503,2020-06-02,0.00000000
+503,2020-06-03,0.00000000
+503,2020-06-04,0.00000000
+503,2020-06-05,0.00000000
diff --git a/google_health/cache/Data_504_anosmia_ms.csv b/google_health/cache/Data_504_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 504,2020-05-13,557.11659680
 504,2020-05-14,363.57013223
 504,2020-05-15,847.97233675
+504,2020-05-16,444.98346642
+504,2020-05-17,534.41672775
+504,2020-05-18,545.05004653
+504,2020-05-19,183.43119011
+504,2020-05-20,554.50828444
+504,2020-05-21,191.32736853
+504,2020-05-22,184.24842629
+504,2020-05-23,622.69156751
+504,2020-05-24,1108.10718092
+504,2020-05-25,663.20789572
+504,2020-05-26,772.27266461
+504,2020-05-27,190.29199506
+504,2020-05-28,363.83261971
+504,2020-05-29,0.00000000
+504,2020-05-30,0.00000000
+504,2020-05-31,219.38533138
+504,2020-06-01,921.24370291
+504,2020-06-02,559.06455322
+504,2020-06-03,178.48945958
+504,2020-06-04,568.41047377
+504,2020-06-05,184.42864118
diff --git a/google_health/cache/Data_505_anosmia_ms.csv b/google_health/cache/Data_505_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 505,2020-05-13,1033.24114744
 505,2020-05-14,966.23647884
 505,2020-05-15,671.32931951
+505,2020-05-16,0.00000000
+505,2020-05-17,0.00000000
+505,2020-05-18,1081.27786689
+505,2020-05-19,336.54208784
+505,2020-05-20,1359.58768990
+505,2020-05-21,713.28805930
+505,2020-05-22,0.00000000
+505,2020-05-23,1253.98677913
+505,2020-05-24,437.29613611
+505,2020-05-25,0.00000000
+505,2020-05-26,0.00000000
+505,2020-05-27,330.85299711
+505,2020-05-28,0.00000000
+505,2020-05-29,335.79555758
+505,2020-05-30,0.00000000
+505,2020-05-31,401.38591473
+505,2020-06-01,674.71132597
+505,2020-06-02,0.00000000
+505,2020-06-03,320.48723068
+505,2020-06-04,0.00000000
+505,2020-06-05,326.18573894
diff --git a/google_health/cache/Data_506_anosmia_ms.csv b/google_health/cache/Data_506_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 506,2020-05-13,224.16752446
 506,2020-05-14,690.35335010
 506,2020-05-15,677.35234002
+506,2020-05-16,0.00000000
+506,2020-05-17,537.89871869
+506,2020-05-18,502.89870470
+506,2020-05-19,1105.62934682
+506,2020-05-20,230.84100350
+506,2020-05-21,942.86292166
+506,2020-05-22,490.42349066
+506,2020-05-23,269.12959025
+506,2020-05-24,564.57111674
+506,2020-05-25,1049.97217083
+506,2020-05-26,0.00000000
+506,2020-05-27,683.47100277
+506,2020-05-28,677.34375290
+506,2020-05-29,0.00000000
+506,2020-05-30,0.00000000
+506,2020-05-31,272.29992876
+506,2020-06-01,675.67841894
+506,2020-06-02,455.37555267
+506,2020-06-03,429.08720078
+506,2020-06-04,769.12885310
+506,2020-06-05,668.76250905
diff --git a/google_health/cache/Data_507_anosmia_ms.csv b/google_health/cache/Data_507_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 507,2020-05-13,0.00000000
 507,2020-05-14,0.00000000
 507,2020-05-15,0.00000000
+507,2020-05-16,2329.37339856
+507,2020-05-17,0.00000000
+507,2020-05-18,2010.05025126
+507,2020-05-19,0.00000000
+507,2020-05-20,0.00000000
+507,2020-05-21,0.00000000
+507,2020-05-22,2102.16523019
+507,2020-05-23,0.00000000
+507,2020-05-24,0.00000000
+507,2020-05-25,2227.66763199
+507,2020-05-26,0.00000000
+507,2020-05-27,0.00000000
+507,2020-05-28,0.00000000
+507,2020-05-29,0.00000000
+507,2020-05-30,0.00000000
+507,2020-05-31,0.00000000
+507,2020-06-01,0.00000000
+507,2020-06-02,0.00000000
+507,2020-06-03,0.00000000
+507,2020-06-04,0.00000000
+507,2020-06-05,0.00000000
diff --git a/google_health/cache/Data_508_anosmia_ms.csv b/google_health/cache/Data_508_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 508,2020-05-13,0.00000000
 508,2020-05-14,0.00000000
 508,2020-05-15,0.00000000
+508,2020-05-16,0.00000000
+508,2020-05-17,0.00000000
+508,2020-05-18,1926.35941942
+508,2020-05-19,621.38222934
+508,2020-05-20,0.00000000
+508,2020-05-21,0.00000000
+508,2020-05-22,652.92892362
+508,2020-05-23,806.94053088
+508,2020-05-24,0.00000000
+508,2020-05-25,1602.45687507
+508,2020-05-26,0.00000000
+508,2020-05-27,650.70612411
+508,2020-05-28,0.00000000
+508,2020-05-29,0.00000000
+508,2020-05-30,757.45450997
+508,2020-05-31,0.00000000
+508,2020-06-01,1330.88994440
+508,2020-06-02,0.00000000
+508,2020-06-03,612.60192703
+508,2020-06-04,617.44562715
+508,2020-06-05,0.00000000
diff --git a/google_health/cache/Data_509_anosmia_ms.csv b/google_health/cache/Data_509_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 509,2020-05-13,0.00000000
 509,2020-05-14,0.00000000
 509,2020-05-15,0.00000000
+509,2020-05-16,0.00000000
+509,2020-05-17,0.00000000
+509,2020-05-18,5783.96860533
+509,2020-05-19,0.00000000
+509,2020-05-20,0.00000000
+509,2020-05-21,0.00000000
+509,2020-05-22,3252.03252033
+509,2020-05-23,0.00000000
+509,2020-05-24,0.00000000
+509,2020-05-25,0.00000000
+509,2020-05-26,0.00000000
+509,2020-05-27,0.00000000
+509,2020-05-28,0.00000000
+509,2020-05-29,0.00000000
+509,2020-05-30,0.00000000
+509,2020-05-31,0.00000000
+509,2020-06-01,3394.43312967
+509,2020-06-02,0.00000000
+509,2020-06-03,0.00000000
+509,2020-06-04,0.00000000
+509,2020-06-05,0.00000000
diff --git a/google_health/cache/Data_510_anosmia_ms.csv b/google_health/cache/Data_510_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 510,2020-05-13,0.00000000
 510,2020-05-14,413.01426379
 510,2020-05-15,1253.64272964
+510,2020-05-16,0.00000000
+510,2020-05-17,1523.97865145
+510,2020-05-18,1283.33674933
+510,2020-05-19,0.00000000
+510,2020-05-20,0.00000000
+510,2020-05-21,473.06015464
+510,2020-05-22,909.11066682
+510,2020-05-23,0.00000000
+510,2020-05-24,2253.58845766
+510,2020-05-25,0.00000000
+510,2020-05-26,1219.69895589
+510,2020-05-27,0.00000000
+510,2020-05-28,1241.83651857
+510,2020-05-29,461.58151086
+510,2020-05-30,1091.95517512
+510,2020-05-31,547.97312154
+510,2020-06-01,0.00000000
+510,2020-06-02,0.00000000
+510,2020-06-03,0.00000000
+510,2020-06-04,0.00000000
+510,2020-06-05,0.00000000
diff --git a/google_health/cache/Data_511_anosmia_ms.csv b/google_health/cache/Data_511_anosmia_ms.csv
@@ -131,3 +131,24 @@ geo_id,timestamp,val
 511,2020-05-13,583.13736147
 511,2020-05-14,192.40387059
 511,2020-05-15,1588.86531513
+511,2020-05-16,0.00000000
+511,2020-05-17,473.31623810
+511,2020-05-18,864.06666640
+511,2020-05-19,0.00000000
+511,2020-05-20,581.07640622
+511,2020-05-21,599.86759204
+511,2020-05-22,499.89624366
+511,2020-05-23,0.00000000
+511,2020-05-24,490.34399874
+511,2020-05-25,2879.01618651
+511,2020-05-26,989.87515426
+511,2020-05-27,384.72657197
+511,2020-05-28,570.59347775
+511,2020-05-29,607.56038687
+511,2020-05-30,715.99513220
+511,2020-05-31,493.77148344
+511,2020-06-01,804.22369591
+511,2020-06-02,1190.95407243
+511,2020-06-03,853.37872746
+511,2020-06-04,365.94774005
+511,2020-06-05,1312.45218728