Skip to content

Commit 7b8cace

Browse files
krivardkrivard
authored
Merge pull request #970 from cmu-delphi/release/delphi-epidata-0.3.21
Release Delphi Epidata 0.3.21
2 parents 9fa62f2 + 6ea028f commit 7b8cace

29 files changed

+1548
-1259
lines changed

‎.bumpversion.cfg

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 0.3.20
2+
current_version = 0.3.21
33
commit = False
44
tag = False
55

‎README.md

+45-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,51 @@ This is the home of [Delphi](https://delphi.cmu.edu/)'s epidemiological data
88
API. See our [API documentation](https://cmu-delphi.github.io/delphi-epidata/)
99
for details on the available data sets, APIs, and clients.
1010

11-
## COVIDcast
11+
## Development Quickstart
12+
13+
Requires: Docker, possibly sudo access (depending on your Docker installation and OS).
14+
15+
In the directory where you want to work run the following:
16+
17+
```sh
18+
# Make folder structure, download dependent repos, and symlink Makefile
19+
$ curl "https://raw.githubusercontent.com/cmu-delphi/delphi-epidata/dev/dev/local/install.sh" | bash
20+
```
21+
22+
You should now have the following directory structure:
23+
24+
```sh
25+
├── driver
26+
│ ├── .dockerignore -> repos/delphi/delphi-epidata/dev/local/.dockerignore
27+
│ ├── Makefile -> repos/delphi/delphi-epidata/dev/local/Makefile
28+
│ ├── repos
29+
│ │ └── delphi
30+
│ │ ├── delphi-epidata
31+
│ │ ├── flu-contest
32+
│ │ ├── github-deploy-repo
33+
│ │ ├── nowcast
34+
│ │ ├── operations
35+
│ │ └── utils
36+
```
37+
38+
and you should now be in the `driver` directory.
39+
You can now execute make commands
40+
41+
```sh
42+
# Create all docker containers: db, web, and python
43+
$ [sudo] make all
44+
45+
# Run tests
46+
$ [sudo] make test
47+
48+
# To drop into debugger on error
49+
$ [sudo] make test pdb=1
50+
51+
# To test only a subset of tests
52+
$ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition
53+
```
54+
55+
# COVIDcast
1256

1357
At the present, our primary focus is developing and expanding the
1458
[COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html),

‎dev/local/.dockerignore

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Ignore everything by default
2+
*
3+
# Don't ignore repos dir
4+
!repos
5+
# Ignore everything to do with git
6+
**/*.git

‎dev/local/Makefile

+140
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Docker control panel for delphi-epidata development.
2+
#
3+
# Usage: make <command> [pdb=1] [test=<test-subdir>]
4+
#
5+
# Assumes you have installed your environment using
6+
# delphi-epidata/dev/local/install.sh.
7+
#
8+
# Checks for the delphi-net bridge and creates if it doesn't exist.
9+
#
10+
# Creates all prereq images (delphi_database, delphi_python) only if they don't
11+
# exist. If you need to rebuild a prereq, you're probably doing something
12+
# complicated, and can figure out the rebuild command on your own.
13+
#
14+
#
15+
# Commands:
16+
#
17+
# web: Stops currently-running delphi_web_epidata instances, if any.
18+
# Rebuilds delphi_web_epidata image.
19+
# Runs image in the background and pipes stdout to a log file.
20+
#
21+
# db: Stops currently-running delphi_database_epidata instances, if any.
22+
# Rebuilds delphi_database_epidata image.
23+
# Runs image in the background and pipes stdout to a log file.
24+
# Blocks until database is ready to receive connections.
25+
#
26+
# python: Rebuilds delphi_web_python image. You shouldn't need to do this
27+
# often; only if you are installing a new environment, or have
28+
# made changes to delphi-epidata/dev/docker/python/Dockerfile.
29+
#
30+
# all: Runs the commands 'web' 'db' and 'python'.
31+
#
32+
# test: Runs test and integrations in delphi-epidata. If test
33+
# optional arg is provided, then only the tests in that subdir
34+
# are run.
35+
#
36+
# clean: Cleans up dangling Docker images.
37+
#
38+
#
39+
# Optional arguments:
40+
# pdb=1 Drops you into debug mode upon test failure, if running tests.
41+
# test= Only runs tests in the directories provided here, e.g.
42+
# repos/delphi/delphi-epidata/tests/acquisition/covidcast
43+
44+
45+
# Set optional argument defaults
46+
ifdef pdb
47+
override pdb=--pdb
48+
else
49+
pdb=
50+
endif
51+
52+
ifndef test
53+
test=repos/delphi/delphi-epidata/tests repos/delphi/delphi-epidata/integrations
54+
endif
55+
56+
SHELL:=/bin/sh
57+
58+
# Get the Makefile's absolute path: https://stackoverflow.com/a/324782/4784655
59+
# (if called from a symlink, the path is the location of the symlink)
60+
CWD:=$(dir $(abspath $(lastword $(MAKEFILE_LIST))))
61+
NOW:=$(shell date "+%Y-%m-%d")
62+
LOG_WEB:=delphi_web_epidata_$(NOW).log
63+
LOG_DB:=delphi_database_epidata_$(NOW).log
64+
WEB_CONTAINER_ID:=$(shell docker ps -q --filter 'name=delphi_web_epidata')
65+
DATABASE_CONTAINER_ID:=$(shell docker ps -q --filter 'name=delphi_database_epidata')
66+
67+
68+
.PHONY=web
69+
web:
70+
@# Stop container if running
71+
@if [ $(WEB_CONTAINER_ID) ]; then\
72+
docker stop $(WEB_CONTAINER_ID);\
73+
fi
74+
75+
@# Setup virtual network if it doesn't exist
76+
@docker network ls | grep delphi-net || docker network create --driver bridge delphi-net
77+
78+
@# Build the web_epidata image
79+
@cd repos/delphi/delphi-epidata;\
80+
docker build -t delphi_web_epidata -f ./devops/Dockerfile .;\
81+
cd ../../../
82+
83+
@# Run the web server
84+
@docker run --rm -p 127.0.0.1:10080:80 \
85+
--env "SQLALCHEMY_DATABASE_URI=mysql+mysqldb://user:pass@delphi_database_epidata:3306/epidata" \
86+
--env "FLASK_SECRET=abc" --env "FLASK_PREFIX=/epidata" \
87+
--network delphi-net --name delphi_web_epidata \
88+
delphi_web_epidata >$(LOG_WEB) 2>&1 &
89+
90+
.PHONY=db
91+
db:
92+
@# Stop container if running
93+
@if [ $(DATABASE_CONTAINER_ID) ]; then\
94+
docker stop $(DATABASE_CONTAINER_ID);\
95+
fi
96+
97+
@# Only build prereqs if we need them
98+
@docker images delphi_database | grep delphi || \
99+
docker build -t delphi_database -f repos/delphi/operations/dev/docker/database/Dockerfile .
100+
101+
@# Build the database_epidata image
102+
@docker build -t delphi_database_epidata \
103+
-f repos/delphi/delphi-epidata/dev/docker/database/epidata/Dockerfile .
104+
105+
@# Run the database
106+
@docker run --rm -p 127.0.0.1:13306:3306 \
107+
--network delphi-net --name delphi_database_epidata \
108+
delphi_database_epidata >$(LOG_DB) 2>&1 &
109+
110+
@# Block until DB is ready
111+
@while true; do \
112+
sed -n '/Temporary server stopped/,/mysqld: ready for connections/p' $(LOG_DB) | grep "ready for connections" && break; \
113+
tail -1 $(LOG_DB); \
114+
sleep 1; \
115+
done
116+
117+
.PHONY=py
118+
py:
119+
@# Build the python image
120+
@docker build -t delphi_python \
121+
-f repos/delphi/operations/dev/docker/python/Dockerfile .
122+
123+
@docker build -t delphi_web_python \
124+
-f repos/delphi/delphi-epidata/dev/docker/python/Dockerfile .
125+
126+
.PHONY=all
127+
all: web db py
128+
129+
.PHONY=test
130+
test:
131+
@docker run -i --rm --network delphi-net \
132+
--mount type=bind,source=$(CWD)repos/delphi/delphi-epidata,target=/usr/src/app/repos/delphi/delphi-epidata,readonly \
133+
--mount type=bind,source=$(CWD)repos/delphi/delphi-epidata/src,target=/usr/src/app/delphi/epidata,readonly \
134+
--env "SQLALCHEMY_DATABASE_URI=mysql+mysqldb://user:pass@delphi_database_epidata:3306/epidata" \
135+
--env "FLASK_SECRET=abc" \
136+
delphi_web_python python -m pytest --import-mode importlib $(pdb) $(test) | tee test_output_$(NOW).log
137+
138+
.PHONY=clean
139+
clean:
140+
@docker images -f "dangling=true" -q | xargs docker rmi >/dev/null 2>&1

‎dev/local/install.sh

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/bash
2+
# Bootstrap delphi-epidata development
3+
#
4+
# Downloads the repos needed for local delphi-epidata development into current dir
5+
# and provides a Makefile with Docker control commands.
6+
#
7+
# Creates the directory structure:
8+
#
9+
# driver/
10+
# .dockerignore
11+
# Makefile
12+
# repos/
13+
# delphi/
14+
# operations/
15+
# delphi-epidata/
16+
# utils/
17+
# flu-contest/
18+
# nowcast/
19+
# github-deploy-repo/
20+
# undefx/
21+
# py3tester/
22+
# undef-analysis/
23+
#
24+
# Leaves you in driver, the main workdir.
25+
#
26+
27+
28+
mkdir -p driver/repos/delphi
29+
cd driver/repos/delphi
30+
git clone https://github.com/cmu-delphi/operations
31+
git clone https://github.com/cmu-delphi/delphi-epidata
32+
git clone https://github.com/cmu-delphi/utils
33+
git clone https://github.com/cmu-delphi/flu-contest
34+
git clone https://github.com/cmu-delphi/nowcast
35+
git clone https://github.com/cmu-delphi/github-deploy-repo
36+
cd ../../
37+
38+
mkdir -p repos/undefx
39+
cd repos/undefx
40+
git clone https://github.com/undefx/py3tester
41+
git clone https://github.com/undefx/undef-analysis
42+
cd ../../
43+
44+
ln -s repos/delphi/delphi-epidata/dev/local/Makefile
45+
ln -s repos/delphi/delphi-epidata/dev/local/.dockerignore

‎docs/Gemfile.lock

+1-1
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,7 @@ GEM
246246
thread_safe (0.3.6)
247247
typhoeus (1.4.0)
248248
ethon (>= 0.9.0)
249-
tzinfo (1.2.9)
249+
tzinfo (1.2.10)
250250
thread_safe (~> 0.1)
251251
tzinfo-data (1.2021.1)
252252
tzinfo (>= 1.0.0)

‎docs/api/covidcast-signals/chng.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ all of them, and so this source only represents those encounters known to
191191
them. Their coverage may vary across the United States, but they report on about
192192
45% of all doctor's visits nationally.
193193

194-
Standard errors are not available for this data source.
194+
Standard errors and sample sizes are not available for this data source.
195195

196196
Due to changes in medical-seeking behavior on holidays, this data source has
197197
upward spikes in the fraction of doctor's visits that are COVID-related around

‎docs/api/covidcast-signals/doctor-visits.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@ outpatient doctor's visits, but not all of them, and so this source only
160160
represents those visits known to them. Their coverage may vary across the United
161161
States.
162162

163-
Standard errors are not available for this data source.
163+
Standard errors and sample sizes are not available for this data source.
164164

165165
Due to changes in medical-seeking behavior on holidays, this data source has
166166
upward spikes in the fraction of doctor's visits that are COVID-related around

‎docs/api/covidcast-signals/dsew-cpr.md

+2
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,8 @@ This data source is susceptible to large corrections that can create strange dat
6363

6464
Not all CPRs have the same lag between the CPR date (listed in the filename) and the date for a particular signal.
6565

66+
Standard errors and sample sizes are not applicable to these metrics.
67+
6668
### Differences with HHS reports
6769

6870
An analysis comparing the

‎docs/api/covidcast-signals/fb-survey.md

+5-4
Original file line numberDiff line numberDiff line change
@@ -19,17 +19,18 @@ grand_parent: COVIDcast Epidata API
1919

2020
This data source is based on the [COVID-19 Trends and Impact Survey (CTIS)](../../symptom-survey/)
2121
run by the Delphi group at Carnegie Mellon.
22-
Facebook directs a random sample of its users to these surveys, which are
23-
voluntary. Users age 18 or older are eligible to complete the surveys, and
22+
Facebook directed a random sample of its users to these surveys, which were
23+
voluntary. Users age 18 or older were eligible to complete the surveys, and
2424
their survey responses are held by CMU and are sharable with other health
2525
researchers under a data use agreement. No individual survey responses are
2626
shared back to Facebook. See our [surveys
2727
page](https://delphi.cmu.edu/covid19/ctis/) for more detail about how the
2828
surveys work and how they are used outside the COVIDcast API.
2929

3030
As of November 2021, the average number of Facebook survey responses we
31-
receive each day is about 40,000, and the total number of survey responses we
32-
have received is over 25 million.
31+
received each day was about 40,000. The survey ran from April 6, 2020 to June
32+
25, 2022, collecting roughly 29.5 million responses in the United States in that
33+
time.
3334

3435
We produce several sets of signals based on the survey data, listed and
3536
described in the sections below:

‎docs/api/covidcast-signals/google-symptoms.md

+2
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,8 @@ popularity of symptoms in searches within each geographical region individually.
116116
This means that the resulting values of symptom set popularity are **NOT**
117117
comparable across geographic regions, while the values of different symptom sets are comparable within the same location.
118118

119+
Standard errors and sample sizes are not available for this data source.
120+
119121
More details about the limitations of this dataset are available in [Google's Search
120122
Trends symptoms dataset documentation](https://storage.googleapis.com/gcp-public-data-symptom-search/COVID-19%20Search%20Trends%20symptoms%20dataset%20documentation%20.pdf).
121123

‎docs/api/covidcast-signals/hhs.md

+11-8
Original file line numberDiff line numberDiff line change
@@ -32,15 +32,16 @@ adult and pediatric COVID-19 hospital admissions. This sum is used as the
3232
We also include influenza hospital admissions.
3333

3434

35-
| Signal | 7-day average signal | Geography | Resolution | Description |
36-
| --- | --- | --- | --- | --- |
37-
| `confirmed_admissions_covid_1d` | `confirmed_admissions_covid_1d_7dav`| state | 1 day | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31 |
38-
| `confirmed_admissions_covid_1d_prop` | `confirmed_admissions_covid_1d_prop_7dav`| state | 1 day | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
39-
| `sum_confirmed_suspected_admissions_covid_1d` | `sum_confirmed_suspected_admissions_covid_1d_7dav` | state | 1 day | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31 |
40-
| `sum_confirmed_suspected_admissions_covid_1d_prop` | `sum_confirmed_suspected_admissions_covid_1d_prop_7dav` | state | 1 day | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
41-
| `confirmed_admissions_influenza_1d` | `confirmed_admissions_influenza_1d_7dav` | state | 1 day | All confirmed influenza hospital admissions occurring each day. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
42-
| `confirmed_admissions_influenza_1d_prop` | `confirmed_admissions_influenza_1d_prop_7dav` | state | 1 day | All confirmed influenza hospital admissions occurring each day, per 100,000 population. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
35+
| Signal and 7-day average signal | Description |
36+
|:---------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
37+
| `confirmed_admissions_covid_1d` and `confirmed_admissions_covid_1d_7dav` | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31 |
38+
| `confirmed_admissions_covid_1d_prop` and `confirmed_admissions_covid_1d_prop_7dav` | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
39+
| `sum_confirmed_suspected_admissions_covid_1d` and `sum_confirmed_suspected_admissions_covid_1d_7dav` | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31 |
40+
| `sum_confirmed_suspected_admissions_covid_1d_prop` and <br/> `sum_confirmed_suspected_admissions_covid_1d_prop_7dav` | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
41+
| `confirmed_admissions_influenza_1d` and `confirmed_admissions_influenza_1d_7dav` | All confirmed influenza hospital admissions occurring each day. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
42+
| `confirmed_admissions_influenza_1d_prop` and `confirmed_admissions_influenza_1d_prop_7dav` | All confirmed influenza hospital admissions occurring each day, per 100,000 population. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
4343

44+
*for all the above signals & 7-day average signals, their geography is state, and resolution is 1 day.
4445

4546
The 7-day average signals are computed by Delphi by calculating
4647
moving averages of the preceding 7 days, so e.g. the signal for June 7 is the
@@ -96,6 +97,8 @@ psychiatric and rehabilitation facilities, Indian Health Service (IHS)
9697
facilities, U.S. Department of Veterans Affairs (VA) facilities, Defense Health
9798
Agency (DHA) facilities, and religious non-medical facilities.
9899

100+
Standard errors and sample sizes are not applicable to these metrics.
101+
99102
## Lag and Backfill
100103

101104
HHS issues updates to this timeseries once a week, and occasionally more

‎docs/api/covidcast-signals/hospital-admissions.md

+2
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,8 @@ hospitalizations, but not all of them, and so this source only represents those
6868
hospitalizations known to them. Their coverage may vary across the United
6969
States.
7070

71+
Standard errors and sample sizes are not available for this data source.
72+
7173
## Qualifying Admissions
7274

7375
We receive two daily data streams of new hospital admissions recorded by the health system partners at each location. One stream is based on electronic medical records, and the other comes from claims records.

0 commit comments

Comments
 (0)