Skip to content

Commit 61be514

Browse files
melange396dmytrotskorzatscapnrefsmmatnmdefries
authored
Merge pull request #1267 from cmu-delphi/release/delphi-epidata-4.1.8
* Performance testing testing workflow (#1253) * New CTIS publication (#1255) * Newer CTIS publication * bring loggers in sync and add multiproc capabilities (#1254) * add syntax feature documentation (#1256) * Newest CTIS publication * chore: sync to www-covidcast release v3.2.7 * moving quidel signals to non-public access (#1261) with integration tests! --------- Co-authored-by: Dmytro Trotsko <[email protected]> * chore: release delphi-epidata 4.1.8 --------- Co-authored-by: Dmytro Trotsko <[email protected]> Co-authored-by: Rostyslav Zatserkovnyi <[email protected]> Co-authored-by: Alex Reinhart <[email protected]> Co-authored-by: nmdefries <[email protected]> Co-authored-by: melange396 <[email protected]> Co-authored-by: minhkhul <[email protected]> Co-authored-by: minhkhul <[email protected]> Co-authored-by: melange396 <[email protected]>
2 parents 896acf1 + 72778e9 commit 61be514

File tree

21 files changed

+480
-61
lines changed

21 files changed

+480
-61
lines changed

‎.bumpversion.cfg

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 4.1.7
2+
current_version = 4.1.8
33
commit = False
44
tag = False
55

‎.github/workflows/performance-tests-periodic.yml

+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
name: One-time performance testing - 9th August 2023
2+
3+
# Run "At every 30th minute on day-of-month 9 in August"
4+
on:
5+
schedule:
6+
- cron: '*/30 * 9 8 *'
7+
8+
# Add some extra perms to comment on a PR
9+
permissions:
10+
pull-requests: write
11+
contents: read
12+
13+
jobs:
14+
run-perftests:
15+
runs-on: ubuntu-latest
16+
outputs:
17+
request_count: ${{ steps.output.outputs.request_count }}
18+
failure_count: ${{ steps.output.outputs.failure_count }}
19+
med_time: ${{ steps.output.outputs.med_time }}
20+
avg_time: ${{ steps.output.outputs.avg_time }}
21+
min_time: ${{ steps.output.outputs.min_time }}
22+
max_time: ${{ steps.output.outputs.max_time }}
23+
requests_per_sec: ${{ steps.output.outputs.requests_per_sec }}
24+
steps:
25+
- name: Set up WireGuard
26+
uses: egor-tensin/[email protected]
27+
with:
28+
endpoint: '${{ secrets.WG_PERF_ENDPOINT }}'
29+
endpoint_public_key: '${{ secrets.WG_PERF_ENDPOINT_PUBLIC_KEY }}'
30+
ips: '${{ secrets.WG_PERF_IPS }}'
31+
allowed_ips: '${{ secrets.WG_PERF_ALLOWED_IPS }}'
32+
private_key: '${{ secrets.WG_PERF_PRIVATE_KEY }}'
33+
- name: Check out repository
34+
uses: actions/checkout@v3
35+
- name: Set up repository # mimics install.sh in the README except that delphi is cloned from the PR rather than main
36+
run: |
37+
cd ..
38+
mkdir -p driver/repos/delphi
39+
cd driver/repos/delphi
40+
git clone https://github.com/cmu-delphi/operations
41+
git clone https://github.com/cmu-delphi/utils
42+
git clone https://github.com/cmu-delphi/flu-contest
43+
git clone https://github.com/cmu-delphi/nowcast
44+
cd ../../
45+
46+
cd ..
47+
cp -R delphi-epidata driver/repos/delphi/delphi-epidata
48+
cd -
49+
50+
ln -s repos/delphi/delphi-epidata/dev/local/Makefile
51+
- name: Build & run epidata
52+
run: |
53+
cd ../driver
54+
sudo make web sql="${{ secrets.DB_CONN_STRING }}"
55+
sudo make redis
56+
- name: Check out delphi-admin
57+
uses: actions/checkout@v3
58+
with:
59+
repository: cmu-delphi/delphi-admin
60+
token: ${{ secrets.CMU_DELPHI_DEPLOY_MACHINE_PAT }}
61+
path: delphi-admin
62+
- name: Build & run Locust
63+
continue-on-error: true # sometimes ~2-5 queries fail, we shouldn't end the run if that's the case
64+
run: |
65+
cd delphi-admin/load-testing/locust
66+
docker build -t locust .
67+
export CSV=v4-requests-small.csv
68+
touch output_stats.csv && chmod 666 output_stats.csv
69+
touch output_stats_history.csv && chmod 666 output_stats_history.csv
70+
touch output_failures.csv && chmod 666 output_failures.csv
71+
touch output_exceptions.csv && chmod 666 output_exceptions.csv
72+
docker run --net=host -v $PWD:/mnt/locust -e CSV="/mnt/locust/${CSV}" locust -f /mnt/locust/v4.py --host http://127.0.0.1:10080/ --users 10 --spawn-rate 1 --headless -i "$(cat ${CSV} | wc -l)" --csv=/mnt/locust/output
73+
- name: Produce output for summary
74+
id: output
75+
uses: jannekem/run-python-script-action@v1
76+
with:
77+
script: |
78+
import os
79+
80+
def write_string(name, value):
81+
with open(os.environ['GITHUB_OUTPUT'], 'a') as fh:
82+
print(f'{name}={value}', file=fh)
83+
84+
def write_float(name, value):
85+
write_string(name, "{:.2f}".format(float(value)))
86+
87+
with open("delphi-admin/load-testing/locust/output_stats.csv", "r", encoding="utf-8", errors="ignore") as scraped:
88+
final_line = scraped.readlines()[-1].split(",")
89+
write_string('request_count', final_line[2])
90+
write_string('failure_count', final_line[3])
91+
write_float('med_time', final_line[4])
92+
write_float('avg_time', final_line[5])
93+
write_float('min_time', final_line[6])
94+
write_float('max_time', final_line[7])
95+
write_float('requests_per_sec', final_line[9])
96+
97+
- name: Archive results as artifacts
98+
uses: actions/upload-artifact@v3
99+
with:
100+
name: locust-output
101+
path: |
102+
delphi-admin/load-testing/locust/output_*.csv

‎dev/local/setup.cfg

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[metadata]
22
name = Delphi Development
3-
version = 4.1.7
3+
version = 4.1.8
44

55
[options]
66
packages =

‎docs/api/covidcast-signals/quidel-inactive.md

+1
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ grand_parent: COVIDcast Main Endpoint
1515
1. TOC
1616
{:toc}
1717

18+
## Accessibility: Delphi-internal only
1819

1920
## COVID-19 Tests
2021
These signals are still active. Documentation is available on the [Quidel page](quidel.md).

‎docs/api/covidcast-signals/quidel.md

+2
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ grand_parent: COVIDcast Main Endpoint
1515
1. TOC
1616
{:toc}
1717

18+
## Accessibility: Delphi-internal only
19+
1820
## COVID-19 Tests
1921

2022
* **Earliest issue available:** July 29, 2020

‎docs/api/covidcast.md

+16
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,16 @@ and lists.
113113
The current set of signals available for each data source is returned by the
114114
[`covidcast_meta`](covidcast_meta.md) endpoint.
115115

116+
#### Alternate Required Parameters
117+
118+
The following parameters help specify multiple source-signal, timetype-timevalue or geotype-geovalue pairs. Use them instead of the usual required parameters.
119+
120+
| Parameter | Replaces | Format | Description | Example |
121+
| --- | --- | --- | --- | --- |
122+
| `signal` | `data_source`, `signal` | `signal={source}:{signal1},{signal2}` | Specify multiple source-signal pairs, grouped by source | `signal=src1:sig1`, `signal=src1:sig1,sig2`, `signal=src1:*`, `signal=src1:sig1;src2:sig3` |
123+
| `time` | `time_type`, `time_values` | `time={timetype}:{timevalue1},{timevalue2}` | Specify multiple timetype-timevalue pairs, grouped by timetype | `time=day:*`, `time=day:20201201`, `time=day:20201201,20201202`, `time=day:20201201-20201204` |
124+
| `geo` | `geo_type`, `geo_value` | `geo={geotype}:{geovalue1},{geovalue2}` | Specify multiple geotype-geovalue pairs, grouped by geotype | `geo=fips:*`, `geo=fips:04019`, `geo=fips:04019,19143`, `geo=fips:04019;msa:40660`, `geo=fips:*;msa:*` |
125+
116126
#### Optional
117127

118128
Estimates for a specific `time_value` and `geo_value` are sometimes updated
@@ -209,6 +219,12 @@ The `fields` parameter can be used to limit which fields are included in each re
209219

210220
https://api.delphi.cmu.edu/epidata/covidcast/?data_source=fb-survey&signal=smoothed_cli&time_type=day&geo_type=county&time_values=20200406-20200410&geo_value=06001
211221

222+
or
223+
224+
https://api.delphi.cmu.edu/epidata/covidcast/?signal=fb-survey:smoothed_cli&time=day:20200406-20200410&geo=county:06001
225+
226+
Both of these URLs are equivalent and can be used to get the following result:
227+
212228
```json
213229
{
214230
"result": 1,

‎docs/api/covidcast_signals.md

-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,6 @@ dashboard](https://delphi.cmu.edu/covidcast/):
3636
| Early Indicators | COVID-Like Symptoms | [`fb-survey`](covidcast-signals/fb-survey.md) | `smoothed_wcli` |
3737
| Early Indicators | COVID-Like Symptoms in Community | [`fb-survey`](covidcast-signals/fb-survey.md) | `smoothed_whh_cmnty_cli` |
3838
| Early Indicators | COVID-Related Doctor Visits | [`doctor-visits`](covidcast-signals/doctor-visits.md) | `smoothed_adj_cli` |
39-
| Cases and Testing | COVID Antigen Test Positivity (Quidel) | [`quidel`](covidcast-signals/quidel.md) | `covid_ag_smoothed_pct_positive` |
4039
| Cases and Testing | COVID Cases | [`jhu-csse`](covidcast-signals/jhu-csse.md) | `confirmed_7dav_incidence_prop` |
4140
| Late Indicators | COVID Hospital Admissions | [`hhs`](covidcast-signals/hhs.md) | `confirmed_admissions_covid_1d_prop_7dav` |
4241
| Late Indicators | Deaths | [`jhu-csse`](covidcast-signals/jhu-csse.md) | `deaths_7dav_incidence_prop` |

‎docs/symptom-survey/publications.md

+11
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,17 @@ Pandemic"](https://www.pnas.org/topic/548) in *PNAS*:
2626

2727
Research publications using the survey data include:
2828

29+
- C.K. Ettman, E. Badillo Goicoechea, and E.A. Stuart (2023). [Evolution of
30+
depression and anxiety over the COVID-19 pandemic and across demographic
31+
groups in a large sample of U.S. adults](https://doi.org/10.1016/j.focus.2023.100140).
32+
*AJPM Focus*.
33+
- M. Rubinstein, Z. Branson, and E.H. Kennedy (2023). [Heterogeneous
34+
interventional effects with multiple mediators: Semiparametric and
35+
nonparametric approaches](https://doi.org/10.1515/jci-2022-0070). *Journal of
36+
Causal Inference* 11 (1), 20220070.
37+
- Uyheng, J., Robertson, D.C. & Carley, K.M. (2023). [Bridging online and offline
38+
dynamics of the face mask infodemic](https://doi.org/10.1186/s44247-023-00026-z).
39+
*BMC Digital Health* 1, 27.
2940
- Kobayashi H, Saenz-Escarcega R, Fulk A, Agusto FB (2023). [Understanding
3041
mental health trends during COVID-19 pandemic in the United States using
3142
network analysis](https://doi.org/10.1371/journal.pone.0286857). *PLoS

‎integrations/server/test_covidcast_endpoints.py

+65-2
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,19 @@ def localSetUp(self):
2626
# reset the `covidcast_meta_cache` table (it should always have one row)
2727
self._db._cursor.execute('update covidcast_meta_cache set timestamp = 0, epidata = "[]"')
2828

29-
def _fetch(self, endpoint="/", is_compatibility=False, **params):
29+
cur = self._db._cursor
30+
# NOTE: we must specify the db schema "epidata" here because the cursor/connection are bound to schema "covid"
31+
cur.execute("TRUNCATE TABLE epidata.api_user")
32+
cur.execute("TRUNCATE TABLE epidata.user_role")
33+
cur.execute("TRUNCATE TABLE epidata.user_role_link")
34+
cur.execute("INSERT INTO epidata.api_user (api_key, email) VALUES ('quidel_key', 'quidel_email')")
35+
cur.execute("INSERT INTO epidata.user_role (name) VALUES ('quidel')")
36+
cur.execute(
37+
"INSERT INTO epidata.user_role_link (user_id, role_id) SELECT api_user.id, user_role.id FROM epidata.api_user JOIN epidata.user_role WHERE api_key='quidel_key' and user_role.name='quidel'"
38+
)
39+
cur.execute("INSERT INTO epidata.api_user (api_key, email) VALUES ('key', 'email')")
40+
41+
def _fetch(self, endpoint="/", is_compatibility=False, auth=AUTH, **params):
3042
# make the request
3143
if is_compatibility:
3244
url = BASE_URL_OLD
@@ -37,7 +49,7 @@ def _fetch(self, endpoint="/", is_compatibility=False, **params):
3749
params.setdefault("data_source", params.get("source"))
3850
else:
3951
url = f"{BASE_URL}{endpoint}"
40-
response = requests.get(url, params=params, auth=AUTH)
52+
response = requests.get(url, params=params, auth=auth)
4153
response.raise_for_status()
4254
return response.json()
4355

@@ -67,6 +79,28 @@ def test_basic(self):
6779
out = self._fetch("/", signal=first.signal_pair(), geo=first.geo_pair(), time="day:*")
6880
self.assertEqual(len(out["epidata"]), len(rows))
6981

82+
def test_basic_restricted_source(self):
83+
"""Request a signal from the / endpoint."""
84+
rows = [CovidcastTestRow.make_default_row(time_value=2020_04_01 + i, value=i, source="quidel") for i in range(10)]
85+
first = rows[0]
86+
self._insert_rows(rows)
87+
88+
with self.subTest("validation"):
89+
out = self._fetch("/")
90+
self.assertEqual(out["result"], -1)
91+
92+
with self.subTest("no_roles"):
93+
out = self._fetch("/", signal=first.signal_pair(), geo=first.geo_pair(), time="day:*")
94+
self.assertEqual(len(out["epidata"]), 0)
95+
96+
with self.subTest("no_api_key"):
97+
out = self._fetch("/", auth=None, signal=first.signal_pair(), geo=first.geo_pair(), time="day:*")
98+
self.assertEqual(len(out["epidata"]), 0)
99+
100+
with self.subTest("quidel_role"):
101+
out = self._fetch("/", auth=("epidata", "quidel_key"), signal=first.signal_pair(), geo=first.geo_pair(), time="day:*")
102+
self.assertEqual(len(out["epidata"]), len(rows))
103+
70104
def test_compatibility(self):
71105
"""Request at the /api.php endpoint."""
72106
rows = [CovidcastTestRow.make_default_row(source="src", signal="sig", time_value=2020_04_01 + i, value=i) for i in range(10)]
@@ -271,6 +305,35 @@ def test_meta(self):
271305
out = self._fetch("/meta", signal=f"{first.source}:X")
272306
self.assertEqual(len(out), 0)
273307

308+
def test_meta_restricted(self):
309+
"""Request 'restricted' signals from the /meta endpoint."""
310+
# NOTE: this method is nearly identical to ./test_covidcast_meta.py:test_restricted_sources()
311+
# ...except the self._fetch() methods are different, as is the format of those methods' outputs
312+
# (the other covidcast_meta endpoint uses APrinter, this one returns its own unadulterated json).
313+
# additionally, the sample data used here must match entries (that is, named sources and signals)
314+
# from covidcast_utils.model.data_sources (the `data_sources` variable from file
315+
# src/server/endpoints/covidcast_utils/model.py, which is created by the _load_data_sources() method
316+
# and fed by src/server/endpoints/covidcast_utils/db_sources.csv, but also surreptitiously augmened
317+
# by _load_data_signals() which attaches a list of signals to each source,
318+
# in turn fed by src/server/endpoints/covidcast_utils/db_signals.csv)
319+
320+
# insert data from two different sources, one restricted/protected (quidel), one not
321+
self._insert_rows([
322+
CovidcastTestRow.make_default_row(source="quidel", signal="raw_pct_negative"),
323+
CovidcastTestRow.make_default_row(source="hhs", signal="confirmed_admissions_covid_1d")
324+
])
325+
326+
# update metadata cache
327+
update_cache(args=None)
328+
329+
# verify unauthenticated (no api key) or unauthorized (user w/o privilege) only see metadata for one source
330+
self.assertEqual(len(self._fetch("/meta", auth=None)), 1)
331+
self.assertEqual(len(self._fetch("/meta", auth=AUTH)), 1)
332+
333+
# verify authorized user sees metadata for both sources
334+
qauth = ('epidata', 'quidel_key')
335+
self.assertEqual(len(self._fetch("/meta", auth=qauth)), 2)
336+
274337
def test_coverage(self):
275338
"""Request a signal from the /coverage endpoint."""
276339

‎integrations/server/test_covidcast_meta.py

+37-5
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99

1010
#first party
1111
from delphi_utils import Nans
12+
from delphi.epidata.acquisition.covidcast.test_utils import CovidcastBase, CovidcastTestRow
1213
from delphi.epidata.maintenance.covidcast_meta_cache_updater import main as update_cache
1314
import delphi.operations.secrets as secrets
1415

@@ -17,7 +18,7 @@
1718
AUTH = ('epidata', 'key')
1819

1920

20-
class CovidcastMetaTests(unittest.TestCase):
21+
class CovidcastMetaTests(CovidcastBase):
2122
"""Tests the `covidcast_meta` endpoint."""
2223

2324
src_sig_lookups = {
@@ -48,7 +49,7 @@ class CovidcastMetaTests(unittest.TestCase):
4849
%d, %d)
4950
'''
5051

51-
def setUp(self):
52+
def localSetUp(self):
5253
"""Perform per-test setup."""
5354

5455
# connect to the `epidata` database and clear the `covidcast` table
@@ -68,6 +69,17 @@ def setUp(self):
6869
# reset the `covidcast_meta_cache` table (it should always have one row)
6970
cur.execute('update covidcast_meta_cache set timestamp = 0, epidata = "[]"')
7071

72+
# NOTE: we must specify the db schema "epidata" here because the cursor/connection are bound to schema "covid"
73+
cur.execute("TRUNCATE TABLE epidata.api_user")
74+
cur.execute("TRUNCATE TABLE epidata.user_role")
75+
cur.execute("TRUNCATE TABLE epidata.user_role_link")
76+
cur.execute("INSERT INTO epidata.api_user (api_key, email) VALUES ('quidel_key', 'quidel_email')")
77+
cur.execute("INSERT INTO epidata.user_role (name) VALUES ('quidel')")
78+
cur.execute(
79+
"INSERT INTO epidata.user_role_link (user_id, role_id) SELECT api_user.id, user_role.id FROM epidata.api_user JOIN epidata.user_role WHERE api_key='quidel_key' and user_role.name='quidel'"
80+
)
81+
cur.execute("INSERT INTO epidata.api_user (api_key, email) VALUES ('key', 'email')")
82+
7183
# populate dimension tables
7284
for (src,sig) in self.src_sig_lookups:
7385
cur.execute('''
@@ -93,7 +105,7 @@ def setUp(self):
93105
secrets.db.epi = ('user', 'pass')
94106

95107

96-
def tearDown(self):
108+
def localTearDown(self):
97109
"""Perform per-test teardown."""
98110
self.cur.close()
99111
self.cnx.close()
@@ -138,10 +150,10 @@ def _get_id(self):
138150
return self.id_counter
139151

140152
@staticmethod
141-
def _fetch(**kwargs):
153+
def _fetch(auth=AUTH, **kwargs):
142154
params = kwargs.copy()
143155
params['endpoint'] = 'covidcast_meta'
144-
response = requests.get(BASE_URL, params=params, auth=AUTH)
156+
response = requests.get(BASE_URL, params=params, auth=auth)
145157
response.raise_for_status()
146158
return response.json()
147159

@@ -161,6 +173,26 @@ def test_round_trip(self):
161173
'message': 'success',
162174
})
163175

176+
def test_restricted_sources(self):
177+
# NOTE: this method is nearly identical to ./test_covidcast_endpoints.py:test_meta_restricted()
178+
179+
# insert data from two different sources, one restricted/protected (quidel), one not
180+
self._insert_rows([
181+
CovidcastTestRow.make_default_row(source="quidel"),
182+
CovidcastTestRow.make_default_row(source="not-quidel")
183+
])
184+
185+
# generate metadata cache
186+
update_cache(args=None)
187+
188+
# verify unauthenticated (no api key) or unauthorized (user w/o privilege) only see metadata for one source
189+
self.assertEqual(len(self._fetch(auth=None)['epidata']), 1)
190+
self.assertEqual(len(self._fetch(auth=AUTH)['epidata']), 1)
191+
192+
# verify authorized user sees metadata for both sources
193+
qauth = ('epidata', 'quidel_key')
194+
self.assertEqual(len(self._fetch(auth=qauth)['epidata']), 2)
195+
164196
def test_filter(self):
165197
"""Test filtering options some sample data."""
166198

‎src/client/delphi_epidata.R

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Epidata <- (function() {
1515
# API base url
1616
BASE_URL <- getOption('epidata.url', default = 'https://api.delphi.cmu.edu/epidata/')
1717

18-
client_version <- '4.1.7'
18+
client_version <- '4.1.8'
1919

2020
auth <- getOption("epidata.auth", default = NA)
2121

‎src/client/delphi_epidata.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
}
2323
})(this, function (exports, fetchImpl, jQuery) {
2424
const BASE_URL = "https://api.delphi.cmu.edu/epidata/";
25-
const client_version = "4.1.7";
25+
const client_version = "4.1.8";
2626

2727
// Helper function to cast values and/or ranges to strings
2828
function _listitem(value) {

0 commit comments

Comments
 (0)