Skip to content

Commit 60a5202

Browse files
benjaminysmithbenjaminysmith
committed
2 parents 0d10717 + 753c688 commit 60a5202

File tree

977 files changed

+212452
-274843
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

977 files changed

+212452
-274843
lines changed

‎.github/CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ git checkout -b dev-my-feature-branch
6161

6262
### Creating your indicator
6363

64-
Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach.
64+
Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. If using Python, add the name of the directory to the list found in `jobs > build > strategy > matrix > packages` in `.github/workflows/python-ci.yml`, which will enable automated checks for your indicator when you make PRs. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach.
6565

6666
- Consult your peers with questions! :handshake:
6767

‎.github/ISSUE_TEMPLATE/feature_release.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,14 @@ assignees: 'benjaminysmith'
1515

1616
<!-- relevant for most work -->
1717

18-
- [ ] API [documentation](https://github.com/cmu-delphi/delphi-epidata/tree/main/docs/api) and/or [changelog](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast_changelog.md)
18+
- [ ] API [documentation](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast-signals/_source-template.md) and/or [changelog](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast_changelog.md)
1919
- [ ] API mailing list notification
2020

2121
<!-- relevant for new signals -->
2222

2323
- [ ] Statistical review (usually [correlations](https://github.com/cmu-delphi/covidcast/tree/main/docs/R-notebooks))
2424
- [ ] Signal / source name review (usually [Roni](https://docs.google.com/document/d/10hGd4Evce4lJ4VkWaQEKFQxvmw2P4xyYGtIAWF52Sf8/edit?usp=sharing))
25+
- [ ] Verify licensing terms and any applicable DUA restrictions
2526

2627
<!-- relevant for new map signals -->
2728

‎.github/pull_request_template.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
### Description
2+
Type of change (bug fix, new feature, etc), brief description, and motivation for these changes.
3+
4+
### Changelog
5+
Itemize code/test/documentation changes and files added/removed.
6+
- change1
7+
- change2
8+
9+
### Fixes
10+
- Fixes #(issue)

‎.github/workflows/json-check.yml

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# This workflow will verify all JSON files in the directory, excluding venv files
2+
3+
name: JSON check
4+
5+
on:
6+
push:
7+
branches: [ main, prod ]
8+
pull_request:
9+
types: [ opened, synchronize, reopened, ready_for_review ]
10+
branches: [ main, prod ]
11+
12+
jobs:
13+
build:
14+
runs-on: ubuntu-20.04
15+
if: github.event.pull_request.draft == false
16+
steps:
17+
- uses: actions/checkout@v2
18+
- name: Set up Python 3.8
19+
uses: actions/setup-python@v2
20+
with:
21+
python-version: 3.8
22+
- name: Check JSONs
23+
run: |
24+
python3 -c "import json; from pathlib import Path
25+
for i in Path('.').glob('**/*.json*'):
26+
if '/env/' not in str(i):
27+
print(i)
28+
json.load(open(str(i)))"

‎.github/workflows/python-ci.yml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
2+
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
3+
4+
name: Python package
5+
6+
on:
7+
push:
8+
branches: [ main, prod ]
9+
pull_request:
10+
types: [ opened, synchronize, reopened, ready_for_review ]
11+
branches: [ main, prod ]
12+
13+
jobs:
14+
build:
15+
runs-on: ubuntu-20.04
16+
if: github.event.pull_request.draft == false
17+
strategy:
18+
matrix:
19+
packages: [_delphi_utils_python, cdc_covidnet, changehc, claims_hosp, combo_cases_and_deaths, google_symptoms, hhs_hosp, hhs_facilities, jhu, nchs_mortality, quidel, quidel_covidtest, safegraph, safegraph_patterns, usafacts]
20+
defaults:
21+
run:
22+
working-directory: ${{ matrix.packages }}
23+
steps:
24+
- uses: actions/checkout@v2
25+
- name: Set up Python 3.8
26+
uses: actions/setup-python@v2
27+
with:
28+
python-version: 3.8
29+
- name: Install testing dependencies
30+
run: |
31+
python -m pip install --upgrade pip
32+
pip install pylint pytest pydocstyle wheel
33+
- name: Install
34+
run: |
35+
make install
36+
- name: Lint
37+
if: ${{ matrix.packages != 'claims_hosp' }}
38+
run: |
39+
make lint
40+
- name: Test
41+
run: |
42+
make test

‎.gitignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,10 +119,17 @@ venv.bak/
119119
# VSCode settings
120120
*.vscode
121121

122+
# Pycharm settings
123+
.idea/
124+
122125
# mypy
123126
.mypy_cache/
124127

125128
# Ansible
126129
.retry
127130
.indicators-ansible-vault-pass
128131
indicators-ansible-vault-pass
132+
133+
# testing_utils
134+
testing_utils/cache
135+
testing_utils/*.csv

‎Jenkinsfile

Lines changed: 42 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,81 +1,73 @@
11
#!groovy
22

3-
// import shared library: https://github.com/cmu-delphi/jenkins-shared-library
3+
// Import shared lib.
44
@Library('jenkins-shared-library') _
55

6-
pipeline {
6+
/*
7+
Declare variables.
8+
- indicator_list should contain all the indicators to handle in the pipeline.
9+
- Keep in sync with '.github/workflows/python-ci.yml'.
10+
- TODO: #527 Get this list automatically from python-ci.yml at runtime.
11+
*/
12+
def indicator_list = ["cdc_covidnet", "changehc", "claims_hosp", "combo_cases_and_deaths", "google_health", "google_symptoms", "hhs_hosp", "jhu", "nchs_mortality", "quidel", "quidel_covidtest", "safegraph", "safegraph_patterns", "sir_complainsalot", "usafacts"]
13+
def build_package = [:]
14+
def deploy_staging = [:]
15+
def deploy_production = [:]
716

17+
pipeline {
818
agent any
9-
1019
stages {
11-
12-
stage ("Environment") {
20+
stage('Build and Package') {
1321
when {
14-
anyOf {
15-
branch "deploy-*";
16-
changeRequest target: "deploy-*", comparator: "GLOB"
17-
}
22+
branch "main";
1823
}
1924
steps {
2025
script {
21-
// Get the indicator name from the pipeline env.
22-
if ( env.CHANGE_TARGET ) {
23-
INDICATOR = env.CHANGE_TARGET.replaceAll("deploy-", "")
24-
}
25-
else if ( env.BRANCH_NAME ) {
26-
INDICATOR = env.BRANCH_NAME.replaceAll("deploy-", "")
27-
}
28-
else {
29-
INDICATOR = ""
26+
indicator_list.each { indicator ->
27+
build_package[indicator] = {
28+
sh "jenkins/build-and-package.sh ${indicator}"
29+
}
3030
}
31-
}
32-
}
33-
}
34-
35-
stage('Build') {
36-
when {
37-
changeRequest target: "deploy-*", comparator: "GLOB"
38-
}
39-
steps {
40-
sh "jenkins/${INDICATOR}-jenkins-build.sh"
41-
}
42-
}
43-
44-
stage('Test') {
45-
when {
46-
changeRequest target: "deploy-*", comparator: "GLOB"
47-
}
48-
steps {
49-
sh "jenkins/${INDICATOR}-jenkins-test.sh"
31+
parallel build_package
32+
}
5033
}
5134
}
52-
53-
stage('Package') {
35+
stage('Deploy staging') {
5436
when {
55-
changeRequest target: "deploy-*", comparator: "GLOB"
37+
branch "main";
5638
}
5739
steps {
58-
sh "jenkins/${INDICATOR}-jenkins-package.sh"
40+
script {
41+
indicator_list.each { indicator ->
42+
deploy_staging[indicator] = {
43+
sh "jenkins/deploy-staging.sh ${indicator}"
44+
}
45+
}
46+
parallel deploy_staging
47+
}
48+
sh "jenkins/deploy-staging-api-match-list.sh"
5949
}
6050
}
61-
62-
stage('Deploy') {
51+
stage('Deploy production') {
6352
when {
64-
branch "deploy-*"
53+
branch "prod";
6554
}
6655
steps {
67-
sh "jenkins/${INDICATOR}-jenkins-deploy.sh"
56+
script {
57+
indicator_list.each { indicator ->
58+
deploy_production[indicator] = {
59+
sh "jenkins/deploy-production.sh ${indicator}"
60+
}
61+
}
62+
parallel deploy_production
63+
}
6864
}
6965
}
7066
}
71-
7267
post {
7368
always {
7469
script {
75-
/*
76-
Use slackNotifier.groovy from shared library and provide current
77-
build result as parameter.
78-
*/
70+
// Use slackNotifier.groovy from shared lib.
7971
slackNotifier(currentBuild.currentResult)
8072
}
8173
}

‎_delphi_utils_python/.pylintrc

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,22 @@
1-
[DESIGN]
21

3-
min-public-methods=1
2+
[MESSAGES CONTROL]
43

4+
disable=logging-format-interpolation,
5+
too-many-locals,
6+
too-many-arguments,
7+
# Allow pytest functions to be part of a class.
8+
no-self-use,
9+
# Allow pytest classes to have one test.
10+
too-few-public-methods
511

6-
[MESSAGES CONTROL]
12+
[BASIC]
13+
14+
# Allow arbitrarily short-named variables.
15+
variable-rgx=[a-z_][a-z0-9_]*
16+
argument-rgx=[a-z_][a-z0-9_]*
17+
attr-rgx=[a-z_][a-z0-9_]*
18+
19+
[DESIGN]
720

8-
disable=R0801, C0330, E1101, E0611, C0114, C0116, C0103, R0913, R0914, W0702
21+
# Don't complain about pytest "unused" arguments.
22+
ignored-argument-names=(_.*|run_as_module)

‎_delphi_utils_python/Makefile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
.PHONY = venv, lint, test, clean
2+
3+
dir = $(shell find ./delphi_* -name __init__.py | grep -o 'delphi_[_[:alnum:]]*')
4+
5+
venv:
6+
python3.8 -m venv env
7+
8+
install: venv
9+
. env/bin/activate; \
10+
pip install wheel ; \
11+
pip install -e .
12+
13+
lint:
14+
. env/bin/activate; \
15+
pylint $(dir); \
16+
pydocstyle $(dir)
17+
18+
test:
19+
. env/bin/activate ;\
20+
(cd tests && ../env/bin/pytest --cov=$(dir) --cov-report=term-missing)
21+
22+
clean:
23+
rm -rf env
24+
rm -f params.json

0 commit comments

Comments
 (0)