Skip to content

Commit e99873d

Browse files
antonpirkerantonpirker
authored
Better test coverage reports (#3498)
Our coverage reports are broken. This PR tries to fix them. - Sometimes the coverage report XML files contain references to files in `/tmp/...` (this can happen if dependencies write those files) so the first change is to omit those files. - We created our coverage reports with `coverage xml -i` where the `-i` means "ignore errors". This is why we never found out about problems generating coverage reports. Report generation fails now verbose (everywhere except in Python 3.6, because there are always some errors there because it can not parse python files with async code, but I guess those can be savely ignored) - For Python 3.6 we know have a special coverage config (`.coveragerc36`) because the option `exclude_also` was named `exclude_lines` in older coverage.py versions.
1 parent 0fb9606 commit e99873d

17 files changed

+223
-69
lines changed

Diff for: ‎.coveragerc36

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# This is the coverage.py config for Python 3.6
2+
# The config for newer Python versions is in pyproject.toml.
3+
4+
[run]
5+
branch = true
6+
omit =
7+
/tmp/*
8+
*/tests/*
9+
*/.venv/*
10+
11+
12+
[report]
13+
exclude_lines =
14+
"if TYPE_CHECKING:",

Diff for: ‎.github/workflows/test-integrations-ai.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -65,11 +65,17 @@ jobs:
6565
run: |
6666
set -x # print commands that are executed
6767
./scripts/runtox.sh "py${{ matrix.python-version }}-huggingface_hub-latest"
68+
- name: Generate coverage XML (Python 3.6)
69+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
70+
run: |
71+
export COVERAGE_RCFILE=.coveragerc36
72+
coverage combine .coverage-sentry-*
73+
coverage xml --ignore-errors
6874
- name: Generate coverage XML
69-
if: ${{ !cancelled() }}
75+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
7076
run: |
71-
coverage combine .coverage*
72-
coverage xml -i
77+
coverage combine .coverage-sentry-*
78+
coverage xml
7379
- name: Upload coverage to Codecov
7480
if: ${{ !cancelled() }}
7581
uses: codecov/[email protected]
@@ -127,11 +133,17 @@ jobs:
127133
run: |
128134
set -x # print commands that are executed
129135
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-huggingface_hub"
136+
- name: Generate coverage XML (Python 3.6)
137+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
138+
run: |
139+
export COVERAGE_RCFILE=.coveragerc36
140+
coverage combine .coverage-sentry-*
141+
coverage xml --ignore-errors
130142
- name: Generate coverage XML
131-
if: ${{ !cancelled() }}
143+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
132144
run: |
133-
coverage combine .coverage*
134-
coverage xml -i
145+
coverage combine .coverage-sentry-*
146+
coverage xml
135147
- name: Upload coverage to Codecov
136148
if: ${{ !cancelled() }}
137149
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-aws-lambda.yml

+9-3
Original file line numberDiff line numberDiff line change
@@ -84,11 +84,17 @@ jobs:
8484
run: |
8585
set -x # print commands that are executed
8686
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-aws_lambda"
87+
- name: Generate coverage XML (Python 3.6)
88+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
89+
run: |
90+
export COVERAGE_RCFILE=.coveragerc36
91+
coverage combine .coverage-sentry-*
92+
coverage xml --ignore-errors
8793
- name: Generate coverage XML
88-
if: ${{ !cancelled() }}
94+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
8995
run: |
90-
coverage combine .coverage*
91-
coverage xml -i
96+
coverage combine .coverage-sentry-*
97+
coverage xml
9298
- name: Upload coverage to Codecov
9399
if: ${{ !cancelled() }}
94100
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-cloud-computing.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -61,11 +61,17 @@ jobs:
6161
run: |
6262
set -x # print commands that are executed
6363
./scripts/runtox.sh "py${{ matrix.python-version }}-gcp-latest"
64+
- name: Generate coverage XML (Python 3.6)
65+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
66+
run: |
67+
export COVERAGE_RCFILE=.coveragerc36
68+
coverage combine .coverage-sentry-*
69+
coverage xml --ignore-errors
6470
- name: Generate coverage XML
65-
if: ${{ !cancelled() }}
71+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
6672
run: |
67-
coverage combine .coverage*
68-
coverage xml -i
73+
coverage combine .coverage-sentry-*
74+
coverage xml
6975
- name: Upload coverage to Codecov
7076
if: ${{ !cancelled() }}
7177
uses: codecov/[email protected]
@@ -119,11 +125,17 @@ jobs:
119125
run: |
120126
set -x # print commands that are executed
121127
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-gcp"
128+
- name: Generate coverage XML (Python 3.6)
129+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
130+
run: |
131+
export COVERAGE_RCFILE=.coveragerc36
132+
coverage combine .coverage-sentry-*
133+
coverage xml --ignore-errors
122134
- name: Generate coverage XML
123-
if: ${{ !cancelled() }}
135+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
124136
run: |
125-
coverage combine .coverage*
126-
coverage xml -i
137+
coverage combine .coverage-sentry-*
138+
coverage xml
127139
- name: Upload coverage to Codecov
128140
if: ${{ !cancelled() }}
129141
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-common.yml

+9-3
Original file line numberDiff line numberDiff line change
@@ -49,11 +49,17 @@ jobs:
4949
run: |
5050
set -x # print commands that are executed
5151
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-common"
52+
- name: Generate coverage XML (Python 3.6)
53+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
54+
run: |
55+
export COVERAGE_RCFILE=.coveragerc36
56+
coverage combine .coverage-sentry-*
57+
coverage xml --ignore-errors
5258
- name: Generate coverage XML
53-
if: ${{ !cancelled() }}
59+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
5460
run: |
55-
coverage combine .coverage*
56-
coverage xml -i
61+
coverage combine .coverage-sentry-*
62+
coverage xml
5763
- name: Upload coverage to Codecov
5864
if: ${{ !cancelled() }}
5965
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-data-processing.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -79,11 +79,17 @@ jobs:
7979
run: |
8080
set -x # print commands that are executed
8181
./scripts/runtox.sh "py${{ matrix.python-version }}-spark-latest"
82+
- name: Generate coverage XML (Python 3.6)
83+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
84+
run: |
85+
export COVERAGE_RCFILE=.coveragerc36
86+
coverage combine .coverage-sentry-*
87+
coverage xml --ignore-errors
8288
- name: Generate coverage XML
83-
if: ${{ !cancelled() }}
89+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
8490
run: |
85-
coverage combine .coverage*
86-
coverage xml -i
91+
coverage combine .coverage-sentry-*
92+
coverage xml
8793
- name: Upload coverage to Codecov
8894
if: ${{ !cancelled() }}
8995
uses: codecov/[email protected]
@@ -155,11 +161,17 @@ jobs:
155161
run: |
156162
set -x # print commands that are executed
157163
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-spark"
164+
- name: Generate coverage XML (Python 3.6)
165+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
166+
run: |
167+
export COVERAGE_RCFILE=.coveragerc36
168+
coverage combine .coverage-sentry-*
169+
coverage xml --ignore-errors
158170
- name: Generate coverage XML
159-
if: ${{ !cancelled() }}
171+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
160172
run: |
161-
coverage combine .coverage*
162-
coverage xml -i
173+
coverage combine .coverage-sentry-*
174+
coverage xml
163175
- name: Upload coverage to Codecov
164176
if: ${{ !cancelled() }}
165177
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-databases.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -88,11 +88,17 @@ jobs:
8888
run: |
8989
set -x # print commands that are executed
9090
./scripts/runtox.sh "py${{ matrix.python-version }}-sqlalchemy-latest"
91+
- name: Generate coverage XML (Python 3.6)
92+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
93+
run: |
94+
export COVERAGE_RCFILE=.coveragerc36
95+
coverage combine .coverage-sentry-*
96+
coverage xml --ignore-errors
9197
- name: Generate coverage XML
92-
if: ${{ !cancelled() }}
98+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
9399
run: |
94-
coverage combine .coverage*
95-
coverage xml -i
100+
coverage combine .coverage-sentry-*
101+
coverage xml
96102
- name: Upload coverage to Codecov
97103
if: ${{ !cancelled() }}
98104
uses: codecov/[email protected]
@@ -173,11 +179,17 @@ jobs:
173179
run: |
174180
set -x # print commands that are executed
175181
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-sqlalchemy"
182+
- name: Generate coverage XML (Python 3.6)
183+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
184+
run: |
185+
export COVERAGE_RCFILE=.coveragerc36
186+
coverage combine .coverage-sentry-*
187+
coverage xml --ignore-errors
176188
- name: Generate coverage XML
177-
if: ${{ !cancelled() }}
189+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
178190
run: |
179-
coverage combine .coverage*
180-
coverage xml -i
191+
coverage combine .coverage-sentry-*
192+
coverage xml
181193
- name: Upload coverage to Codecov
182194
if: ${{ !cancelled() }}
183195
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-graphql.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -61,11 +61,17 @@ jobs:
6161
run: |
6262
set -x # print commands that are executed
6363
./scripts/runtox.sh "py${{ matrix.python-version }}-strawberry-latest"
64+
- name: Generate coverage XML (Python 3.6)
65+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
66+
run: |
67+
export COVERAGE_RCFILE=.coveragerc36
68+
coverage combine .coverage-sentry-*
69+
coverage xml --ignore-errors
6470
- name: Generate coverage XML
65-
if: ${{ !cancelled() }}
71+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
6672
run: |
67-
coverage combine .coverage*
68-
coverage xml -i
73+
coverage combine .coverage-sentry-*
74+
coverage xml
6975
- name: Upload coverage to Codecov
7076
if: ${{ !cancelled() }}
7177
uses: codecov/[email protected]
@@ -119,11 +125,17 @@ jobs:
119125
run: |
120126
set -x # print commands that are executed
121127
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-strawberry"
128+
- name: Generate coverage XML (Python 3.6)
129+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
130+
run: |
131+
export COVERAGE_RCFILE=.coveragerc36
132+
coverage combine .coverage-sentry-*
133+
coverage xml --ignore-errors
122134
- name: Generate coverage XML
123-
if: ${{ !cancelled() }}
135+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
124136
run: |
125-
coverage combine .coverage*
126-
coverage xml -i
137+
coverage combine .coverage-sentry-*
138+
coverage xml
127139
- name: Upload coverage to Codecov
128140
if: ${{ !cancelled() }}
129141
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-miscellaneous.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -65,11 +65,17 @@ jobs:
6565
run: |
6666
set -x # print commands that are executed
6767
./scripts/runtox.sh "py${{ matrix.python-version }}-trytond-latest"
68+
- name: Generate coverage XML (Python 3.6)
69+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
70+
run: |
71+
export COVERAGE_RCFILE=.coveragerc36
72+
coverage combine .coverage-sentry-*
73+
coverage xml --ignore-errors
6874
- name: Generate coverage XML
69-
if: ${{ !cancelled() }}
75+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
7076
run: |
71-
coverage combine .coverage*
72-
coverage xml -i
77+
coverage combine .coverage-sentry-*
78+
coverage xml
7379
- name: Upload coverage to Codecov
7480
if: ${{ !cancelled() }}
7581
uses: codecov/[email protected]
@@ -127,11 +133,17 @@ jobs:
127133
run: |
128134
set -x # print commands that are executed
129135
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-trytond"
136+
- name: Generate coverage XML (Python 3.6)
137+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
138+
run: |
139+
export COVERAGE_RCFILE=.coveragerc36
140+
coverage combine .coverage-sentry-*
141+
coverage xml --ignore-errors
130142
- name: Generate coverage XML
131-
if: ${{ !cancelled() }}
143+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
132144
run: |
133-
coverage combine .coverage*
134-
coverage xml -i
145+
coverage combine .coverage-sentry-*
146+
coverage xml
135147
- name: Upload coverage to Codecov
136148
if: ${{ !cancelled() }}
137149
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-networking.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -61,11 +61,17 @@ jobs:
6161
run: |
6262
set -x # print commands that are executed
6363
./scripts/runtox.sh "py${{ matrix.python-version }}-requests-latest"
64+
- name: Generate coverage XML (Python 3.6)
65+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
66+
run: |
67+
export COVERAGE_RCFILE=.coveragerc36
68+
coverage combine .coverage-sentry-*
69+
coverage xml --ignore-errors
6470
- name: Generate coverage XML
65-
if: ${{ !cancelled() }}
71+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
6672
run: |
67-
coverage combine .coverage*
68-
coverage xml -i
73+
coverage combine .coverage-sentry-*
74+
coverage xml
6975
- name: Upload coverage to Codecov
7076
if: ${{ !cancelled() }}
7177
uses: codecov/[email protected]
@@ -119,11 +125,17 @@ jobs:
119125
run: |
120126
set -x # print commands that are executed
121127
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-requests"
128+
- name: Generate coverage XML (Python 3.6)
129+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
130+
run: |
131+
export COVERAGE_RCFILE=.coveragerc36
132+
coverage combine .coverage-sentry-*
133+
coverage xml --ignore-errors
122134
- name: Generate coverage XML
123-
if: ${{ !cancelled() }}
135+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
124136
run: |
125-
coverage combine .coverage*
126-
coverage xml -i
137+
coverage combine .coverage-sentry-*
138+
coverage xml
127139
- name: Upload coverage to Codecov
128140
if: ${{ !cancelled() }}
129141
uses: codecov/[email protected]

Diff for: ‎.github/workflows/test-integrations-web-frameworks-1.yml

+18-6
Original file line numberDiff line numberDiff line change
@@ -79,11 +79,17 @@ jobs:
7979
run: |
8080
set -x # print commands that are executed
8181
./scripts/runtox.sh "py${{ matrix.python-version }}-fastapi-latest"
82+
- name: Generate coverage XML (Python 3.6)
83+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
84+
run: |
85+
export COVERAGE_RCFILE=.coveragerc36
86+
coverage combine .coverage-sentry-*
87+
coverage xml --ignore-errors
8288
- name: Generate coverage XML
83-
if: ${{ !cancelled() }}
89+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
8490
run: |
85-
coverage combine .coverage*
86-
coverage xml -i
91+
coverage combine .coverage-sentry-*
92+
coverage xml
8793
- name: Upload coverage to Codecov
8894
if: ${{ !cancelled() }}
8995
uses: codecov/[email protected]
@@ -155,11 +161,17 @@ jobs:
155161
run: |
156162
set -x # print commands that are executed
157163
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-fastapi"
164+
- name: Generate coverage XML (Python 3.6)
165+
if: ${{ !cancelled() && matrix.python-version == '3.6' }}
166+
run: |
167+
export COVERAGE_RCFILE=.coveragerc36
168+
coverage combine .coverage-sentry-*
169+
coverage xml --ignore-errors
158170
- name: Generate coverage XML
159-
if: ${{ !cancelled() }}
171+
if: ${{ !cancelled() && matrix.python-version != '3.6' }}
160172
run: |
161-
coverage combine .coverage*
162-
coverage xml -i
173+
coverage combine .coverage-sentry-*
174+
coverage xml
163175
- name: Upload coverage to Codecov
164176
if: ${{ !cancelled() }}
165177
uses: codecov/[email protected]

0 commit comments

Comments
 (0)