Skip to content

Commit f995d8c

Browse files
sentrivanaszokeasaurusrex
sentrivana
and
szokeasaurusrex
authored
[1] Add tox generation script, but don't use it yet (#3971)
Add: * tox generation script * tox template * script for generating tox and CI yamls in one go * readme for the script In this PR, the script is set to ignore all integrations, so no tox configuration is actually added. However, it's still the script actually generating the real `tox.ini` from the `tox.jinja` template. See follow-up PRs for more. --------- Co-authored-by: Daniel Szoke <[email protected]>
1 parent c1cf0fe commit f995d8c

File tree

7 files changed

+1651
-0
lines changed

7 files changed

+1651
-0
lines changed

‎scripts/generate-test-files.sh

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/bin/sh
2+
3+
# This script generates tox.ini and CI YAML files in one go.
4+
5+
set -xe
6+
7+
cd "$(dirname "$0")"
8+
9+
python -m venv toxgen.venv
10+
. toxgen.venv/bin/activate
11+
12+
pip install -e ..
13+
pip install -r populate_tox/requirements.txt
14+
pip install -r split_tox_gh_actions/requirements.txt
15+
16+
python populate_tox/populate_tox.py
17+
python split_tox_gh_actions/split_tox_gh_actions.py

‎scripts/populate_tox/README.md

+159
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Populate Tox
2+
3+
We integrate with a number of frameworks and libraries and have a test suite for
4+
each. The tests run against different versions of the framework/library to make
5+
sure we support everything we claim to.
6+
7+
This `populate_tox.py` script is responsible for picking reasonable versions to
8+
test automatically and generating parts of `tox.ini` to capture this.
9+
10+
## How it works
11+
12+
There is a template in this directory called `tox.jinja` which contains a
13+
combination of hardcoded and generated entries.
14+
15+
The `populate_tox.py` script fills out the auto-generated part of that template.
16+
It does this by querying PyPI for each framework's package and its metadata and
17+
then determining which versions make sense to test to get good coverage.
18+
19+
The lowest supported and latest version of a framework are always tested, with
20+
a number of releases in between:
21+
- If the package has majors, we pick the highest version of each major. For the
22+
latest major, we also pick the lowest version in that major.
23+
- If the package doesn't have multiple majors, we pick two versions in between
24+
lowest and highest.
25+
26+
#### Caveats
27+
28+
- Make sure the integration name is the same everywhere. If it consists of
29+
multiple words, use an underscore instead of a hyphen.
30+
31+
## Defining constraints
32+
33+
The `TEST_SUITE_CONFIG` dictionary defines, for each integration test suite,
34+
the main package (framework, library) to test with; any additional test
35+
dependencies, optionally gated behind specific conditions; and optionally
36+
the Python versions to test on.
37+
38+
Constraints are defined using the format specified below. The following sections describe each key.
39+
40+
```
41+
integration_name: {
42+
"package": name_of_main_package_on_pypi,
43+
"deps": {
44+
rule1: [package1, package2, ...],
45+
rule2: [package3, package4, ...],
46+
},
47+
"python": python_version_specifier,
48+
}
49+
```
50+
51+
### `package`
52+
53+
The name of the third party package as it's listed on PyPI. The script will
54+
be picking different versions of this package to test.
55+
56+
This key is mandatory.
57+
58+
### `deps`
59+
60+
The test dependencies of the test suite. They're defined as a dictionary of
61+
`rule: [package1, package2, ...]` key-value pairs. All packages
62+
in the package list of a rule will be installed as long as the rule applies.
63+
64+
`rule`s are predefined. Each `rule` must be one of the following:
65+
- `*`: packages will be always installed
66+
- a version specifier on the main package (e.g. `<=0.32`): packages will only
67+
be installed if the main package falls into the version bounds specified
68+
- specific Python version(s) in the form `py3.8,py3.9`: packages will only be
69+
installed if the Python version matches one from the list
70+
71+
Rules can be used to specify version bounds on older versions of the main
72+
package's dependencies, for example. If e.g. Flask tests generally need
73+
Werkzeug and don't care about its version, but Flask older than 3.0 needs
74+
a specific Werkzeug version to work, you can say:
75+
76+
```python
77+
"flask": {
78+
"deps": {
79+
"*": ["Werkzeug"],
80+
"<3.0": ["Werkzeug<2.1.0"],
81+
},
82+
...
83+
}
84+
```
85+
86+
If you need to install a specific version of a secondary dependency on specific
87+
Python versions, you can say:
88+
89+
```python
90+
"celery": {
91+
"deps": {
92+
"*": ["newrelic", "redis"],
93+
"py3.7": ["importlib-metadata<5.0"],
94+
},
95+
...
96+
}
97+
```
98+
This key is optional.
99+
100+
### `python`
101+
102+
Sometimes, the whole test suite should only run on specific Python versions.
103+
This can be achieved via the `python` key, which expects a version specifier.
104+
105+
For example, if you want AIOHTTP tests to only run on Python 3.7+, you can say:
106+
107+
```python
108+
"aiohttp": {
109+
"python": ">=3.7",
110+
...
111+
}
112+
```
113+
114+
The `python` key is optional, and when possible, it should be omitted. The script
115+
should automatically detect which Python versions the package supports.
116+
However, if a package has broken
117+
metadata or the SDK is explicitly not supporting some packages on specific
118+
Python versions (because of, for example, broken context vars), the `python`
119+
key can be used.
120+
121+
122+
## How-Tos
123+
124+
### Add a new test suite
125+
126+
1. Add the minimum supported version of the framework/library to `_MIN_VERSIONS`
127+
in `integrations/__init__.py`. This should be the lowest version of the
128+
framework that we can guarantee works with the SDK. If you've just added the
129+
integration, you should generally set this to the latest version of the framework
130+
at the time.
131+
2. Add the integration and any constraints to `TEST_SUITE_CONFIG`. See the
132+
"Defining constraints" section for the format.
133+
3. Add the integration to one of the groups in the `GROUPS` dictionary in
134+
`scripts/split_tox_gh_actions/split_tox_gh_actions.py`.
135+
4. Add the `TESTPATH` for the test suite in `tox.jinja`'s `setenv` section.
136+
5. Run `scripts/generate-test-files.sh` and commit the changes.
137+
138+
### Migrate a test suite to populate_tox.py
139+
140+
A handful of integration test suites are still hardcoded. The goal is to migrate
141+
them all to `populate_tox.py` over time.
142+
143+
1. Remove the integration from the `IGNORE` list in `populate_tox.py`.
144+
2. Remove the hardcoded entries for the integration from the `envlist` and `deps` sections of `tox.jinja`.
145+
3. Run `scripts/generate-test-files.sh`.
146+
4. Run the test suite, either locally or by creating a PR.
147+
5. Address any test failures that happen.
148+
149+
You might have to introduce additional version bounds on the dependencies of the
150+
package. Try to determine the source of the failure and address it.
151+
152+
Common scenarios:
153+
- An old version of the tested package installs a dependency without defining
154+
an upper version bound on it. A new version of the dependency is installed that
155+
is incompatible with the package. In this case you need to determine which
156+
versions of the dependency don't contain the breaking change and restrict this
157+
in `TEST_SUITE_CONFIG`.
158+
- Tests are failing on an old Python version. In this case first double-check
159+
whether we were even testing them on that version in the original `tox.ini`.

‎scripts/populate_tox/config.py

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# The TEST_SUITE_CONFIG dictionary defines, for each integration test suite,
2+
# the main package (framework, library) to test with; any additional test
3+
# dependencies, optionally gated behind specific conditions; and optionally
4+
# the Python versions to test on.
5+
#
6+
# See scripts/populate_tox/README.md for more info on the format and examples.
7+
8+
TEST_SUITE_CONFIG = {}

0 commit comments

Comments
 (0)