Skip to content

Commit ef23d52

Browse files
lucasssvazlucasssvaz
authored
docs(contrib): Add instructions on how to test changes (#10221)
* docs(contrib): Add instructions on how to test changes * fix(path): Fix file path * Remove manual hooks as they do not exist anymore in the repo * Fix typo
1 parent cbe0f2f commit ef23d52

File tree

2 files changed

+289
-23
lines changed

2 files changed

+289
-23
lines changed

Diff for: ‎docs/en/contributing.rst

+288-22
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,8 @@ Before Contributing
1515

1616
Before sending us a Pull Request, please consider this:
1717

18-
* Is the contribution entirely your own work, or is it already licensed under an LGPL 2.1 compatible Open Source License? If not, cannot accept it.
18+
* Is the contribution entirely your own work, or is it already licensed under an LGPL 2.1 compatible Open Source License?
19+
If not, cannot accept it.
1920

2021
* Is the code adequately commented and can people understand how it is structured?
2122

@@ -25,9 +26,10 @@ Before sending us a Pull Request, please consider this:
2526

2627
* Example contributions are also welcome.
2728

28-
* If you are contributing by adding a new example, please use the `Arduino style guide`_ and the example guideline below.
29+
* If you are contributing by adding a new example, please use the `Arduino style guide`_ and the example guideline below.
2930

30-
* If the contribution contains multiple commits, are they grouped together into logical changes (one major change per pull request)? Are any commits with names like "fixed typo" `squashed into previous commits <https://eli.thegreenplace.net/2014/02/19/squashing-github-pull-requests-into-a-single-commit/>`_?
31+
* If the contribution contains multiple commits, are they grouped together into logical changes (one major change per pull request)?
32+
Are any commits with names like "fixed typo" `squashed into previous commits <https://eli.thegreenplace.net/2014/02/19/squashing-github-pull-requests-into-a-single-commit/>`_?
3133

3234
If you're unsure about any of these points, please open the Pull Request anyhow and then ask us for feedback.
3335

@@ -49,20 +51,22 @@ Checklist
4951
* Check if your example proposal has no similarities to the project (**already existing examples**)
5052
* Use the `Arduino style guide`_
5153
* Add the header to all source files
52-
* Add the `README.md` file
54+
* Add the ``README.md`` file
5355
* Add inline comments if needed
5456
* Test the example
5557

5658
Header
5759
******
5860

59-
All the source files must include the header with the example name and license, if applicable. You can change this header as you wish, but it will be reviewed by the community and may not be accepted.
61+
All the source files must include the header with the example name and license, if applicable. You can change this header as you wish,
62+
but it will be reviewed by the community and may not be accepted.
6063

61-
Ideally, you can add some description about the example, links to the documentation, or the author's name. Just have in mind to keep it simple and short.
64+
Ideally, you can add some description about the example, links to the documentation, or the author's name.
65+
Just have in mind to keep it simple and short.
6266

6367
**Header Example**
6468

65-
.. code-block:: arduino
69+
.. code-block:: arduino
6670
6771
/* Wi-Fi FTM Initiator Arduino Example
6872
@@ -77,9 +81,9 @@ Ideally, you can add some description about the example, links to the documentat
7781
README file
7882
***********
7983

80-
The **README.md** file should contain the example details.
84+
The ``README.md`` file should contain the example details.
8185

82-
Please see the recommended **README.md** file in the `example template folder <https://github.com/espressif/arduino-esp32/tree/master/libraries/ESP32/examples/Template/ExampleTemplate>`_.
86+
Please see the recommended ``README.md`` file in the `example template folder <https://github.com/espressif/arduino-esp32/tree/master/libraries/ESP32/examples/Template/ExampleTemplate>`_.
8387

8488
Inline Comments
8589
***************
@@ -88,37 +92,50 @@ Inline comments are important if the example contains complex algorithms or spec
8892

8993
Brief and clear inline comments are really helpful for the example understanding and it's fast usage.
9094

91-
**Example**
95+
See the `FTM example <https://github.com/espressif/arduino-esp32/blob/master/libraries/WiFi/examples/FTM/FTM_Initiator/FTM_Initiator.ino>`_
96+
as a reference:
9297

93-
See the `FTM example <https://github.com/espressif/arduino-esp32/blob/master/libraries/WiFi/examples/FTM/FTM_Initiator/FTM_Initiator.ino>`_ as a reference.
94-
95-
.. code-block:: arduino
98+
.. code-block:: arduino
9699
97100
// Number of FTM frames requested in terms of 4 or 8 bursts (allowed values - 0 (No pref), 16, 24, 32, 64)
98101
99-
and
102+
Also:
100103

101-
.. code-block:: arduino
104+
.. code-block:: arduino
102105
103106
const char * WIFI_FTM_SSID = "WiFi_FTM_Responder"; // SSID of AP that has FTM Enabled
104107
const char * WIFI_FTM_PASS = "ftm_responder"; // STA Password
105108
106109
Testing
107110
*******
108111

109-
Be sure you have tested the example in all the supported targets. If the example works only with specific targets, add this information in the **README.md** file on the **Supported Targets** and in the example code as an inline comment.
112+
Be sure you have tested the example in all the supported targets. If the example works only with specific targets,
113+
edit/add the ``ci.json`` in the same folder as the sketch to specify the supported targets. By default,
114+
all targets are assumed to be supported.
115+
116+
Here is an example of the ``ci.json`` file where the example does not support ESP32-H2 and ESP32-S2:
117+
118+
.. code-block:: json
110119
111-
**Example**
120+
{
121+
"targets": {
122+
"esp32h2": false,
123+
"esp32s2": false
124+
}
125+
}
112126
113-
.. code-block:: arduino
127+
You also need to add this information in the ``README.md`` file, on the **Supported Targets**, and in the example code as an inline comment.
128+
For example, in the sketch:
129+
130+
.. code-block:: arduino
114131
115132
/*
116133
THIS FEATURE IS SUPPORTED ONLY BY ESP32-S2 AND ESP32-C3
117134
*/
118135
119-
and
136+
And in the ``README.md`` file:
120137

121-
.. code-block:: markdown
138+
.. code-block:: markdown
122139
123140
Currently, this example supports the following targets.
124141
@@ -128,11 +145,260 @@ and
128145
Example Template
129146
****************
130147

131-
The example template can be found `here <https://github.com/espressif/arduino-esp32/tree/master/libraries/ESP32/examples/Template/ExampleTemplate>`_ and can be used as a reference.
148+
The example template can be found `here <https://github.com/espressif/arduino-esp32/tree/master/libraries/ESP32/examples/Template/ExampleTemplate>`_
149+
and can be used as a reference.
150+
151+
Documentation
152+
-------------
153+
154+
If you are contributing to the documentation, please follow the instructions described in the
155+
`documentation guidelines <guides/docs_contributing>`_ to properly format and test your changes.
156+
157+
Testing and CI
158+
--------------
159+
160+
After you have made your changes, you should test them. You can do this in different ways depending on the type of change you have made.
161+
162+
Examples
163+
********
164+
165+
The easiest way to test an example is to load it into the Arduino IDE and run it on your board. If you have multiple boards,
166+
you should test it on all of them to ensure that the example works on all supported targets.
167+
168+
You can refer to the `Example Contribution Guideline`_ section for more information on how to write and test examples.
169+
170+
Library Changes
171+
***************
172+
173+
If you have made changes to a library, you should test it on all supported targets. You can do this by loading the library examples (or creating new ones)
174+
into the Arduino IDE and running them on your board. If you have multiple boards, you should test it on all of them to ensure that the library
175+
works as expected on all targets.
176+
177+
You can also add a new test suite to automatically check the library. You can refer to the `Adding a New Test Suite`_ section for more information.
178+
179+
Core Changes
180+
************
181+
182+
If you have made changes to the core, it is important to ensure that the changes do not break the existing functionality. You can do this by running the
183+
tests on all supported targets. You can refer to the `Runtime Tests`_ section for more information.
184+
185+
CI
186+
**
187+
188+
In our repository, we have a Continuous Integration (CI) system that runs tests and fixes on every Pull Request. This system will run the tests
189+
on all supported targets and check if everything is working as expected.
190+
191+
We have many types of tests and checks, including:
192+
193+
* Compilation tests;
194+
* Runtime tests;
195+
* Documentation checks;
196+
* Code style checks;
197+
* And more.
198+
199+
Let's go deeper into each type of test in the CI system:
200+
201+
Compilation Tests
202+
^^^^^^^^^^^^^^^^^
203+
204+
The compilation tests are the first type of tests in the CI system. They check if the code compiles on all supported targets.
205+
If the code does not compile, the CI system will fail the test and the Pull Request will not be merged.
206+
This is important to ensure that the code is compatible with all supported targets and no broken code is merged.
207+
208+
It will go through all the sketches in the repository and check if they compile on all supported targets. This process is automated and controlled
209+
by GitHub Actions. The CI system will run the ``arduino-cli`` tool to compile the sketches on all supported targets.
210+
211+
Testing it locally using the CI scripts would be too time consuming, so we recommend running the tests locally using the Arduino IDE with
212+
a sketch that uses the changes you made (you can also add the sketch as an example if your change is not covered by the existing ones).
213+
Make sure to set ``Compiler Warnings`` to ``All`` in the Arduino IDE to catch any potential issues.
214+
215+
Runtime Tests
216+
^^^^^^^^^^^^^
217+
218+
Another type of test is the runtime tests. They check if the code runs and behaves as expected on all supported targets. If the
219+
code does not run as expected, the CI system will fail the test and the Pull Request will not be merged. This is important to ensure that the code
220+
works as expected on all supported targets and no unintended crashes or bugs are introduced.
221+
222+
These tests are specialized sketches that run on the target board and check if the code behaves as expected. This process is automated and
223+
controlled by the ``pytest_embedded`` tool. You can read more about this tool in its
224+
`documentation <https://docs.espressif.com/projects/pytest-embedded/en/latest/>`_.
225+
226+
The tests are divided into two categories inside the ``tests`` folder:
227+
228+
* ``validation``: These tests are used to validate the behavior of the Arduino core and libraries. They are used to check if the core and libraries
229+
are working as expected;
230+
* ``performance``: These tests are used to check the performance of the Arduino core and libraries. They are used to check if the changes made
231+
to the core and libraries have any big impact on the performance. These tests usually run for a longer time than the validation tests and include
232+
common benchmark tools like `CoreMark <https://github.com/eembc/coremark>`_.
233+
234+
To run the runtime tests locally, first install the required dependencies by running:
235+
236+
.. code-block:: bash
237+
238+
pip install -U -r tests/requirements.txt
239+
240+
Before running the test, we need to build it by running:
241+
242+
.. code-block:: bash
243+
244+
./.github/scripts/tests_build.sh -s <test_name> -t <target>
245+
246+
The ``<test_name>`` is the name of the test you want to run (you can check the available tests in the ``tests/validation`` and
247+
``tests/performance`` folders), and the ``<target>`` is the target board you want to run the test on. For example, to run the ``uart`` test on the
248+
ESP32-C3 target, you would run:
249+
250+
.. code-block:: bash
251+
252+
./.github/scripts/tests_build.sh -s uart -t esp32c3
253+
254+
You should see the output of the build process and the test binary should be generated in the ``~/.arduino/tests/<test_name>/build.tmp`` folder.
255+
256+
Now that the test is built, you can run it in the target board. Connect the target board to your computer and run:
257+
258+
.. code-block:: bash
259+
260+
./.github/scripts/tests_run.sh -s <test_name> -t <target>
261+
262+
For example, to run the ``uart`` test on the ESP32-C3 target, you would run:
263+
264+
.. code-block:: bash
265+
266+
./.github/scripts/tests_run.sh -s uart -t esp32c3
267+
268+
The test will run on the target board and you should see the output of the test in the terminal:
269+
270+
.. code-block:: bash
271+
272+
lucassvaz@Lucas--MacBook-Pro esp32 % ./.github/scripts/tests_run.sh -s uart -t esp32c3
273+
Sketch uart test type: validation
274+
Running test: uart -- Config: Default
275+
pytest tests --build-dir /Users/lucassvaz/.arduino/tests/uart/build.tmp -k test_uart --junit-xml=/Users/lucassvaz/Espressif/Arduino/hardware/espressif/esp32/tests/validation/uart/esp32c3/uart.xml --embedded-services esp,arduino
276+
=============================================================================================== test session starts ================================================================================================
277+
platform darwin -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0
278+
rootdir: /Users/lucassvaz/Espressif/Arduino/hardware/espressif/esp32/tests
279+
configfile: pytest.ini
280+
plugins: cov-5.0.0, embedded-1.11.5, anyio-4.4.0
281+
collected 15 items / 14 deselected / 1 selected
282+
283+
tests/validation/uart/test_uart.py::test_uart
284+
-------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
285+
2024-08-22 11:49:30 INFO Target: esp32c3, Port: /dev/cu.usbserial-2120
286+
PASSED [100%]
287+
------------------------------------------------------------------------------------------------ live log teardown -------------------------------------------------------------------------------------------------
288+
2024-08-22 11:49:52 INFO Created unity output junit report: /private/var/folders/vp/g9wctsvn7b91k3pv_7cwpt_h0000gn/T/pytest-embedded/2024-08-22_14-49-30-392993/test_uart/dut.xml
289+
290+
291+
---------------------------------------------- generated xml file: /Users/lucassvaz/Espressif/Arduino/hardware/espressif/esp32/tests/validation/uart/esp32c3/uart.xml ----------------------------------------------
292+
======================================================================================== 1 passed, 14 deselected in 22.18s =========================================================================================
293+
294+
After the test is finished, you can check the output in the terminal and the generated XML file in the test folder.
295+
Additionally, for performance tests, you can check the generated JSON file in the same folder.
296+
297+
You can also run the tests in `Wokwi <https://docs.wokwi.com/>`_ or `Espressif's QEMU <https://github.com/espressif/esp-toolchain-docs/tree/main/qemu>`_
298+
by using the ``-W <timeout_in_ms>`` and ``-Q`` flags respectively. You will need to have the Wokwi and/or QEMU installed in your system
299+
and set the ``WOKWI_CLI_TOKEN`` and/or ``QEMU_PATH`` environment variables. The ``WOKWI_CLI_TOKEN`` is the CI token that can be obtained from the
300+
`Wokwi website <https://wokwi.com/dashboard/ci>`_ and the ``QEMU_PATH`` is the path to the QEMU binary.
301+
302+
For example, to run the ``uart`` test using Wokwi, you would run:
303+
304+
.. code-block:: bash
305+
306+
WOKWI_CLI_TOKEN=<your_wokwi_token> ./.github/scripts/tests_run.sh -s uart -t esp32c3 -W <timeout_in_ms>
307+
308+
And to run the ``uart`` test using QEMU, you would run:
309+
310+
.. code-block:: bash
311+
312+
QEMU_PATH=<path_to_qemu_binary> ./.github/scripts/tests_run.sh -s uart -t esp32c3 -Q
313+
314+
.. note::
315+
316+
Not all tests are supported by Wokwi and QEMU. QEMU is only supported for ESP32 and ESP32-C3 targets.
317+
Wokwi support depends on the `currently implemented peripherals <https://docs.wokwi.com/guides/esp32#simulation-features>`_.
318+
319+
Adding a New Test Suite
320+
#######################
321+
322+
If you want to add a new test suite, you can create a new folder inside ``tests/validation`` or ``tests/performance`` with the name of the test suite.
323+
You can use the ``hello_world`` test suite as a starting point and the other test suites as a reference.
324+
325+
A test suite contains the following files:
326+
327+
* ``test_<test_name>.py``: The test file that contains the test cases. Required.
328+
* ``<test_name>.ino``: The sketch that will be tested. Required.
329+
* ``ci.json``: The file that specifies how the test suite will be run in the CI system. Optional.
330+
* ``diagram.<target>.json``: The diagram file that specifies the connections between the components in Wokwi. Optional.
331+
* ``scenario.yaml``: The scenario file that specifies how Wokwi will interact with the components. Optional.
332+
* Any other files that are needed for the test suite.
333+
334+
You can read more about the test python API in the `pytest-embedded documentation <https://docs.espressif.com/projects/pytest-embedded/en/latest/usages/expecting.html>`_.
335+
For more information about the Unity testing framework, you can check the `Unity documentation <https://github.com/ThrowTheSwitch/Unity>`_.
336+
337+
After creating the test suite, make sure to test it locally and run it in the CI system to ensure that it works as expected.
338+
339+
CI JSON File
340+
############
341+
342+
The ``ci.json`` file is used to specify how the test suite and sketches will handled by the CI system. It can contain the following fields:
343+
344+
* ``targets``: A dictionary that specifies the supported targets. The key is the target name and the value is a boolean that specifies if the
345+
target is supported. By default, all targets are assumed to be supported. This field is also valid for examples.
346+
* ``platforms``: A dictionary that specifies the supported platforms. The key is the platform name and the value is a boolean that specifies if
347+
the platform is supported. By default, all platforms are assumed to be supported.
348+
* ``extra_tags``: A list of extra tags that the runner will require when running the test suite in hardware. By default, no extra tags are required.
349+
* ``fqbn``: A dictionary that specifies the FQBNs that will be used to compile the sketch. The key is the target name and the value is a list
350+
of FQBNs. The `default FQBNs <https://github.com/espressif/arduino-esp32/blob/a31a5fca1739993173caba995f7785b8eed6b30e/.github/scripts/sketch_utils.sh#L86-L91>`_
351+
are used if this field is not specified.
352+
353+
The ``wifi`` test suite is a good example of how to use the ``ci.json`` file:
354+
355+
.. literalinclude:: ../../tests/validation/wifi/ci.json
356+
:language: json
357+
358+
Documentation Checks
359+
^^^^^^^^^^^^^^^^^^^^
360+
361+
The CI also checks the documentation for any compilation errors. This is important to ensure that the documentation layout is not broken.
362+
To build the documentation locally, please refer to the `documentation guidelines <guides/docs_contributing>`_.
363+
364+
Code Style Checks
365+
^^^^^^^^^^^^^^^^^
366+
367+
For checking the code style and other code quality checks, we use pre-commit hooks.
368+
These hooks will be automatically run by the CI when a Pull Request is marked as ``Status: Pending Merge``.
369+
You can check which hooks are being run in the ``.pre-commit-config.yaml`` file.
370+
371+
You can read more about the pre-commit hooks in the `pre-commit documentation <https://pre-commit.com/>`_.
372+
373+
If you want to run the pre-commit hooks locally, you first need to install the required dependencies by running:
374+
375+
.. code-block:: bash
376+
377+
pip install -U -r tools/pre-commit/requirements.txt
378+
379+
Then, you can run the pre-commit hooks staging your changes and running:
380+
381+
.. code-block:: bash
382+
383+
pre-commit run
384+
385+
To run a specific hook, you can use the hook name as an argument. For example, to run the ``codespell`` hook, you would run:
386+
387+
.. code-block:: bash
388+
389+
pre-commit run codespell
390+
391+
If you want to run the pre-commit hooks automatically against the changed files on every ``git commit``,
392+
you can install the pre-commit hooks by running:
393+
394+
.. code-block:: bash
395+
396+
pre-commit install
132397
133398
Legal Part
134399
----------
135400

136-
Before a contribution can be accepted, you will need to sign our contributor agreement. You will be prompted for this automatically as part of the Pull Request process.
401+
Before a contribution can be accepted, you will need to sign our contributor agreement. You will be prompted for this automatically as part of
402+
the Pull Request process.
137403

138404
.. _Arduino style guide: https://docs.arduino.cc/learn/contributions/arduino-writing-style-guide

Diff for: ‎docs/en/guides/docs_contributing.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ If everything is ok, you will see some output logs similar to this one:
104104
dumping object inventory... done
105105
build succeeded.
106106
107-
The HTML pages are in `_build/en/generic/html`.
107+
The HTML pages are in ``_build/en/generic/html``.
108108

109109
Sections
110110
--------

0 commit comments

Comments
 (0)