|
1 | 1 | ---
|
2 | 2 | title: How to deal with flaky tests?
|
3 | 3 | type: section
|
4 |
| -description: ??? |
| 4 | +description: Describe the flaky tag in MUnit. |
5 | 5 | num: 10
|
6 | 6 | previous-page: munit-resources
|
7 | 7 | next-page: munit-assumptions
|
8 | 8 | ---
|
9 | 9 |
|
10 | 10 | {% include markdown.html path="_markdown/install-munit.md" %}
|
11 | 11 |
|
12 |
| -<!-- Tutorial about the `.flaky` tag and how to configure the `MUNIT_FLAKY_OK` environment variable in the CI. |
| 12 | +A flaky test is a test that fails randomly depending on some outside factor: network issue, concurrency issue, high memory or CPU usage, etc. |
| 13 | +Flaky tests are inconvenient because they can make your CI fail regardless of the actual changes in the code. |
13 | 14 |
|
14 |
| -See https://scalameta.org/munit/docs/tests.html#tag-flaky-tests |
15 |
| ---> |
| 15 | +Stabilizing a flaky test is sometimes hard to do. |
| 16 | +Removing the test would make it riskier to introduce bugs. |
| 17 | +Alternatively, in Munit, you can flag a test with `flaky`. |
| 18 | +It will still be run, but its result will be ignored, under some conditions. |
| 19 | + |
| 20 | +{% tabs 'flaky-1' class=tabs-scala-version %} |
| 21 | +{% tab 'Scala 2' %} |
| 22 | +```scala |
| 23 | +class MyTests extends munit.FunSuite { |
| 24 | + test("send some request".flaky) { |
| 25 | + // body of the test |
| 26 | + } |
| 27 | +} |
| 28 | +``` |
| 29 | +{% endtab %} |
| 30 | +{% tab 'Scala 3' %} |
| 31 | +```scala |
| 32 | +class MyTests extends munit.FunSuite: |
| 33 | + test("send some request".flaky) { |
| 34 | + // body of the test |
| 35 | + } |
| 36 | +``` |
| 37 | +{% endtab %} |
| 38 | +{% endtabs %} |
| 39 | + |
| 40 | + |
| 41 | +In your CI, you can set the `MUNIT_FLAKY_OK` environment variable to `true` to ignore the failure of the tests flagged with `flaky`. |
| 42 | + |
| 43 | +{% tabs 'flaky-2' class=tabs-ci %} |
| 44 | +{% tab 'Github Action' %} |
| 45 | +In a Github action this is how you can set `MUNIT_FLAKY_OK`. |
| 46 | + |
| 47 | +```yaml |
| 48 | +env: |
| 49 | + MUNIT_FLAKY_OK: true |
| 50 | +``` |
| 51 | +{% endtab %} |
| 52 | +{% endtabs %} |
| 53 | +
|
| 54 | +The CI will pass even if the flaky test fails. |
| 55 | +
|
| 56 | +Beware not to abuse the flaky tag, as it can make your tests fail silently because of a regression in your code. |
| 57 | +To avoid this you should, once in while, run the flaky tests locally, or without the `MUNIT_FLAKY_OK`. |
0 commit comments