Skip to content

Commit e405cd9

Browse files
committed
Add tutorial about flaky tests
1 parent 0378714 commit e405cd9

File tree

1 file changed

+46
-4
lines changed

1 file changed

+46
-4
lines changed

_overviews/toolkit/munit-flakiness.md

Lines changed: 46 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,57 @@
11
---
22
title: How to deal with flaky tests?
33
type: section
4-
description: ???
4+
description: Describe the flaky tag in MUnit.
55
num: 10
66
previous-page: munit-resources
77
next-page: munit-assumptions
88
---
99

1010
{% include markdown.html path="_markdown/install-munit.md" %}
1111

12-
<!-- Tutorial about the `.flaky` tag and how to configure the `MUNIT_FLAKY_OK` environment variable in the CI.
12+
A flaky test is a test that fails randomly depending on some outside factor: network issue, concurrency issue, high memory or CPU usage.
13+
The problem with flaky test is that it makes your CI fails regardless of the actual changes in the code.
1314

14-
See https://scalameta.org/munit/docs/tests.html#tag-flaky-tests
15-
-->
15+
As a first step you should try to fix the flaky test.
16+
However, it is sometimes hard to stabilize a test, and yet it is better to keep it than to remove it.
17+
18+
In MUnit you can flag a flaky test with `.flaky`.
19+
20+
{% tabs 'flaky-1' class=tabs-scala-version %}
21+
{% tab 'Scala 2' %}
22+
```scala
23+
class MyTests extends munit.FunSuite {
24+
test("send some request".flaky) {
25+
// body of the test
26+
}
27+
}
28+
```
29+
{% endtab %}
30+
{% tab 'Scala 3' %}
31+
```scala
32+
class MyTests extends munit.FunSuite:
33+
test("send some request".flaky) {
34+
// body of the test
35+
}
36+
```
37+
{% endtab %}
38+
{% endtabs %}
39+
40+
41+
In your CI you can set the `MUNIT_FLAKY_OK` environment variable to true to ignore the failure of the flaky tests.
42+
43+
{% tabs 'flaky-2' class=tabs-ci %}
44+
{% tab 'Github Action' %}
45+
In a Github action this is how you can set `MUNIT_FLAKY_OK`.
46+
47+
```yaml
48+
env:
49+
MUNIT_FLAKY_OK: true
50+
```
51+
{% endtab %}
52+
{% endtabs %}
53+
54+
The CI will pass even if the flaky test fails.
55+
56+
Beware not to abuse the flaky tag, as it can make your tests fail silently because of a regression in your code.
57+
To avoid this you should, once in while, run the flaky tests locally, or without the `MUNIT_FLAKY_OK`.

0 commit comments

Comments
 (0)