Skip to content

Add parameterized tests #13618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 29, 2024
Merged

Add parameterized tests #13618

merged 8 commits into from
May 29, 2024

Conversation

josevalim
Copy link
Member

@josevalim josevalim commented May 29, 2024

Sometimes you want to run the same tests but with different parameters.
In ExUnit, it is possible to do so by passing a :parameterize key to
ExUnit.Case. The value must be a list of maps which will be the
parameters merged into the test context.

For example, Elixir has a module called Registry, which can have type
:unique or :duplicate, and can control its concurrency factor using
the :partitions option. If you have a number of tests that behave the
same
across all of those values, you can parameterize those tests with:

use ExUnit.Case,
  async: true,
  parameterize:
    for(kind <- [:unique, :duplicate],
        partitions <- [1, 8],
        do: %{kind: kind, partitions: partitions})

Then, in your tests, you can access the parameters as part of the context:

test "starts a registry", %{kind: kind, partitions: partitions} do
  ...
end

Use parameterized tests with care:

  • Although parameterized tests run concurrently when async: true is also given,
    abuse of parameterized tests may make your test suite slower

  • If you use parameterized tests and then find yourself adding conditionals
    in your tests to deal with different parameters, then parameterized tests
    may be the wrong solution to your problem. Consider creating separated
    tests and sharing logic between them using regular functions

Sometimes you want to run the same tests but with different parameters.
In ExUnit, it is possible to do by returning a `:parameterize` key in
your `setup_all` context. The value must be a list of maps which will be
the parameters merged into the test context.

For example, Elixir has a module called `Registry`, which can have type
`:unique` or `:duplicate`, and can control its concurrency factor using
the `:partitions` option. If you have a number of tests that *behave the
same* across all of those values, I can parameterize those tests with:

    setup_all do
      parameters =
        for kind <- [:unique, :duplicate],
            partitions <- [1, 8],
            do: %{kind: kind, partitions: partitions}

      [parameterize: parameters]
    end

Look at the changes to registry_test.exs in this pull request as an
example. As a benefit, tests now run faster too, as there is less code
to compile.

Use parameterized tests with care:

* Abuse of parameterized tests may make your test suite considerably slower

* If you use parameterized tests and then find yourself adding conditionals
  in your tests to deal with different parameters, then parameterized tests
  may be the wrong solution to your problem. Consider creating separated
  tests and sharing logic between them using regular functions
@josevalim
Copy link
Member Author

When a test fails, this is what it looks like:

Screenshot 2024-05-29 at 10 31 40

Copy link
Member

@whatyouhide whatyouhide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet 🍬

@josevalim
Copy link
Member Author

I have pushed a new implementation that allows parameterized tests to run concurrently. Thank you @sorentwo and @wojtekmach for the feedback.

Comment on lines +346 to +348
{parameterize, opts} = Keyword.pop(opts, :parameterize, nil)

unless parameterize == nil or (is_list(parameterize) and Enum.all?(parameterize, &is_map/1)) do
Copy link
Contributor

@v0idpwn v0idpwn May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if truly better:

Suggested change
{parameterize, opts} = Keyword.pop(opts, :parameterize, nil)
unless parameterize == nil or (is_list(parameterize) and Enum.all?(parameterize, &is_map/1)) do
{parameterize, opts} = Keyword.pop(opts, :parameterize)
if parameterize && not (is_list(parameterize) and Enum.all?(parameterize, &is_map/1)) do

@josevalim josevalim merged commit b665ddd into main May 29, 2024
18 checks passed
@josevalim josevalim deleted the jv-parameterized-tests branch May 29, 2024 13:44
@josevalim
Copy link
Member Author

💚 💙 💜 💛 ❤️

SteffenDE added a commit to SteffenDE/elixir that referenced this pull request Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants