Skip to content

Get rid of legacy tests #3896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 tasks
smarter opened this issue Jan 23, 2018 · 1 comment
Closed
2 tasks

Get rid of legacy tests #3896

smarter opened this issue Jan 23, 2018 · 1 comment

Comments

@smarter
Copy link
Member

smarter commented Jan 23, 2018

Our legacy test infrastructure is a maintenance burden and uses up resources on our CI. To get rid of it I think we need:

  • Proper integration meta-tests. We already have unit meta-tests in https://github.com/lampepfl/dotty/blob/master/compiler/test/dotty/tools/vulpix/VulpixTests.scala but these don't run vulpix end-to-end. We should set up a directory of test files designed to stress vulpix in some ways (compiler crash, infinite loop at runtime, excessive memory usage, ...) and exercise common features (proper diff between check and run files, correct number of errors reported, ...). Then vulpix should be run on this directory as it is currently being run on tests/, and its output should be compared against some expected output.
  • Optional: a test mode where the list of files passed to the compiler is sorted in a random order (controlled by a seed). We sometimes catch bugs just because the legacy tests happen to pass the files to the compiler in a different order than vulpix. We should have a way to intentionally try different orders. While it's impractical to try every possible order all the time, we could do nightly tests using a few different random orders each time.
@OlivierBlanvillain
Copy link
Contributor

About the second one, compileDir already has an optional argument randomOrder: Option[Int] to do that, maybe it should be wired to something more global and easy to change than a function argument.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants