Skip to content

Limit counterexample size to speed up regression tests #7539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 15, 2023

Conversation

tautschnig
Copy link
Collaborator

These regression tests only require 1-2 seconds of solving time, but may produce counterexample traces that are greater than 8 GB on disk. Limit the maximum string length (in none of the tests does the length even matter) to avoid such artificial blow-up. Also, these tests were possibly a lot slower with other solvers, simply because these would produce different models. Before this PR, we had:

  • assume-inputs-non-null/string_assume.desc:
    • cadical: 165 seconds
    • minisat: 0 seconds
  • context-include-exclude/test_exclude_from_include.desc:
    • cadical: 796 seconds
    • minisat: 1 seconds
  • context-include-exclude/test_include.desc:
    • cadical: 802 seconds
    • minisat: 1 seconds
  • exceptions29/test.desc:
    • cadical: 168 seconds
    • minisat: 0 seconds
      when the actual solver time was less than 2 seconds.
  • Each commit message has a non-empty body, explaining why the change was made.
  • n/a Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.
  • n/a The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/
  • Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).
  • My commit message includes data points confirming performance improvements (if claimed).
  • My PR is restricted to a single feature or bugfix.
  • n/a White-space or formatting changes outside the feature-related changed lines are in commits of their own.

These regression tests only require 1-2 seconds of solving time, but may
produce counterexample traces that are greater than 8 GB on disk. Limit
the maximum string length (in none of the tests does the length even
matter) to avoid such artificial blow-up. Also, these tests were
possibly a lot slower with other solvers, simply because these would
produce different models. Before this PR, we had:
* assume-inputs-non-null/string_assume.desc:
** cadical: 165 seconds
** minisat: 0 seconds
* context-include-exclude/test_exclude_from_include.desc:
** cadical: 796 seconds
** minisat: 1 seconds
* context-include-exclude/test_include.desc:
** cadical: 802 seconds
** minisat: 1 seconds
* exceptions29/test.desc:
** cadical: 168 seconds
** minisat: 0 seconds
when the actual solver time was less than 2 seconds.
@codecov
Copy link

codecov bot commented Feb 15, 2023

Codecov Report

Base: 78.49% // Head: 78.46% // Decreases project coverage by -0.03% ⚠️

Coverage data is based on head (d9b0744) compared to base (1396782).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #7539      +/-   ##
===========================================
- Coverage    78.49%   78.46%   -0.03%     
===========================================
  Files         1667     1667              
  Lines       191431   191431              
===========================================
- Hits        150255   150214      -41     
- Misses       41176    41217      +41     
Impacted Files Coverage Δ
src/big-int/bigint.hh 89.47% <0.00%> (-7.90%) ⬇️
src/big-int/bigint.cc 83.35% <0.00%> (-5.65%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@tautschnig tautschnig merged commit 4ba6285 into diffblue:develop Feb 15, 2023
@tautschnig tautschnig deleted the bugfixes/limit-string-size branch February 15, 2023 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants