Adding regression_mcnc & vtr_reg_multiclock to CI

[ArashAhmadian]

Description

This is a pull request to address #1806. regression_mcnc task and vtr_reg_multiclock suite have been added to CI.

Related Issue

#1806

How Has This Been Tested?

odin_reg_basic regression test has been tested to run properly since vtr:vtr_reg_multiclock is one of the suites it runs.
vtr_reg_multiclock suite has been tested on its own to make sure it goes through the flow.
regression_mcncsuite has been tested on its own to make it goes through the flow.

Types of changes

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation
• I have updated the documentation accordingly
• I have added tests to cover my changes
• All new and existing tests passed

[ArashAhmadian]

@vaughnbetz I believe that odin_reg_basic also runs run_vtr_task.py on the tasklist provided in light_suite under ODIN-II regression tests, after the Odin specific tests are done. So the tasks in vtr_reg_multiclock are already tested. In that sense, putting them under another regression would be redundant. However, since they take ~4-5 mins in total to run serially; so putting them under vtr_reg_nightly_test2 just to organize all of them isn't terrible. Let me know your thoughts.

[ArashAhmadian]

Aside from that, the PR is ready for review.

[vaughnbetz]

Thanks Arash. I wouldn't test them twice -- it will be confusing to have redundant failures. If they are tested in a roundabount way by Odin, please add a comment to the config file or task list or a README in the proper directory explaining that (since this was not obvious at all!).

[ArashAhmadian]

Thanks Arash. I wouldn't test them twice -- it will be confusing to have redundant failures. If they are tested in a roundabount way by Odin, please add a comment to the config file or task list or a README in the proper directory explaining that (since this was not obvious at all!).

Makes sense. I've made the changes; the PR is ready for review @vaughnbetz

[ArashAhmadian]

For some reason, the basic sanitizer test isn't using up 14x memory anymore :))

[vaughnbetz]

@ArashAhmadian : looks good, but can you also update the golden results for this regtest so we don't fail on the high peak_vpr_memory (14x what we expect). Hopefully we'll find the root cause of that high memory, but in the interim we should update the golden results for this regtest so we don't get spurious failures. I'm assuming this regtest has its own golden result file (i.e. not shared with some runs that don't use sanitization); if not we'll need to make one.

[ArashAhmadian]

@ArashAhmadian : looks good, but can you also update the golden results for this regtest so we don't fail on the high peak_vpr_memory (14x what we expect). Hopefully we'll find the root cause of that high memory, but in the interim we should update the golden results for this regtest so we don't get spurious failures. I'm assuming this regtest has its own golden result file (i.e. not shared with some runs that don't use sanitization); if not we'll need to make one.

Sure, I have updated the golden_results for vtr_reg_multiclock and ODIN-II Basic Tests passes QoR now. @vaughnbetz

[ArashAhmadian]

However, for other PRs like #1810, #1805 and others there is an OSError after Odin tests finish and vtr regression testing is starts:

==========================================================================
                  Verilog-to-Routing Regression Testing
==========================================================================
Traceback (most recent call last):
  File "run_reg_test.py", line 342, in <module>
    vtr_command_main(sys.argv[1:])
  File "run_reg_test.py", line 179, in vtr_command_main
    vtr_task_list_files = collect_task_list(reg_test)
  File "run_reg_test.py", line 300, in collect_task_list
    raise IOError("Test does not exist: {}".format(reg_test))
OSError: Test does not exist: ../../../ODIN_II/regression_test/run001/vtr
ran test in: 0:23:47.587
no run failure!
PASSED test 'odin_reg_basic'

So the actual QoR results are not outputted. The test is still passes though which I think should be fixed after the correction @sdamghan made to verify_odin.sh unless I am misunderstanding something. Is this what's expected @sdamghan?

[sdamghan]

@ArashAhmadian Yes, once this PR is merged into the master branch, this issue will be solved. The error actually says the vtr_reg_multiclock does not exist in regression_tests. After merging, the error will be disappeared since you added the test. With the commit I pushed in this PR, the vtr_reg_multiclock should pass (as it does). The CI test will also fail if vtr_reg_multiclock fails in Odin reg basic.

[ArashAhmadian]

@ArashAhmadian Yes, once this PR is merged into the master branch, this issue will be solved. The error actually says the vtr_reg_multiclock does not exist in regression_tests. After merging, the error will be disappeared since you added the test. With the commit I pushed in this PR, the vtr_reg_multiclock should pass (as it does). The CI test will also fail if vtr_reg_multiclock fails in Odin reg basic.

Alrighty, I just wanted to make sure. Thanks for the explanation :) ! @sdamghan