Skip to content

Commit 5509cc1

Browse files
author
amin1377
committed
Add some additional instructions on how to re-run CI tests
1 parent 623673b commit 5509cc1

File tree

3 files changed

+24
-8
lines changed

3 files changed

+24
-8
lines changed

README.developers.md

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ There are 4 main regression testing suites:
199199
**Architectures:** A few simple architectures
200200

201201
This regression test is *not* suitable for evaluating QoR or performance.
202-
It's primary purpose is to make sure the various tools do not crash/fail in the basic VTR flow.
202+
Its primary purpose is to make sure the various tools do not crash/fail in the basic VTR flow.
203203

204204
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
205205
Occasionally, code changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
@@ -217,7 +217,7 @@ Occasionally, code changes can cause QoR failures (e.g. due to CAD noise -- part
217217
**Architectures:** A variety of architectures, including special architectures to exercise specific features
218218

219219
This regression test is *not* suitable for evaluating QoR or performance.
220-
It's primary purpose is try and achieve high functionality coverage.
220+
Its primary purpose is try and achieve high functionality coverage.
221221

222222
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
223223
Occasionally, changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
@@ -564,6 +564,10 @@ You need at least two sets of QoR measurements:
564564
1. The baseline QoR (i.e. unmodified VTR).
565565
2. The modified QoR (i.e. VTR with your changes).
566566

567+
The following tests can be run locally by running the given commands on the local machine. In addition, since CI tests are run whenever
568+
changes are pushed to the remote repository, one can use the CI test results to measure the impact
569+
of his/her changes. The instructions to gather CI tests' results are [here](./README.developers.md#Example:-CI-Tests-QoR-Measurement).
570+
567571
Note that it is important to generate both sets of QoR measurements on the same computing infrastructure to ensure a fair run-time comparison.
568572

569573
The following examples show how a single set of QoR measurements can be produced using the VTR flow infrastructure.
@@ -696,15 +700,27 @@ This would download the results for all CI tests.
696700
3. Scroll down and download "artifact"
697701
![Artifact](./doc/src/dev/eval_qor/artifact.png)
698702

699-
Assume that we want to get the QoR results for "vtr_reg_nightly_test3". In artifact, there is a file named
700-
"qor_results_vtr_reg_nightly_test3.tar.gz." After unzipping this file, a new directory named "vtr_flow" would be
701-
created. Go to "vtr_flow/tasks/regression_tests/vtr_reg_nightly_test3." In this directory, you can find the **parsed**
702-
results of this test.
703-
704-
In the case that you want to re-run the CI tests, go to the "Action" tab and find your workflow under Test Workflow.
703+
Assume that we want to get the QoR results for "vtr_reg_nightly_test3". In the artifact, there is a file named
704+
"qor_results_vtr_reg_nightly_test3.tar.gz." Unzip this file, and a new directory named "vtr_flow" is created. Go to
705+
"vtr_flow/tasks/regression_tests/vtr_reg_nightly_test3." In this directory, you can find a directory for each test circuit
706+
containing in this test suit(vtr_reg_nightly_test3.) For instance, results related to *vtr_reg_qor* are located in
707+
"vtr_flow/tasks/regression_tests/vtr_reg_nightly_test3/vtr_reg_qor." In this directory, results for each run are stored separately.
708+
For example, if you want to get the results for the first run, there is a file in *run001* directory named *parse_results.txt*.
709+
Using these parsed results, you can do a detailed QoR comparison using the instructions given [here](./README.developers.md#Comparing QoR Measurements).
710+
![Parse File Dir](./doc/src/dev/eval_qor/parse_result_dir.png)
711+
712+
#### Re-run CI Tests
713+
In the case that you want to re-run the CI tests, due to certain issues such as infrastructure failure,
714+
go to the "Action" tab and find your workflow under Test Workflow.
705715
Select the test which you want to re-run. There is a re-run button on the top-right corner of the newly appeared window.
706716
![Rerun CI Test](./doc/src/dev/eval_qor/re_run_tests.png)
707717

718+
** **Attention** ** If the previous run is not finished, you will not be able to re-run the CI tests. To circumvent this limitation,
719+
there are two options:
720+
1. Cancel the workflow. After a few minutes, you would be able to re-run the workflow
721+
![Rerun CI Test](./doc/src/dev/eval_qor/cancel_workflow.png)
722+
2. Wait until the workflow finishes, then re-run the failed jobs
723+
708724

709725

710726
## Comparing QoR Measurements
95.2 KB
Loading
108 KB
Loading

0 commit comments

Comments
 (0)