Skip to content

Reorganizing vtr_flow/tasks #1728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

ArashAhmadian
Copy link
Contributor

Reorganized vtr-tasks by removing/moving the outdated tasks not run in CI.

Description

Deleted tasks: timing_small,timing_chain,checkin_reg,basic_flow,regression_bidir, regression_fpu_hard_block_arch,regression_fpu_soft_block_arch

Moved tasks:arithmetic_tasks,func_multiclock,power,multiclock_mcnc to vtr_reg_weekly
timing to vtr_reg_nightly,regression_mcnc to vtr_reg_basic

Motivation and Context

To make sure that all the tasks are updated by being included in the CI and deleting outdated ones.

@ArashAhmadian ArashAhmadian self-assigned this May 16, 2021
@github-actions github-actions bot added the docs Documentation label May 16, 2021
@MohamedElgammal
Copy link
Contributor

@ArashAhmadian Thanks for getting this done. I think you are missing something (maybe you forgot to compile vpr) before running the tasks and generating the golden_solution as all the runs errored out and the golden solutions are empty. Please revisit this.

@MohamedElgammal MohamedElgammal added the VTR Flow VTR Design Flow (scripts/benchmarks/architectures) label May 16, 2021
@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

Looks like nightly_test3 worked in 240m and nightly_test2 worked by took 428m and nightly_test1 failed.

@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

Looks like .github/kokoro/steps/vtr-test.sh needs to be updated to understand the three nightly output too.

@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

Error: 1622 tests failed
++ TEST_RESULT=86
++ set -e
++ kill 20438
++ echo ========================================
========================================
++ echo 'Cleaning benchmarks files'
Cleaning benchmarks files
++ echo ========================================
========================================
++ find vtr_flow/arch/symbiflow/ -type f -not -name 'README.*' -delete
/tmpfs/src/github/vtr-verilog-to-routing/.github/kokoro/steps/vtr-test.sh: line 59: 20438 Terminated              ( while :; do
    date; uptime; free -h; sleep 300;
done )
++ find vtr_flow/benchmarks/symbiflow/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/ispd_blif/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/titan_blif/ -type f -not -name 'README.*' -delete
++ find . -type f -regex '.*\.tar\.\(gz\|xz\)' -delete
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly/ -type f -print0
+++ nproc
find: ‘vtr_flow/tasks/regression_tests/vtr_reg_nightly/’: No such file or directory
++ xargs -0 -P 8 gzip
���!B�`��+++ du -sh
++ echo 'Working directory size: 25G	.'
Working directory size: 25G	.
+++ du -s
+++ cut -d '	' -f 1
+++ expr 1024 '*' 1024 '*' 90
++ [[ 25835312 -gt 94371840 ]]
++ exit 86


[ID: 1118326] Build finished after 13078 secs, exit value: 86

@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

Calculating QoR results...
regression_tests/vtr_reg_nightly_test2/vtr_reg_netlist_writer...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_func_formal...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_bidir...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_bidir...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_complex_switch...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_titan...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_error_check...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_timing_update_diff...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_timing_update_diff_titan...[Pass]
vtr_reg_nightly_test2/func_multiclock/multiclock_mcnc...[Pass]
regression_tests/vtr_reg_nightly_test2/titan_other...[Pass]
regression_tests/vtr_reg_nightly_test2/titan_quick_qor...[Pass]

Test 'vtr_reg_nightly_test2' had 0 qor test failures

Test 'vtr_reg_nightly_test2' had 0 run failures

All tests passed
++ TEST_RESULT=0
++ set -e
++ kill 20468
++ echo ========================================
========================================
++ echo 'Cleaning benchmarks files'
Cleaning benchmarks files
++ echo ========================================
========================================
++ find vtr_flow/arch/symbiflow/ -type f -not -name 'README.*' -delete
/tmpfs/src/github/vtr-verilog-to-routing/.github/kokoro/steps/vtr-test.sh: line 59: 20468 Terminated              ( while :; do
    date; uptime; free -h; sleep 300;
done )
++ find vtr_flow/benchmarks/symbiflow/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/ispd_blif/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/titan_blif/ -type f -not -name 'README.*' -delete
++ find . -type f -regex '.*\.tar\.\(gz\|xz\)' -delete
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly/ -type f -print0
find: ‘vtr_flow/tasks/regression_tests/vtr_reg_nightly/’: No such file or directory
+++ nproc
++ xargs -0 -P 8 gzip
����f�`��+++ du -sh
++ echo 'Working directory size: 77G	.'
Working directory size: 77G	.
+++ du -s
+++ cut -d '	' -f 1
+++ expr 1024 '*' 1024 '*' 90
++ [[ 79767248 -gt 94371840 ]]
++ exit 0


[ID: 1504800] Build finished after 22423 secs, exit value: 0

@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

Calculating QoR results...
regression_tests/vtr_reg_nightly_test3/vtr_reg_qor_chain...[Pass]
regression_tests/vtr_reg_nightly_test3/vtr_reg_qor_chain_depop...[Pass]
regression_tests/vtr_reg_nightly_test3/vtr_reg_qor_chain_predictor_off...[Pass]
regression_tests/vtr_reg_nightly_test3/vtr_reg_qor...[Pass]
regression_tests/vtr_reg_nightly_test3/complex_switch...[Pass]

Test 'vtr_reg_nightly_test3' had 0 qor test failures

Test 'vtr_reg_nightly_test3' had 0 run failures

All tests passed
++ TEST_RESULT=0
++ set -e
++ kill 22857
++ echo ========================================
========================================
++ echo 'Cleaning benchmarks files'
Cleaning benchmarks files
++ echo ========================================
========================================
++ find vtr_flow/arch/symbiflow/ -type f -not -name 'README.*' -delete
/tmpfs/src/github/vtr-verilog-to-routing/.github/kokoro/steps/vtr-test.sh: line 59: 22857 Terminated              ( while :; do
    date; uptime; free -h; sleep 300;
done )
++ find vtr_flow/benchmarks/symbiflow/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/ispd_blif/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/titan_blif/ -type f -not -name 'README.*' -delete
++ find . -type f -regex '.*\.tar\.\(gz\|xz\)' -delete
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly/ -type f -print0
+++ nproc
find: ‘vtr_flow/tasks/regression_tests/vtr_reg_nightly/’: No such file or directory
++ xargs -0 -P 8 gzip
����E�`��+++ du -sh
++ echo 'Working directory size: 19G	.'
Working directory size: 19G	.
+++ du -s
+++ cut -d '	' -f 1
+++ expr 1024 '*' 1024 '*' 90
++ [[ 19579428 -gt 94371840 ]]
++ exit 0


[ID: 4980023] Build finished after 14021 secs, exit value: 0

@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

Calculating QoR results...
regression_tests/vtr_reg_nightly_test1/vpr_reg_mcnc...[Pass]
regression_tests/vtr_reg_nightly_test1/vtr_reg_fpu_soft_logic_arch...[Pass]
regression_tests/vtr_reg_nightly_test1/vtr_reg_fpu_hard_block_arch...[Pass]
vtr_reg_nightly_test1/arithmetic_tasks/figure_8...[Pass]
vtr_reg_nightly_test1/arithmetic_tasks/FIR_filters...[Pass]

vtr_reg_nightly_test1/arithmetic_tasks/FIR_filters_frac...[Fail]
[Fail]
k6_frac_N8_22nm.xml/fir_nopipe_38.v/common min_chan_width_route_time relative value 10.580357142857144 outside of range [0.1,10.0], above absolute threshold 2.0 and not equal to golden value: 10.08
vtr_reg_nightly_test1/arithmetic_tasks/multless_consts...[Pass]
vtr_reg_nightly_test1/arithmetic_tasks/open_cores...[Pass]
vtr_reg_nightly_test1/arithmetic_tasks/open_cores_frac...[Pass]
regression_tests/vtr_reg_nightly_test1/power_extended_arch_list...[Pass]
regression_tests/vtr_reg_nightly_test1/power_extended_circuit_list...[Pass]
regression_tests/vtr_reg_nightly_test1/vpr_ispd...[Pass]

Test 'vtr_reg_nightly_test1' had 1 qor test failures

Test 'vtr_reg_nightly_test1' had 0 run failures

Error: 1 tests failed
++ TEST_RESULT=1
++ set -e
++ kill 16520
++ echo ========================================
========================================
++ echo 'Cleaning benchmarks files'
Cleaning benchmarks files
++ echo ========================================
========================================
++ find vtr_flow/arch/symbiflow/ -type f -not -name 'README.*' -delete
/tmpfs/src/github/vtr-verilog-to-routing/.github/kokoro/steps/vtr-test.sh: line 59: 16520 Terminated              ( while :; do
    date; uptime; free -h; sleep 300;
done )
++ find vtr_flow/benchmarks/symbiflow/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/ispd_blif/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/titan_blif/ -type f -not -name 'README.*' -delete
++ find . -type f -regex '.*\.tar\.\(gz\|xz\)' -delete
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/ -type f -print0
+++ nproc
++ xargs -0 -P 8 gzip
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly_test2/ -type f -print0
+++ nproc
++ xargs -0 -P 8 gzip
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly_test3/ -type f -print0
+++ nproc
++ xargs -0 -P 8 gzip
+++ du -sh
++ echo 'Working directory size: 2.9G	.'
Working directory size: 2.9G	.
+++ du -s
+++ cut -d '	' -f 1
+++ expr 1024 '*' 1024 '*' 90
++ [[ 3019356 -gt 94371840 ]]
++ exit 1


[ID: 3933417] Build finished after 12808 secs, exit value: 1

@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

There still seems to be a lot of files being saved;

[05:15:56] Copied /tmp/workspace/workspace/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/artifacts/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_frac_2uripple_N8_22nm.xml/mult_060.v/common/report_clocks.abc.out.gz to /bigstore/vtr-verilog-to-routing/artifacts/prod/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/4/20210609-012326/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_frac_2uripple_N8_22nm.xml/mult_060.v/common/report_clocks.abc.out.gz
[05:15:56] Copied /tmp/workspace/workspace/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/artifacts/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/figure_8/run001/fixed_k6_N8_unbalanced_ripple_chain_gate_boost_0.2V_22nm.xml/adder_028bits.v/common/report_clocks.abc.out.gz to /bigstore/vtr-verilog-to-routing/artifacts/prod/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/4/20210609-012326/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/figure_8/run001/fixed_k6_N8_unbalanced_ripple_chain_gate_boost_0.2V_22nm.xml/adder_028bits.v/common/report_clocks.abc.out.gz
[05:15:56] Copied /tmp/workspace/workspace/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/artifacts/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_N8_gate_boost_0.2V_22nm.xml/mult_036.v/common/abc0.out.gz to /bigstore/vtr-verilog-to-routing/artifacts/prod/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/4/20210609-012326/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_N8_gate_boost_0.2V_22nm.xml/mult_036.v/common/abc0.out.gz
[05:15:56] Copied /tmp/workspace/workspace/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/artifacts/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/FIR_filters_frac/run001/k6_frac_ripple_N8_22nm.xml/fir_pipe_14.v/common/vpr.crit_path.out.gz to /bigstore/vtr-verilog-to-routing/artifacts/prod/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/4/20210609-012326/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/FIR_filters_frac/run001/k6_frac_ripple_N8_22nm.xml/fir_pipe_14.v/common/vpr.crit_path.out.gz
[05:15:56] Copied /tmp/workspace/workspace/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/artifacts/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_N8_gate_boost_0.2V_22nm.xml/mult_120.v/common/0_blackboxing_latch.out.gz to /bigstore/vtr-verilog-to-routing/artifacts/prod/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/4/20210609-012326/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_N8_gate_boost_0.2V_22nm.xml/mult_120.v/common/0_blackboxing_latch.out.gz
[05:15:56] Copied /tmp/workspace/workspace/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/artifacts/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_N8_gate_boost_0.2V_22nm.xml/mult_067.v/common/vtr_flow.out.gz to /bigstore/vtr-verilog-to-routing/artifacts/prod/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/4/20210609-012326/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_N8_gate_boost_0.2V_22nm.xml/mult_067.v/common/vtr_flow.out.gz
[05:15:56] Copied /tmp/workspace/workspace/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/artifacts/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_frac_2uripple_N8_22nm.xml/mult_101.v/common/report_clocks.abc.out.gz to /bigstore/vtr-verilog-to-routing/artifacts/prod/foss-fpga-tools/verilog-to-routing/upstream/presubmit/nightly_test1/4/20210609-012326/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/arithmetic_tasks/multless_consts/run001/fixed_k6_frac_2uripple_N8_22nm.xml/mult_101.v/common/report_clocks.abc.out.gz

@mithro
Copy link
Contributor

mithro commented Jun 9, 2021

[05:16:11] Copied 23581 build artifacts to GCS in 135519ms

@ArashAhmadian
Copy link
Contributor Author

Thanks @mithro. I updated vtr_test.sh and the golden_results.txt for the tasks that were failing qor. Waiting for new tests to complete.

Regarding the unexpected run-time of vtr_reg_nightly2, the longest flow run took for that suite took ~288mins in the latest run of the CI which took 314 mins.

@ArashAhmadian
Copy link
Contributor Author

stratixiv_arch.timing/directrf_stratixiv_arch_timing		OK (took 17331.66 seconds)
Elapsed time: 17332.10 seconds

Parsing test results...
scripts/parse_vtr_task.py -l /tmpfs/src/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test2/task_list.txt
Elapsed time: 8.49 seconds

Calculating QoR results...
regression_tests/vtr_reg_nightly_test2/vtr_reg_netlist_writer...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_func_formal...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_bidir...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_bidir...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_complex_switch...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_titan...[Pass]
regression_tests/vtr_reg_nightly_test2/vpr_verify_rr_graph_error_check...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_timing_update_diff...[Pass]
regression_tests/vtr_reg_nightly_test2/vtr_timing_update_diff_titan...[Pass]
vtr_reg_nightly_test2/func_multiclock/multiclock_mcnc...[Pass]
regression_tests/vtr_reg_nightly_test2/titan_other...[Pass]
regression_tests/vtr_reg_nightly_test2/titan_quick_qor...[Pass]

Test 'vtr_reg_nightly_test2' had 0 qor test failures

Test 'vtr_reg_nightly_test2' had 0 run failures

All tests passed
++ TEST_RESULT=0
++ set -e
++ kill 16514
++ echo ========================================
========================================
++ echo 'Cleaning benchmarks files'
Cleaning benchmarks files
++ echo ========================================
========================================
++ find vtr_flow/arch/symbiflow/ -type f -not -name 'README.*' -delete
/tmpfs/src/github/vtr-verilog-to-routing/.github/kokoro/steps/vtr-test.sh: line 59: 16514 Terminated              ( while :; do
    date; uptime; free -h; sleep 300;
done )
++ find vtr_flow/benchmarks/symbiflow/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/ispd_blif/ -type f -not -name 'README.*' -delete
++ find vtr_flow/benchmarks/titan_blif/ -type f -not -name 'README.*' -delete
++ find . -type f -regex '.*\.tar\.\(gz\|xz\)' -delete
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly_test1/ -type f -print0
+++ nproc
++ xargs -0 -P 8 gzip
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly_test2/ -type f -print0
+++ nproc
++ xargs -0 -P 8 gzip
++ find vtr_flow/tasks/regression_tests/vtr_reg_nightly_test3/ -type f -print0
+++ nproc
++ xargs -0 -P 8 gzip
+++ du -sh
++ echo 'Working directory size: 4.5G	.'
Working directory size: 4.5G	.
+++ du -s
+++ cut -d '	' -f 1
+++ expr 1024 '*' 1024 '*' 90
++ [[ 4662048 -gt 94371840 ]]
++ exit 0


[ID: 4893247] Build finished after 18560 secs, exit value: 0

@vaughnbetz
Copy link
Contributor

Thanks @ArashAhmadian . The changes look good; waiting for CI to go green before merging.
Is there another PR to update documentation (explain that nightly is now really 3 separate tests to optimize CI runtime, and developers should add to one of the 3, or create a new one for high runtime tests)? If not, good to get that PR going next.

@ArashAhmadian
Copy link
Contributor Author

Thanks @ArashAhmadian . The changes look good; waiting for CI to go green before merging.
Is there another PR to update documentation (explain that nightly is now really 3 separate tests to optimize CI runtime, and developers should add to one of the 3, or create a new one for high runtime tests)? If not, good to get that PR going next.

I have added a README_nightly_regression_parallelism.md under regression_tests which goes over the strategy behind the 3 way split and how to add tests. Would you prefer for this description to be added or moved to the doc?

@ArashAhmadian
Copy link
Contributor Author

@vaughnbetz All CI tests passed. Updating golden results for the FIR_filters & FIR_filters_frac seemed to have fixed the qor fails.

@acomodi
Copy link
Collaborator

acomodi commented Jun 10, 2021

@ArashAhmadian I was looking at the results from the nightly tests and I may have missed it but I think that the symbiflow tests are not currently being run, given also the corresponding config file seems to have been removed.

@vaughnbetz
Copy link
Contributor

@ArashAhmadian : I like the new README and think it should be included.
We don't want to lose the symbiflow tests from CI; can you look into if the symbiflow tests have been included, and if not include them? (As pointed out by @acomodi )

@mithro
Copy link
Contributor

mithro commented Jun 10, 2021

Looks like the Nightly tests are succeeding now?

@vaughnbetz
Copy link
Contributor

@ArashAhmadian : I did a quick search for the symbiflow config being run in the nightly regtests (all 3 sub-tests) and didn't find it. So it does look like it has gone AWOL. This is the old path to it: https://github.com/verilog-to-routing/vtr-verilog-to-routing/tree/master/vtr_flow/tasks/regression_tests/vtr_reg_nightly/symbiflow/config

@ArashAhmadian
Copy link
Contributor Author

@acomodi Thanks for pointing this out. I have added the symbiflow tests under vtr_reg_nightly_test1.

@vaughnbetz vaughnbetz merged commit a7ae65d into verilog-to-routing:master Jun 11, 2021
@vaughnbetz
Copy link
Contributor

Nice work Arash! This is a big improvement in our test coverage and turnaround time.

@ArashAhmadian
Copy link
Contributor Author

Nice work Arash! This is a big improvement in our test coverage and turnaround time.

Thanks! 🙂

@ArashAhmadian ArashAhmadian deleted the vtr_tasks_reorganization branch June 14, 2021 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation infra Project Infrastructure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants