Skip to content

Vtr task and test script python rewrite #1509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 17, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/kokoro/continuous/nightly.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ env_vars {
value: "vtr_reg_nightly"
}

#Options for run_reg_test.pl
#Options for run_reg_test.py
# -show_failures: show tool failures in main log output
env_vars {
key: "VTR_TEST_OPTIONS"
Expand Down
2 changes: 1 addition & 1 deletion .github/kokoro/continuous/strong.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ env_vars {
value: "vtr_reg_strong"
}

#Options for run_reg_test.pl
#Options for run_reg_test.py
# -show_failures: show tool failures in main log output
env_vars {
key: "VTR_TEST_OPTIONS"
Expand Down
2 changes: 1 addition & 1 deletion .github/kokoro/continuous/strong_sanitized.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ env_vars {
value: "vtr_reg_strong"
}

#Options for run_reg_test.pl
#Options for run_reg_test.py
# -show_failures: show tool failures in main log output
# -skip_qor: Skip QoR checks (since we expect run-time failures due to sanitizers)
env_vars {
Expand Down
2 changes: 1 addition & 1 deletion .github/kokoro/continuous/weekly.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ env_vars {
value: "vtr_reg_weekly"
}

#Options for run_reg_test.pl
#Options for run_reg_test.py
# -show_failures: show tool failures in main log output
env_vars {
key: "VTR_TEST_OPTIONS"
Expand Down
2 changes: 1 addition & 1 deletion .github/kokoro/presubmit/nightly.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ env_vars {
value: "vtr_reg_nightly"
}

#Options for run_reg_test.pl
#Options for run_reg_test.py
# -show_failures: show tool failures in main log output
env_vars {
key: "VTR_TEST_OPTIONS"
Expand Down
2 changes: 1 addition & 1 deletion .github/kokoro/presubmit/strong.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ env_vars {
value: "vtr_reg_strong"
}

#Options for run_reg_test.pl
#Options for run_reg_test.py
# -show_failures: show tool failures in main log output
env_vars {
key: "VTR_TEST_OPTIONS"
Expand Down
2 changes: 1 addition & 1 deletion .github/kokoro/presubmit/strong_sanitized.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ env_vars {
value: "vtr_reg_strong"
}

#Options for run_reg_test.pl
#Options for run_reg_test.py
# -show_failures: show tool failures in main log output
# -skip_qor: Skip QoR checks (since we expect run-time failures due to sanitizers)
env_vars {
Expand Down
2 changes: 1 addition & 1 deletion .github/kokoro/steps/vtr-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,5 +45,5 @@ echo "========================================"
echo "Running Tests"
echo "========================================"
export VPR_NUM_WORKERS=1
./run_reg_test.pl $VTR_TEST $VTR_TEST_OPTIONS -j$NUM_CORES
./run_reg_test.py $VTR_TEST $VTR_TEST_OPTIONS -j$NUM_CORES
kill $MONITOR
14 changes: 7 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ jobs:
- ./dev/check-format-py.sh
- stage: Test
name: "Python Lint"
script:
script:
- python3 ./dev/pylint_check.py
- stage: Test
name: "C++ Unit Tests"
Expand All @@ -112,7 +112,7 @@ jobs:
- MATRIX_EVAL="CC=gcc-5 && CXX=g++-5"
script:
- ./.github/travis/build.sh
- travis_wait 30 ./run_reg_test.pl vtr_reg_basic -show_failures -j2
- travis_wait 30 ./run_reg_test.py vtr_reg_basic -show_failures -j2
- stage: Test
name: "Basic Regression Tests with NO_GRAPHICS"
env:
Expand All @@ -136,15 +136,15 @@ jobs:
- MATRIX_EVAL="CC=gcc-5 && CXX=g++-5"
script:
- ./.github/travis/build.sh
- travis_wait 30 ./run_reg_test.pl vtr_reg_strong -show_failures -j2
- travis_wait 30 ./run_reg_test.py vtr_reg_strong -show_failures -j2
- stage: Test
name: "Basic Valgrind Memory Tests"
env:
- CMAKE_PARAMS="-DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on"
- MATRIX_EVAL="CC=gcc-5 && CXX=g++-5"
script:
- ./.github/travis/build.sh
- travis_wait 30 ./run_reg_test.pl vtr_reg_valgrind_small -show_failures -j2
- travis_wait 30 ./run_reg_test.py vtr_reg_valgrind_small -show_failures -j2
- stage: Test
name: "Sanitized Basic Regression Tests"
env:
Expand All @@ -155,7 +155,7 @@ jobs:
script:
- ./.github/travis/build.sh
#We skip QoR since we are only checking for errors in sanitizer runs
- travis_wait 50 ./run_reg_test.pl vtr_reg_basic -show_failures -skip_qor -j2
- travis_wait 50 ./run_reg_test.py vtr_reg_basic -show_failures -skip_qor -j2
#Currently strong regression with sanitizers is disabled as it exceeds the maximum travis job run-time
#- stage: Test
#name: "Sanitized Strong Regression Tests"
Expand All @@ -167,7 +167,7 @@ jobs:
#script:
#- ./.github/travis/build.sh
##We skip QoR since we are only checking for errors in sanitizer runs
#- travis_wait 60 ./run_reg_test.pl vtr_reg_strong -show_failures -skip_qor -j2
#- travis_wait 60 ./run_reg_test.py vtr_reg_strong -show_failures -skip_qor -j2
- stage: Test
name: "ODIN-II Micro Tests"
env:
Expand All @@ -176,7 +176,7 @@ jobs:
- BUILD_TYPE=debug
script:
- ./.github/travis/build.sh
- ./run_reg_test.pl odin_reg_micro -show_failures -j2
- ./run_reg_test.py odin_reg_micro -show_failures -j2

- stage: Test
name: "Build Compatibility: GCC 5 (Ubuntu Xenial - 16.04)"
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ ifneq (,$(findstring pgo,$(BUILD_TYPE)))
#Need titan benchmarks for pgo_profile task
@+$(MAKE) -C $(BUILD_DIR) get_titan_benchmarks
#Note profiling must be done serially to avoid corrupting the generated profiles
./run_reg_test.pl pgo_profile
./run_reg_test.py pgo_profile
#
#Configure 2nd-stage build to use profiling data to guide compiler optimization
#
Expand Down
2 changes: 1 addition & 1 deletion ODIN_II/verify_odin.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1255,7 +1255,7 @@ FILTERED_VTR_TASK_PATH="${NEW_RUN_DIR}/vtr/task_list.txt"
function run_vtr_reg() {
pushd "${VTR_DIR}" &> /dev/null
RELATIVE_PATH_TO_TEST=$(realapath_from "${FILTERED_VTR_TASK_PATH}" "${VTR_REG_DIR}")
/usr/bin/env perl run_reg_test.pl -j "${_NUMBER_OF_PROCESS}" "${RELATIVE_PATH_TO_TEST}"
/usr/bin/env perl run_reg_test.py -j "${_NUMBER_OF_PROCESS}" "${RELATIVE_PATH_TO_TEST}"
popd &> /dev/null
}

Expand Down
44 changes: 22 additions & 22 deletions README.developers.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ The overall approach is similar, but we call out the differences below.
At the bare minimum it is recommended to run:
```
make #Rebuild the code
./run_reg_test.pl vtr_reg_basic vtr_reg_strong #Run tests
./run_reg_test.py vtr_reg_basic vtr_reg_strong #Run tests
```

See [Running Tests](#running-tests) for more details.
Expand Down Expand Up @@ -250,11 +250,11 @@ There are 4 main regression tests:
QoR checks in this regression are aimed at evaluating quality and run-time of the VTR flow.
As a result any QoR failures are a concern and should be investigated and understood.

These can be run with `run_reg_test.pl`:
These can be run with `run_reg_test.py`:
```shell
#From the VTR root directory
$ ./run_reg_test.pl vtr_reg_basic
$ ./run_reg_test.pl vtr_reg_strong
$ ./run_reg_test.py vtr_reg_basic
$ ./run_reg_test.py vtr_reg_strong
```

The *nightly* and *weekly* regressions require the Titan and ISPD benchmarks
Expand All @@ -263,22 +263,22 @@ which can be integrated into your VTR tree with:
make get_titan_benchmarks
make get_ispd_benchmarks
```
They can then be run using `run_reg_test.pl`:
They can then be run using `run_reg_test.py`:
```shell
$ ./run_reg_test.pl vtr_reg_nightly
$ ./run_reg_test.pl vtr_reg_weekly
$ ./run_reg_test.py vtr_reg_nightly
$ ./run_reg_test.py vtr_reg_weekly
```

To speed-up things up, individual sub-tests can be run in parallel using the `-j` option:
```shell
#Run up to 4 tests in parallel
$ ./run_reg_test.pl vtr_reg_strong -j4
$ ./run_reg_test.py vtr_reg_strong -j4
```

You can also run multiple regression tests together:
```shell
#Run both the basic and strong regression, with up to 4 tests in parallel
$ ./run_reg_test.pl vtr_reg_basic vtr_reg_strong -j4
$ ./run_reg_test.py vtr_reg_basic vtr_reg_strong -j4
```

## Odin Functionality Tests
Expand All @@ -291,8 +291,8 @@ Odin has its own set of tests to verify the correctness of its synthesis results
These can be run with:
```shell
#From the VTR root directory
$ ./run_reg_test.pl odin_reg_micro
$ ./run_reg_test.pl odin_reg_full
$ ./run_reg_test.py odin_reg_micro
$ ./run_reg_test.py odin_reg_full
```
and should be used when making changes to Odin.

Expand Down Expand Up @@ -391,7 +391,7 @@ Lets assume we have a failure in `vtr_reg_basic`:

```shell
#In the VTR root directory
$ ./run_reg_test.pl vtr_reg_strong
$ ./run_reg_test.py vtr_reg_strong
#Output trimmed...
regression_tests/vtr_reg_basic/basic_no_timing
-----------------------------------------
Expand Down Expand Up @@ -421,7 +421,7 @@ latest run002 run004 run005
There we see there is a `config` directory (which defines the test), and a set of run-directories.
Each time a test is run it creates a new `runXXX` directory (where `XXX` is an incrementing number).
From the above we can tell that our last run was `run005` (the symbolic link `latest` also points to the most recent run directory).
From the output of `run_reg_test.pl` we know that one of the failing architecture/circuit/parameters combinations was `k4_N10_memSize16384_memData64/ch_intrinsics/common`.
From the output of `run_reg_test.py` we know that one of the failing architecture/circuit/parameters combinations was `k4_N10_memSize16384_memData64/ch_intrinsics/common`.
Each architecture/circuit/parameter combination is run in its own sub-folder.
Lets move to that directory:
```shell
Expand Down Expand Up @@ -561,12 +561,12 @@ A typical approach to evaluating an algorithm change would be to run `vtr_reg_qo
$ cd vtr_flow/tasks

#Run the VTR benchmarks
$ ../scripts/run_vtr_task.pl regression_tests/vtr_reg_nightly/vtr_reg_qor_chain
$ ../scripts/run_vtr_task.py regression_tests/vtr_reg_nightly/vtr_reg_qor_chain

#Several hours later... they complete

#Parse the results
$ ../scripts/parse_vtr_task.pl regression_tests/vtr_reg_nightly/vtr_reg_qor_chain
$ ../scripts/python_libs/vtr/parse_vtr_task.py regression_tests/vtr_reg_nightly/vtr_reg_qor_chain

#The run directory should now contain a summary parse_results.txt file
$ head -5 vtr_reg_nightly/vtr_reg_qor_chain/latest/parse_results.txt
Expand Down Expand Up @@ -596,12 +596,12 @@ $ make get_titan_benchmarks
$ cd vtr_flow/tasks

#Run the VTR benchmarks
$ ../scripts/run_vtr_task.pl regression_tests/vtr_reg_weekly/vtr_reg_titan
$ ../scripts/run_vtr_task.py regression_tests/vtr_reg_weekly/vtr_reg_titan

#Several days later... they complete

#Parse the results
$ ../scripts/parse_vtr_task.pl regression_tests/vtr_reg_weekly/vtr_reg_titan
$ ../scripts/python_libs/vtr/parse_vtr_task.py regression_tests/vtr_reg_weekly/vtr_reg_titan

#The run directory should now contain a summary parse_results.txt file
$ head -5 vtr_reg_nightly/vtr_reg_qor_chain/latest/parse_results.txt
Expand Down Expand Up @@ -793,22 +793,22 @@ This describes adding a test to `vtr_reg_strong`, but the process is similar for
```shell
#From the VTR root
$ cd vtr_flow/tasks
$ ../scripts/run_vtr_task.pl regression_tests/vtr_reg_strong/strong_mytest
$ ../scripts/run_vtr_task.py regression_tests/vtr_reg_strong/strong_mytest

regression_tests/vtr_reg_strong/strong_mytest
-----------------------------------------
Current time: Jan-25 06:51 PM. Expected runtime of next benchmark: Unknown
k6_frac_N10_mem32K_40nm/ch_intrinsics...OK
```

Next we can generate the golden reference results using `parse_vtr_task.pl` with the `-create_golden` option:
Next we can generate the golden reference results using `parse_vtr_task.py` with the `-create_golden` option:
```shell
$ ../scripts/parse_vtr_task.pl regression_tests/vtr_reg_strong/strong_mytest -create_golden
$ ../scripts/python_libs/vtr/parse_vtr_task.py regression_tests/vtr_reg_strong/strong_mytest -create_golden
```

And check that everything matches with `-check_golden`:
```shell
$ ../scripts/parse_vtr_task.pl regression_tests/vtr_reg_strong/strong_mytest -check_golden
$ ../scripts/python_libs/vtr/parse_vtr_task.py regression_tests/vtr_reg_strong/strong_mytest -check_golden
regression_tests/vtr_reg_strong/strong_mytest...[Pass]
```

Expand All @@ -825,7 +825,7 @@ This describes adding a test to `vtr_reg_strong`, but the process is similar for
Now, when we run `vtr_reg_strong`:
```shell
#From the VTR root directory
$ ./run_reg_test.pl vtr_reg_strong
$ ./run_reg_test.py vtr_reg_strong
#Output trimmed...
regression_tests/vtr_reg_strong/strong_mytest
-----------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion dev/DOCKER_DEPLOY.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ First, use one of the terminals and compile VTR:
make && make installation/

Second, ensure that a basic regression test passes:
./run_reg_test.pl vtr_reg_basic
./run_reg_test.py vtr_reg_basic

Third, run and/or modify VTR in the usual way.

Expand Down
2 changes: 1 addition & 1 deletion dev/pylint_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ def main():
ignore_list.append("C0330")

# Build pylint command
cmd = ["pylint", path, "-s", "n", "--disable=C0330"]
cmd = ["pylint", path, "-s", "n"]
if ignore_list:
cmd.append("--disable=" + ",".join(ignore_list))

Expand Down
4 changes: 2 additions & 2 deletions doc/src/quickstart/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ Lets make a new directory to work in named ``blink_run_flow``:
> mkdir -p ~/vtr_work/quickstart/blink_run_flow
> cd ~/vtr_work/quickstart/blink_run_flow

Now lets run the script (``$VTR_ROOT/vtr_flow/scripts/run_vtr_flow.pl``) passing in:
Now lets run the script (``$VTR_ROOT/vtr_flow/scripts/run_vtr_flow.py``) passing in:

* The circuit verilog file (``$VTR_ROOT/doc/src/quickstart/blink.v``)
* The FPGA architecture file (``$VTR_ROOT/vtr_flow/arch/timing/EArch.xml``)
Expand All @@ -390,7 +390,7 @@ The resulting command is:

.. code-block:: bash

> $VTR_ROOT/vtr_flow/scripts/run_vtr_flow.pl \
> $VTR_ROOT/vtr_flow/scripts/run_vtr_flow.py \
$VTR_ROOT/doc/src/quickstart/blink.v \
$VTR_ROOT/vtr_flow/arch/timing/EArch.xml \
-temp_dir . \
Expand Down
6 changes: 3 additions & 3 deletions doc/src/tutorials/flow/basic_flow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The following steps show you to run the VTR design flow to map a sample circuit

.. code-block:: shell

../scripts/run_vtr_task.pl basic_flow
../scripts/run_vtr_task.py basic_flow

This command will run the VTR flow on a single circuit and a single architecture.
The files generated from the run are stored in ``basic_flow/run[#]`` where ``[#]`` is the number of runs you have done.
Expand All @@ -18,7 +18,7 @@ The following steps show you to run the VTR design flow to map a sample circuit

.. code-block:: shell

../scripts/parse_vtr_task.pl basic_flow/
../scripts/python_libs/vtr/parse_vtr_task.py basic_flow/

This parses out the information of the VTR run and outputs the results in a text file called ``run[#]/parse_results.txt``.

Expand All @@ -30,7 +30,7 @@ The following steps show you to run the VTR design flow to map a sample circuit

.. code-block:: shell

../scripts/parse_vtr_task.pl -check_golden basic_flow
../scripts/python_libs/vtr/parse_vtr_task.py -check_golden basic_flow

It should return: ``basic_flow...[Pass]``

Expand Down
2 changes: 1 addition & 1 deletion doc/src/vtr/install_vtr.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Verifying Installation
~~~~~~~~~~~~~~~~~~~~~~
To verfiy that VTR has been installed correctly run::

$VTR_ROOT/vtr_flow/scripts/run_vtr_task.pl basic_flow
$VTR_ROOT/vtr_flow/scripts/run_vtr_task.py basic_flow

The expected output is::

Expand Down
6 changes: 3 additions & 3 deletions doc/src/vtr/parse_vtr_flow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ This script parses statistics generated by a single execution of the VTR flow.

The script is located at::

$VTR_ROOT/vtr_flow/scripts/parse_vtr_flow.pl
$VTR_ROOT/vtr_flow/scripts/python_libs/vtr/parse_vtr_flow.py

.. program:: parse_vtr_flow.pl
.. program:: parse_vtr_flow.py

Usage
~~~~~
Typical usage is::

parse_vtr_flow.pl <parse_path> <parse_config_file>
parse_vtr_flow.py <parse_path> <parse_config_file>

where:

Expand Down
Loading