Skip to content

Fixing the generated vtr_flow.sh scripts #1733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
May 30, 2021

Conversation

MohamedElgammal
Copy link
Contributor

Description

The run_vtr_task.py script can generate vtr_flow.sh scripts by adding -system scripts option. Each script runs one (design, arch) pair of the specified task
In this PR, some bugs of this generation process is fixed:

  • the script now generates a correct vpr command
  • the script now generates the full cartesian product runs of the designs, architectures, and script parameters.
  • the script adds executable permission to the generated shell scripts

Related Issue

Motivation and Context

The generated shell scripts can be used to run multiple VPR runs in parallel on a large cluster.

How Has This Been Tested?

Types of changes

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My change requires a change to the documentation
  • I have updated the documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed

@github-actions github-actions bot added lang-python Python code scripts Utility & Infrastructure scripts tests labels May 20, 2021
@@ -349,15 +350,15 @@ def run_parallel(args, queued_jobs, run_dirs):


def create_run_scripts(args, jobs, run_dirs):
""" Create the bash script files for each job run """
"""Create the bash script files for each job run"""
Copy link
Contributor

@vaughnbetz vaughnbetz May 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should add more commenting around the "scripts" option. For example around line 310, "elif args.system == "scripts":"
you could describe what this argument does.
Same thing for the documentation of the command line option --system (line 160 or so):
The help says "What system to run this on" which isn't very detailed. Should explain what the --system scripts option does somewhere, likely in the help. Details might be in a comment block.
parser.add_argument(
"-system",
choices=["local", "scripts"],
default="local",
help="What system to run the tasks on.", // can improve this.
)

@vaughnbetz
Copy link
Contributor

The clang errors should be fixed already (was an unrelated missing #include). Updating the branch to kick that off.

There are two scripts with python lint errors that should be fixed.

E.g.: vtr_flow/scripts/python_libs/vtr/task.py has lint errors
************* Module vtr.task
vtr_flow/scripts/python_libs/vtr/task.py:463:4: W0612: Unused variable 'num_spaces_before' (unused-variable)
vtr_flow/scripts/python_libs/vtr/task.py:464:4: W0612: Unused variable 'num_spaces_after' (unused-variable)

@MohamedElgammal MohamedElgammal force-pushed the fix_generated_shell_scripts branch from a5dc904 to 18304b2 Compare May 28, 2021 09:10
@vaughnbetz
Copy link
Contributor

Still some python lint errors:
vtr_flow/scripts/run_vtr_task.py has lint errors
************* Module run_vtr_task
vtr_flow/scripts/run_vtr_task.py:126:0: C0301: Line too long (157/100) (line-too-long)
vtr_flow/scripts/run_vtr_task.py:303:0: C0301: Line too long (102/100) (line-too-long)
vtr_flow/scripts/run_vtr_task.py:354:22: W0613: Unused argument 'args' (unused-argument)
vtr_flow/scripts/run_vtr_task.py:403:7: C0121: Comparison 'metrics == None' should be 'metrics is None' (singleton-comparison)
vtr_flow/scripts/run_vtr_task.py:418:7: C0121: Comparison 'metrics == None' should be 'metrics is None' (singleton-comparison)
vtr_flow/scripts/run_vtr_task.py:24:0: W0611: Unused find_longest_task_description imported from vtr (unused-import)

@vaughnbetz
Copy link
Contributor

Waiting for CI to go green ...
@MohamedElgammal : would also be good to add more commenting, as noted above. Can merge without that though, but it would be good to comment sooner than later since it tends to get forgotten otherwise.

@MohamedElgammal
Copy link
Contributor Author

@vaughnbetz added more commenting and fixed all python linting issues. Will merge into master.

@MohamedElgammal MohamedElgammal merged commit 606b079 into master May 30, 2021
@MohamedElgammal MohamedElgammal deleted the fix_generated_shell_scripts branch May 30, 2021 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lang-python Python code scripts Utility & Infrastructure scripts tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants