[CI] Logs Memory during CI Run

[ganeshgore]

Description

Logs peak memory usage during VPR CI run. The information is extracted from the vpr.out (which I assumed obtained using valgrid)

Related Issue

Addresses part of issue #1808
#1808 (comment)

Motivation and Context

How Has This Been Tested?

It has been tested locally, I have attached the screenshot.

Types of changes

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation
• I have updated the documentation accordingly
• I have added tests to cover my changes
• All new and existing tests passed

[tangxifan]

@ganeshgore Need to fix errors in Python Lint

[tangxifan]

See if you need to use the function format_human_readable_memory to format the mem values:

vtr-verilog-to-routing/vtr_flow/scripts/run_vtr_task.py

Line 439 in 8861b11

def format_human_readable_memory(num_bytes):

[vaughnbetz]

Looks like a good change -- thanks @ganeshgore . We don't get the resident set size from valgrind; we make os calls so we can get it on every run without slowdown, so parsing it out in CI as you've done is very useful (it's always produced).

To fix the python lint errors, you just have to run the autoformatter on the code and push a new commit:

@@ -488,14 +488,13 @@ def vtr_command_main(arg_list, prog=None):
 
     finally:
         seconds = datetime.now() - start
-        with open(temp_dir/"vpr.out","r") as fpmem:
+        with open(temp_dir / "vpr.out", "r") as fpmem:
             for line in fpmem.readlines():
                 if "Maximum resident set size" in line:
-                    mem_usage = int(line.split()[-1])//1024
+                    mem_usage = int(line.split()[-1]) // 1024
         print(
             "{status} (took {time}, vpr run consumed {max_mem} MB memory)".format(
-                status=error_status, time=vtr.format_elapsed_time(seconds),
-                max_mem=mem_usage
+                status=error_status, time=vtr.format_elapsed_time(seconds), max_mem=mem_usage
             )
         )
         temp_dir.mkdir(parents=True, exist_ok=True)

Run 'make format-py' to apply these changes

[tangxifan]

@ganeshgore Still see errors in the python lint

[tangxifan]

@vaughnbetz Sorry for bothering. We may need to add @ganeshgore to the contributor list so that the kokoro can be triggered.

@ganeshgore Still see lint problems.

[vaughnbetz]

Thanks @ganeshgore and @tangxifan . The code looks good and I started the CI workflow, and added @ganeshgore as a committer for the future.

[ganeshgore]

Thank you @vaughnbetz, For the current changes, I passed all tests in my repo. So we are just waiting for tests to finish here.
https://github.com/ganeshgore/vtr-verilog-to-routing/actions/runs/1244336227

[tangxifan]

@vaughnbetz Not sure why the VtR yosys+Odin-II test is stuck here. See something similar at #1843

If needed, I can rerun kokoro and see if the issue can be resolved.

[vaughnbetz]

@sdamghan : any ideas? Is this a deprecated test? I can force merge anyway, but it would be good to figure out if this is an ongoing issue.

[sdamghan]

@vaughnbetz no it is not deprecated, actually @QuantamHD just changed Yosys+Odin-II CI tests' name on the Google side (issue #1848 ). It's interesting why it appears in this PR and #1843 since I do not see Yosys+Odin-II (presubmit) and Yosys+Odin-II (continuous) in my recent WIP PR (#1844 ). Maybe a new 'kokoro:force-run' would solve the issue.

[tangxifan]

@vaughnbetz no it is not deprecated, actually @QuantamHD just changed Yosys+Odin-II CI tests' name on the Google side (issue #1848 ). It's interesting why it appears in this PR and #1843 since I do not see Yosys+Odin-II (presubmit) and Yosys+Odin-II (continuous) in my recent WIP PR (#1844 ). Maybe a new 'kokoro:force-run' would solve the issue.

Thanks @sdamghan. Let me try.

[tangxifan]

@vaughnbetz The CI is green now. I believe it is ready to be merged.