Strong sanitized test failed with an internal compiler error

[tangxifan]

Expected Behaviour

Strong sanitized test is a required test. It should pass all the time.

Current Behaviour

Strong sanitize tests are failed on current master after merging PR #1916

• The error messages points to an internal error of gcc-5 compiler

/tmpfs/src/github/vtr-verilog-to-routing/utils/fasm/src/main.cpp: In function ‘main’:
/tmpfs/src/github/vtr-verilog-to-routing/utils/fasm/src/main.cpp:61:5: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without
 int main(int argc, const char **argv) {
     ^
/tmpfs/src/github/vtr-verilog-to-routing/vpr/src/route/route_common.cpp: In function ‘init_route_structs’:
/tmpfs/src/github/vtr-verilog-to-routing/vpr/src/route/route_common.cpp:495:0: internal compiler error: Segmentation fault
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
lto-wrapper: fatal error: /usr/bin/c++ returned 1 exit status
compilation terminated.
/usr/bin/ld.gold: fatal error: lto-wrapper failed

• Similar errors are captured in other PRs

  • Enhance Yosys+Odin subcircuits support #1923
  • vpr: interchange: add interchange netlist reading support #1924

• This is weird because PR 1916 passed the sanitized test without errors

Possible Solution

After a discussion with @vaughnbetz, @hzeller and @jgoeders , we currently suspects that it comes from the gcc compiler.
The action items are

• Try a different compiler, e.g., clang, see if we can still reproduce the same error
• In the error log, it is from a gcc-5 compiler, which is quite old. For sanitized tests, @vaughnbetz suggests to follow the latest compiler, e.g., gcc-11. We should consider give a try and see if the errors can be reproduced

Steps to Reproduce

1. Go to the VTR project, click the CI status. Find the VtR_strong sanitized test. Click details
   

2. Click the foss-fpga-tools/verilog-to-routing/upstream/continuous/strong_sanitized in failed target

3. Search internal compiler error

Context

Your Environment

• VTR revision used: e6c2049
• Operating System and version:
• Compiler version: gcc-5 from the CI runner

[tangxifan]

@jgoeders @vaughnbetz See if the issue captures our discussion well.

For @jgoeders , would you mind helping on double checking the gcc on CI runner? I will try gcc-8 or later version on my local computer.

[vaughnbetz]

Thanks @tangxifan . This captures it well I think.

And yes, I think the first thing to try is to simply drop sanitized + g++ 5.0 if we're somehow running that, and also running sanitized plus some recent g++. If we're only running sanitized on g++5.0, we should update.

[vaughnbetz]

We do indeed appear to use only g++5 on the sanitized build. We should fix this.

-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0

[tangxifan]

@umariqbal-rs Since it is seen after merge your PR, would you mind running the sanitized build on your local computer with a later g++ version? I know we have gcc-8, which should be o.k to test.
Detailed cmake parameters and makefile configuration can be found at

vtr-verilog-to-routing/.github/kokoro/presubmit/strong_sanitized.cfg

Lines 38 to 44 in e6c2049

	#VTR build configuration options:
	# -DVTR_ASSERT_LEVEL=3: Enable extra assertion checking
	# -DVTR_ENABLE_SANITIZE=ON: Enable compiler sanitizers (check for memory issues, undefined behaviour, etc.)
	env_vars {
	key: "VTR_CMAKE_PARAMS"
	value: "-DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=ON"
	}

Detailed command run on CI is (from the CI log)

++ make -k 'CMAKE_PARAMS=-DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=ON' -j16
Performing standard build...
cd build && /usr/bin/cmake -DCMAKE_BUILD_TYPE=release -G 'Unix Makefiles' -DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=ON /tmpfs/src/github/vtr-verilog-to-routing

[Raza-jafari-RS]

@umariqbal-rs Since it is seen after merge your PR, would you mind running the sanitized build on your local computer with a later g++ version? I know we have gcc-8, which should be o.k to test. Detailed cmake parameters and makefile configuration can be found at

vtr-verilog-to-routing/.github/kokoro/presubmit/strong_sanitized.cfg

Lines 38 to 44 in e6c2049

	#VTR build configuration options:
	# -DVTR_ASSERT_LEVEL=3: Enable extra assertion checking
	# -DVTR_ENABLE_SANITIZE=ON: Enable compiler sanitizers (check for memory issues, undefined behaviour, etc.)
	env_vars {
	key: "VTR_CMAKE_PARAMS"
	value: "-DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=ON"
	}

Detailed command run on CI is (from the CI log)

++ make -k 'CMAKE_PARAMS=-DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=ON' -j16
Performing standard build...
cd build && /usr/bin/cmake -DCMAKE_BUILD_TYPE=release -G 'Unix Makefiles' -DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=ON /tmpfs/src/github/vtr-verilog-to-routing

@tangxifan
I ran the devised command and the target is successfully built. The results are as follow.

[tangxifan]

@Raza-jafari-RS Thanks for the input.
@jgoeders @vaughnbetz It seems a problem of gcc-5. I suggest to try later gcc. Rapidsilicon team tried gcc-8 and it is working fine.

[vaughnbetz]

Sounds good. Jeff or Xifan: if one of you can update the CI script to use gcc-8 with the sanitized build that would be great, and we can close this.

[tangxifan]

@vaughnbetz I just looked into all the kokoro tests. And I found that they all use the gcc-5.4.0 when compile VTR. Should we update all the kokoro tests to use gcc-8 or later?

@mithro I am not sure any other gcc compilers are available on the kokoro machine. And which command to check CC/CXX in kokoro context. I am looking into this file as an stabbing point. If you can introduce any engineer who is familiar with it, I can contribute. Thank you in advance.

vtr-verilog-to-routing/.github/kokoro/presubmit/strong_sanitized.cfg

Lines 23 to 26 in e6c2049

	env_vars {
	key: "KOKORO_TYPE"
	value: "presubmit"
	}

[vaughnbetz]

Yes, I think updataing them all to a more recent g++ would be wise. I don't have a really strong opinion on version number though, as long as it is fairly recent and not known to be unstable.

[tangxifan]

solved by @acomodi in PR #1924