Failed to run in the project ApolloAuto #62

simon13579 · 2022-06-28T13:26:38Z

Hello,

Thanks for building this tool. Now I am trying to use it to the code ApolloAuto (https://github.com/ApolloAuto/apollo). I check out the code of Apollo v6.0 and make the below changes:

add the below code in the file https://github.com/ApolloAuto/apollo/blob/master/WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")


# Hedron's Compile Commands Extractor for Bazel
# https://github.com/hedronvision/bazel-compile-commands-extractor
http_archive(
    name = "hedron_compile_commands",

    # Replace the commit hash in both places (below) with the latest, rather than using the stale one here.
    # Even better, set up Renovate and let it do the work for you (see "Suggestion: Updates" in the README).
    url = "https://github.com/hedronvision/bazel-compile-commands-extractor/archive/670e86177b6b5c001b03f4efdfba0f8019ff523f.tar.gz",
    strip_prefix = "bazel-compile-commands-extractor-670e86177b6b5c001b03f4efdfba0f8019ff523f",
    # When you first run this tool, it'll recommend a sha256 hash to put here with a message like: "DEBUG: Rule 'hedron_compile_commands' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = ..."
)
load("@hedron_compile_commands//:workspace_setup.bzl", "hedron_compile_commands_setup")
hedron_compile_commands_setup()

I replace the line 251 bazel build ${CMDLINE_OPTIONS} ${job_args} -- ${formatted_targets} of the file https://github.com/ApolloAuto/apollo/blob/master/scripts/apollo_build.sh by bazel run @hedron_compile_commands//:refresh_all -- ${CMDLINE_OPTIONS} ${job_args} -- ${formatted_targets}
Update the python in the container to python 3.7
Add a new line exclude_headers = "all" in the file BUILD which is in the folder Apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/external/hedron_compile_commands. If I don't manually add this, it will report error that it expect the setting of exclude_headers

But when I run it, it reports the error as below:

[WARNING] No nvidia-driver found. CPU will be used.
[WARNING] No nvidia-driver found. CPU will be used.
[INFO] Apollo Environment Settings:
[INFO]     APOLLO_ROOT_DIR: /apollo
[INFO]     APOLLO_CACHE_DIR: /apollo/.cache
[INFO]     APOLLO_IN_DOCKER: true
[INFO]     APOLLO_VERSION: HEAD-2020-09-21-e79f9d6765
[INFO]     DOCKER_IMG: dev-x86_64-18.04-20200914_0742
[INFO]     APOLLO_ENV:  STAGE=dev USE_ESD_CAN=false
[INFO]     USE_GPU: USE_GPU_HOST=0 USE_GPU_TARGET=0
[WARNING] No nvidia-driver found. CPU will be used.
[WARNING] No nvidia-driver found. CPU will be used.
[ OK ] Running CPU build on x86_64 platform.
[WARNING] ESD CAN library supplied by ESD Electronics doesn't exist.
[WARNING] If you need ESD CAN, please refer to:
[WARNING]   third_party/can_card_library/esd_can/README.md
[INFO] Build Overview:
[INFO]     USE_GPU: 0  [ 0 for CPU, 1 for GPU ]
[INFO]     Bazel Options: --config=cpu
[INFO]     Build Targets: //modules/... union //cyber/...
[INFO]     Disabled:      except //modules/drivers/canbus/can_client/esd/... except //modules/perception/... except //modules/planning/open_space/trajectory_smoother:planning_block except //modules/map/pnc_map:cuda_pnc_util except //modules/map/pnc_map:cuda_util_test
(11:38:02) INFO: Current date is 2022-06-28
(11:38:02) INFO: Analyzed target @hedron_compile_commands//:refresh_all (0 packages loaded, 0 targets configured).
(11:38:02) INFO: Found 1 target...
Target @hedron_compile_commands//:refresh_all up-to-date:
  bazel-bin/external/hedron_compile_commands/refresh_all.py
  bazel-bin/external/hedron_compile_commands/refresh_all
(11:38:02) INFO: Elapsed time: 0.184s, Critical Path: 0.00s
(11:38:02) INFO: 0 processes.
(11:38:02) INFO: Build completed successfully, 1 total action
(11:38:02) INFO: Running command line: bazel-bin/external/hedron_compile_commands/refresh_all '--config=cpu' '--jobs=6' '--local_ram_resources=HOST_RAM*0.7' -- //modules/... //cyber/... -//modules/drivers/canbu(11:38:02) INFO: Build completed successfully, 1 total action
>>> Automatically added entries to .gitignore to avoid problems.
>>> Automatically added //external workspace link:
    This link makes it easy for you--and for build tooling--to see the external dependencies you bring in. It also makes your source tree have the same directory structure as the build sandbox.
    It's a win/win: It's easier for you to browse the code you use, and it eliminates whole categories of edge cases for build tooling.
>>> Analyzing commands used in @//...
(11:38:02) ERROR: Error while parsing 'mnemonic('(Objc|Cpp)Compile',deps(@//...)) //modules/... //cyber/... -//modules/drivers/canbus/can_client/esd/... -//modules/perception/... -//modules/planning/open_space/trajectory_smoother:planning_block -//modules/map/pnc_map:cuda_pnc_util -//modules/map/pnc_map:cuda_util_test': unexpected token '//modules/...' after query expression 'mnemonic('(Objc|Cpp)Compile', deps(@//...))'

aquery failed. Command: ['bazel', 'aquery', "mnemonic('(Objc|Cpp)Compile',deps(@//...))", '--output=jsonproto', '--include_artifacts=false', '--ui_event_filters=-info', '--noshow_progress', '--features=-compiler_param_file', '--config=cpu', '--jobs=6', '--local_ram_resources=HOST_RAM*0.7', '--', '//modules/...', '//cyber/...', '-//modules/drivers/canbus/can_client/esd/...', '-//modules/perception/...', '-//modules/planning/open_space/trajectory_smoother:planning_block', '-//modules/map/pnc_map:cuda_pnc_util', '-//modules/map/pnc_map:cuda_util_test']
>>> Failed extracting commands for @//...
    Continuing gracefully...

The text was updated successfully, but these errors were encountered:

cpsauer · 2022-06-29T00:09:09Z

Hey @simon13579! Thanks for trying the tool and for writing in.

Looks like there's a pretty custom build flow here--and Bazel is helpfully telling us what's wrong. :refresh_all is already assuming you want all targets, but the list of targets is being respecified by ${formatted_targets}.

Could you try making that command bazel run @hedron_compile_commands//:refresh_all -- ${CMDLINE_OPTIONS} ${job_args} and seeing if that works?

That'll try to get the compile commands for all targets. If you're looking to get them for just some of the buildable targets, you can specify them with the refresh_compile_commands rule (see README). Looks like you'd want to start by specifying "//modules/... + //cyber/... -//modules/drivers/canbus/can_client/esd/... -//modules/perception/... -//modules/planning/open_space/trajectory_smoother:planning_block -//modules/map/pnc_map:cuda_pnc_util -//modules/map/pnc_map:cuda_util_test" in this case.

We don't currently have support for specifying the targets to analyze at runtime, as opposed to in a BUILD file, but I suppose we could add that if you'd like to. I think it should work, but we'd have to be careful to add + between targets. as we parse. We should probably at least warn about this case. I'll go ahead and add that warning.

For (4), I really want to dive into why the tool is forcing you to manually set exclude_headers. Any chance you could copy in the error message? I'd expect it to just default to None and be fine (as it does for me), and want to fix whatever's forcing you to make that change.

Cheers,
Chris

cpsauer · 2022-06-29T00:57:54Z

Added warnings to help with this case in 2539154
Documented the + syntax in 13e1359

simon13579 · 2022-06-29T05:54:30Z

Hello @cpsauer ,

Thanks for your kind support! For (4), if I don't set exclude_headers, it report the error:

[WARNING] No nvidia-driver found. CPU will be used.
[simon@in-dev-docker:/apollo]$ bash apollo.sh build
[WARNING] No nvidia-driver found. CPU will be used.
[WARNING] No nvidia-driver found. CPU will be used.
[INFO] Apollo Environment Settings:
[INFO]     APOLLO_ROOT_DIR: /apollo
[INFO]     APOLLO_CACHE_DIR: /apollo/.cache
[INFO]     APOLLO_IN_DOCKER: true
[INFO]     APOLLO_VERSION: HEAD-2020-09-21-e79f9d6765
[INFO]     DOCKER_IMG: dev-x86_64-18.04-20200914_0742
[INFO]     APOLLO_ENV:  STAGE=dev USE_ESD_CAN=false
[INFO]     USE_GPU: USE_GPU_HOST=0 USE_GPU_TARGET=0
[WARNING] No nvidia-driver found. CPU will be used.
[WARNING] No nvidia-driver found. CPU will be used.
[ OK ] Running CPU build on x86_64 platform.
[WARNING] ESD CAN library supplied by ESD Electronics doesn't exist.
[WARNING] If you need ESD CAN, please refer to:
[WARNING]   third_party/can_card_library/esd_can/README.md
[INFO] Build Overview:
[INFO]     USE_GPU: 0  [ 0 for CPU, 1 for GPU ]
[INFO]     Bazel Options: --config=cpu
[INFO]     Build Targets: //modules/... union //cyber/...
[INFO]     Disabled:      except //modules/drivers/canbus/can_client/esd/... except //modules/perception/... except //modules/planning/open_space/trajectory_smoother:planning_block except //modules/map/pnc_map:cuda_pnc_util except //modules/map/pnc_map:cuda_util_test
Starting local Bazel server and connecting to it...
(05:05:39) INFO: Current date is 2022-06-29
(05:05:40) ERROR: /apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/external/hedron_compile_commands/BUILD:7:25: @hedron_compile_commands//:refresh_all.py: invalid value in 'exclude_headers' attribute: has to be one of 'all' or 'external' instead of ''
(05:05:40) ERROR: error loading package '@hedron_compile_commands//': Package '' contains errors
(05:05:40) INFO: Elapsed time: 2.668s
(05:05:40) INFO: 0 processes.
(05:05:40) FAILED: Build did NOT complete successfully (1 packages loaded)
(05:05:40) FAILED: Build did NOT complete successfully (1 packages loaded)

And as you suggested, the below two methods both works for me which is great 👍 :

Run the command bazel run @hedron_compile_commands//:refresh_all -- ${CMDLINE_OPTIONS} ${job_args} which doesn't specify the target by ${formatted_targets}
Specify the target with the refresh_compile_commands rule

Thanks,
Simon

cpsauer · 2022-06-29T06:29:21Z

Great! Glad those worked. Sounds like you're fine with those fixes rather than needing to specify targets from the command line?

Thanks for the error message. Gah that's bad, and a bug I definitely didn't want to have slip through. Do you know what Bazel version you're on? If you use bazelisk, could I ask you to try (at least) bazel rolling and latest to see if we can narrow down what versions it's failing?

Thanks so much for your patience and tenacity in working through these.
Chris

simon13579 · 2022-06-29T13:06:15Z

Hello Chris,

Yes, those fixes are good enough for me, thanks for your great work!
The Bazel version I am on is 3.4.1.

[simon@in-dev-docker:/apollo]$ bazel version
Starting local Bazel server and connecting to it...
Build label: 3.4.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jul 14 06:27:53 2020 (1594708073)
Build timestamp: 1594708073
Build timestamp as int: 1594708073

Thanks,
Simon

cpsauer · 2022-06-29T22:15:32Z

Simon, you're very welcome! Glad to hear.

[I therefore won't add the feature to specify targets at runtime right now because I think we want to encourage people to configure things with a BUILD file, absent a burning need. I can however imagine how we'd add it in the future if it were needed. Note to future self: Be careful about merging flags that might each contain --.]

cpsauer · 2022-06-29T22:23:37Z

Re Bazel 3.4.1: That's quite an old version my friend. I can indeed reproduce the issue if I go back that far, but I think I should probably just be advising that you update, since lots of Bazel bugs have been fixed since then. Hopefully that works; I didn't see a .bazelversion file in ApolloAuto.

BTW, if you're using this tool for clangd, I think you'll probably end up wanting headers at some point. You'll know if you see erroneous red underlines in headers. But it's probably worth updating bazel and removing the exclude_headers line.

Anyway, I'll close for now, since I think we've resolved the issues. But you know how to find me.

cpsauer · 2022-08-20T01:17:08Z

@simon13579, I'm curious how this ended up! We had someone else run into the exclude_headers, old bazel version issue over in #73, so I'm wondering if that ended up being ok or a problem for you?

cpsauer · 2022-08-24T00:52:46Z

@simon13579, heads that @cyy5358 contributed a fix, now merged, that will let you not exclude_headers on older bazel versions if you so need!

cpsauer closed this as completed Jun 29, 2022

cpsauer mentioned this issue Aug 19, 2022

Error: invalid value in 'exclude_headers' attribute: has to be one of 'all' or 'external' instead of '' #73

Closed

eschumacher-s mentioned this issue Nov 2, 2022

Proto generated headers (*.pb.h) not included in compile_commands.json output #82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to run in the project ApolloAuto #62

Failed to run in the project ApolloAuto #62

simon13579 commented Jun 28, 2022

cpsauer commented Jun 29, 2022 •

edited

Loading

cpsauer commented Jun 29, 2022

simon13579 commented Jun 29, 2022

cpsauer commented Jun 29, 2022 •

edited

Loading

simon13579 commented Jun 29, 2022 •

edited

Loading

cpsauer commented Jun 29, 2022

cpsauer commented Jun 29, 2022

cpsauer commented Aug 20, 2022

cpsauer commented Aug 24, 2022

Failed to run in the project ApolloAuto #62

Failed to run in the project ApolloAuto #62

Comments

simon13579 commented Jun 28, 2022

cpsauer commented Jun 29, 2022 • edited Loading

cpsauer commented Jun 29, 2022

simon13579 commented Jun 29, 2022

cpsauer commented Jun 29, 2022 • edited Loading

simon13579 commented Jun 29, 2022 • edited Loading

cpsauer commented Jun 29, 2022

cpsauer commented Jun 29, 2022

cpsauer commented Aug 20, 2022

cpsauer commented Aug 24, 2022

cpsauer commented Jun 29, 2022 •

edited

Loading

cpsauer commented Jun 29, 2022 •

edited

Loading

simon13579 commented Jun 29, 2022 •

edited

Loading