Specifying --build-path inside the sketch directory is not handled correctly

[kalj]

Bug Report

Current behavior

If I compile my project twice with the --build-path argument specifying a directory inside the sketch directory, the second time it fails.

I run

arduino-cli compile --fqbn arduino:avr:mega --build-path build-mega

First time:

Sketch uses 8396 bytes (3%) of program storage space. Maximum is 253952 bytes.
Global variables use 1451 bytes (17%) of dynamic memory, leaving 6741 bytes for local variables. Maximum is 8192 bytes.

But second time:

Error during build: unable to read contents of the source item: open <mydir>/build-mega/compile_commands.json: no such file or directory

This confusing message is completely irrelevant since that file was in fact existing since last compilation. However, as part of the recompilation, the whole build directory was wiped but an entry to compile_commands.json (and other files) remained in the program state which lead to the program trying to read it.

Expected behavior

I expect repeated builds to work with the --build-path specifying a directory inside the sketch directory.

Environment

• CLI version (output of arduino-cli version): arduino-cli Version: 0.20.2 Commit: 1378381 Date: 2021-12-09T13:32:05Z
• OS and platform: Ubuntu 21.10, Lenovo Thinkpad P14s with AMD Ryzen 7 PRO 5850U

Additional context

The code I am trying to build: https://github.com/kalj/gep

---

Note: the original title of this issue was "--build-path results in empty/invalid compile_commands.json"

[unixbigot]

Partial workaround is to remove compile_commands.json before rebuilding, but then I see a similar error (this time the file does not exist, rather than being an empty JSON array)

Error during build: unable to read contents of the source item: open /Users/chris/Documents/Microcode/hammerhead/.build/espressif.esp32.ttgo_t7_v13_mini32/preproc/ctags_target_for_gcc_minus_e.cpp: no such file or directory
make: *** [.build/espressif.esp32.ttgo_t7_v13_mini32/hammerhead.ino.bin] Error 1

[unixbigot]

Another workaround is to build with --clean, which of course just as counterproductive as nuking the build directory. No that fails too.

Anyone know offhand what version introduced this? An inadvertent 'brew upgrade' has landed me this busted version.

[gtorino]

I suffer the same issue. Any chance to have this solved?

[kalj]

Just dug a little bit more into this. Turns out it is not true that the compile_commands.json file is created empty the first time I compile. It is in fact created correctly with valid contents. However, the second time I run, the file seems to be removed at the time arduino-cli is looking for it. The file only gets this empty-array contents after this second run -- the one where it errors out. Continuing the investigation...

[kalj]

Found out some more information:

• It seems that the problem is that on the one hand, the compile_commands.json file gets included in Sketch.AdditionalFiles (set up here: https://github.com/arduino/arduino-cli/blob/master/legacy/builder/container_setup.go#L59-L65).
• However, then it figures out that it should wipe the build directory, here: https://github.com/arduino/arduino-cli/blob/master/legacy/builder/wipeout_build_path_if_build_options_changed.go#L32 After this point, the compile_commands.json file is gone.
• Finally, it comes to ContainerMergeCopySketchFiles here: https://github.com/arduino/arduino-cli/blob/master/legacy/builder/container_merge_copy_sketch_files.go#L26 where it wants to SketchCopyAdditionalFiles, here:https://github.com/arduino/arduino-cli/blob/master/arduino/builder/sketch.go#L109
  That's when it tries to read a non-existing file.

[kalj]

More experimentation done. Two new key observations:

1. It is clear there is nothing special about compile_commands.json - I get a similar error for other files if I first remove compile_commands.json before rebuilding. E.g., I just now got:

Error during build: unable to read contents of the source item: open <path-to-my-sketch>/build-mega/preproc/ctags_target_for_gcc_minus_e.cpp: no such file or directory

I think we got the message for compile_commands.json simply because it happens to be the first file it tries to copy.

2. It only happens if the specified build directory is inside the sketch directory, e.g.

cd <path-to-my-sketch>
arduino-cli compile --fqbn arduino:avr:mega --build-path build-mega

So I guess arduino-cli gets confused since it thinks the files inside the build dir are part of the sketch.

[kalj]

Anyways, I think the Issue title and description needs to be updated. A more proper description is probably:

Specifying --build-path inside the sketch directory is not handled correctly

If a build path is specified inside the sketch directory, arduino-cli seems to get confused and think that build output files such as compile_commands.json are part of the build input, which leads to subsequent recompilations failing.

Should I update this issue or file a new one? @cmaglie @per1234

[per1234]

Hi @kalj. Thanks for the additional investigation!

Should I update this issue or file a new one?

I think it is best to update the issue.

[kalj]

@per1234 great! I have now updated the title and the description.

[umbynos]

The issue is still there with version 0.29.0

[kelmcc]

Still experiencing the same issue in 0.31.0. The same thing as described above - the first time everything generates fine and then the second time I get:
Error during build: unable to read contents of the source item: open xxx/output/compile_commands.json: no such file or directory

[umbynos]

Hi @kelmcc, yeah, that's because the change of #2084 is not yet part of a release. But please feel free to test it using the latest nightly build

[brunolalb]

Still experiencing the same issue in 0.31.0. The same thing as described above - the first time everything generates fine and then the second time I get: Error during build: unable to read contents of the source item: open xxx/output/compile_commands.json: no such file or directory

check microsoft/vscode-arduino#1630
the problems seem to be related