Support additional libraries

[spoenemann]

When opening the "KeyboardAndMouseControl" example, clangd does not find the Keyboard and Mouse symbols.

Do we need additional include paths? How can we get them from the sketch configuration?

[rsora]

As you can see here:
https://github.com/arduino/arduino-cli/blob/master/arduino/libraries/libraries_location.go

The CLI includes the following library paths during the compile process

Sketchbook
sketchbook/libraries/....
PlatformBuiltIn -> path/to/the/core/avr/1.5.0/libraries/...
ReferencedPlatformBuiltIn -> path/to/a/core/built/on/top/avr/1.5.0/libraries/ a custom core that is built on top arduino officials cores
IDEBuiltIn -> path/all/ide/java/libraries/...

Let me check the code to see if those informations are provided explicitly somewere or they need to be exposed in some way

BTW naive question, those libraries are not included in the Example sketch, did you install them before using clang?

[spoenemann]

I loaded the example provided by the old Arduino IDE. The Keyboard and Mouse libraries are shown as installed there.

Currently I'm using the --show-properties flag to get compiler flags. It would be great if all include paths could be found in those properties.

https://github.com/bcmi-labs/arduino-language-server/blob/c32ed1943cba708fc05f3724bb3cf23c8e95390f/handler/builder.go#L85

[spoenemann]

I get the following error when I try to preprocess the KeyboardAndMouseControl example:

$ arduino-cli compile --fqbn arduino:avr:uno --preprocess KeyboardAndMouseControl.ino

KeyboardAndMouseControl.ino:24:22: fatal error: Keyboard.h: No such file or directory
compilation terminated.
Error during build: exit status 1

[spoenemann]

I understood that the Uno board does not work with the KeyboardAndMouseControl example. Now I used arduino:sam:arduino_due_x_dbg instead and the CLI works. However, the properties I obtain with the --show-properties flag do not contain any include path for the Keyboard and Mouse libraries.

When I click on Verify in the IDE, I see the following in the Output view:

Detecting libraries used...
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/sketch/KeyboardAndMouseControl.ino.cpp -o /dev/null
Alternatives for Keyboard.h: [[email protected]]
ResolveLibrary(Keyboard.h)
  -> candidates: [[email protected]]
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/sketch/KeyboardAndMouseControl.ino.cpp -o /dev/null
Alternatives for HID.h: [[email protected]]
ResolveLibrary(HID.h)
  -> candidates: [[email protected]]
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/sketch/KeyboardAndMouseControl.ino.cpp -o /dev/null
Alternatives for Mouse.h: [[email protected]]
ResolveLibrary(Mouse.h)
  -> candidates: [[email protected]]
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src -I/Users/spoenemann/Documents/Arduino/libraries/Mouse/src /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/sketch/KeyboardAndMouseControl.ino.cpp -o /dev/null
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src -I/Users/spoenemann/Documents/Arduino/libraries/Mouse/src /Users/spoenemann/Documents/Arduino/libraries/Keyboard/src/Keyboard.cpp -o /dev/null
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src -I/Users/spoenemann/Documents/Arduino/libraries/Mouse/src /Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src/HID.cpp -o /dev/null
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src -I/Users/spoenemann/Documents/Arduino/libraries/Mouse/src /Users/spoenemann/Documents/Arduino/libraries/Mouse/src/Mouse.cpp -o /dev/null
Generating function prototypes...
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -w -x c++ -E -CC -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src -I/Users/spoenemann/Documents/Arduino/libraries/Mouse/src /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/sketch/KeyboardAndMouseControl.ino.cpp -o /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/preproc/ctags_target_for_gcc_minus_e.cpp
/Users/spoenemann/Library/Arduino15/packages/builtin/tools/ctags/5.8-arduino11/ctags -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/preproc/ctags_target_for_gcc_minus_e.cpp
Compiling sketch...
/Users/spoenemann/Library/Arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -MMD -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src -I/Users/spoenemann/Documents/Arduino/libraries/Mouse/src /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/sketch/KeyboardAndMouseControl.ino.cpp -o /var/folders/lx/_zc3wg6d2px3lqdwj8d35jvr0000gn/T/arduino-sketch-647BD920C1206A9070C1F514DE932FA7/sketch/KeyboardAndMouseControl.ino.cpp.o

So the CLI seems to resolve the additional libraries when compiling, but not when I just do --show-properties.

How can I use the CLI to give me the required include paths? In this case I need to know where to find

• libraries/Keyboard/src
• libraries/Mouse/src
• libraries/HID/src

[spoenemann]

My preferred solution would be to have a --show-flags option that outputs all compiler flags (except the input/output files) exactly as used in a normal compile run.

So when calling

arduino-cli compile --fqbn arduino:sam:arduino_due_x_dbg --show-flags KeyboardAndMouseControl.ino

I would get

-c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -MMD -mcpu=cortex-m3 -mthumb -DF_CPU=84000000L -DARDUINO=10607 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON "-DUSB_MANUFACTURER=\"Arduino LLC\"" "-DUSB_PRODUCT=\"Arduino Due\"" -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/libsam -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/CMSIS/Include/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/system/CMSIS/Device/ATMEL/ -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/cores/arduino -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/variants/arduino_due_x -I/Users/spoenemann/Documents/Arduino/libraries/Keyboard/src -I/Users/spoenemann/Library/Arduino15/packages/arduino/hardware/sam/1.6.12/libraries/HID/src -I/Users/spoenemann/Documents/Arduino/libraries/Mouse/src

[spoenemann]

Related: bcmi-labs/arduino-editor#63
In that case we would need the CLI to output a JSON file as specified here:
https://clang.llvm.org/docs/JSONCompilationDatabase.html

[spoenemann]

The current approach of including all libraries found in the sketch container seems to work quite well. We can open another issue if there are more problems related to this.

[spoenemann]

There are still sketches that are compiled without errors by the CLI, but the language server reports errors, e.g. due to unresolved symbols.

I created arduino/arduino-cli#557 to develop and discuss the idea of an additional CLI flag that I described above.

[cmaglie]

This has been fixed with a combination of:

• implementing the compile_commands.json support in arduino-cli
• using clangd --query-drivers capabilities
• updating to the latest clangd that has support for avr architecture