Detect programmers

[PeterWone]

It is possible to detect Arduinos connected via serial USB. It would be excellent to be able to detect programmers.

[ubidefeo]

@PeterWone
Programmers can be detected by their tools (openocd, JLink, avrdude, st-util etc).
In order to detect one using the CLI we should implement case-by-case workflows (still based off an FQBN) and I think it's not something "needed" by this project.
Always up to be convinced of the contrary, but you'll have to bring all your friends :D

[PeterWone]

@ubidefeo -- To share my own perspective, I'm writing a VSCode extension that integrates arduino-cli into the IDE because I am unsatisfied with the robustness of the Microsoft supplied equivalent. If I do the case by case workflows in my extension then they are only useful to VSCode users. If it happens in arduino-cli then command line users get the benefit.

The only question remaining is who does the work. Taking care of integration with VS Code is quite enough for me, and I don't doubt that you have plenty to occupy you.

[ubidefeo]

@PeterWone
I like the idea of a VS Code extension for our CLI, I just don't think it's necessary to auto-detect a connected programmer.
If you look at the scene of hardware development kits suppliers you'll notice that everyone detects their own programmer being connected or not, and that is easy, because they're only looking for them and they mostly have network discovery protocols or registered VID/PID pairs that allow them to simply scan for a few options.
Imagine a scenario in which you have to keep scanning for every possible programmer an ATMega328 supports

Id                     Programmer name
usbGemma               Arduino Gemma
usbtinyisp             USBtinyISP
arduinoisp             ArduinoISP
jtag3isp               Atmel JTAGICE3 (ISP mode)
jtag3                  Atmel JTAGICE3 (JTAG mode)
atmel_ice              Atmel-ICE (AVR)
buspirate              BusPirate as ISP
avrispmkii             AVRISP mkII
arduinoisporg          ArduinoISP.org
usbasp                 USBasp
avrisp                 AVR ISP
parallel               Parallel Programmer
arduinoasispatmega32u4 Arduino as ISP (ATmega32U4)
stk500                 Atmel STK500 development board
arduinoasisp           Arduino as ISP

many of them have USB to Serial interface controllers that have generic VID/PID used by plenty of boards (even custom ones) for which the manufacturer doesn't think it's worth it to pay for their own Vendor/Product IDs.
In the specific case, an Arduino used as ISP will be recognised as and Arduino board.

To automate this discovery would mean testing every possible programming tool for every possible programmer, and doing this over and over until something is discovered.
Filtering down allowing the user to pre-select a programmer and seeing when this is connected or disconnected kinda defeats the purpose of auto-discovery of a programmer.

Hope you'll see why I'm skeptical about such a feature.

Now on to more interesting topics: developing your own VS Code extension.
Besides the fact that the current MS Arduino plugin won't work with CLI, what is it about it you don't like?
I'm collecting information around on how people use our CLI, and am curious to know your opinion :)

[PeterWone]

1. Every time a VS Code release uses a new version of node, every extension that uses node native integration breaks until it and all its dependencies are recompiled for the right version of node.
2. Their code is complex in ways the CLI makes unnecessary. As a result they are unable to be responsive to user suggestions and requests.
3. It depends on the presence of the Arduino IDE (I think they're using it to compile)
4. There is absolutely no automation of paths for libraries. The vast majority of Arduino users are new to programming and are not ready to troubleshoot pathing problems for the VS Code C++ language support in order to get Intellisense code completion and edit-time static checking. But it's dead easy to ask your CLI where libs and cores are, and from that I can fix their C++ header paths for the workspace (the folder containing the ino file).
5. Serial monitor: if the ino file sets a speed for Serial then that is the speed the serial monitor should use. It's really not hard to look for Serial.begin\(([0-9]+)\) but they don't.
6. If there is exactly one board detected then autoselect it. But they don't.
7. If you don't specify a satisfied path for arduino-cli then I can download and install it. They don't.
8. I can ask arduino-cli to download and install cores and libs in response to user selections.

I got my system going by manual installation of all the bits and pieces. It took all evening to understand what I needed, how it all fits together and how to get the bits in place. None of it was hard but there was a fair bit to take in and it would have been overwhelming for someone with less development experience. I mostly use Nanos so I can hard code this

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Compile",
      "type": "shell",
      "command": "arduino-cli compile --fqbn arduino:avr:nano",
      "problemMatcher": [
        "$gcc"
      ]
    },
    {
      "label": "Flash via Serial",
      "type": "shell",
      "command": "arduino-cli upload -p COM6 --verbose --fqbn arduino:avr:nano",
      "problemMatcher": [
        "$gcc"
      ],
      "dependsOn": [
        "Compile"
      ]
    },
    {
      "label": "Flash via Programmer",
      "type": "shell",
      "command": "arduino-cli upload -P usbasp --verbose --fqbn arduino:avr:nano",
      "problemMatcher": [
        "$gcc"
      ],
      "dependsOn": [
        "Compile"
      ]
    }
  ]
}

That's easy and doesn't take long to set up once you know what you're doing... but the bar's set pretty high for newbies.

[ubidefeo]

Every time a VS Code release uses a new version of node, every extension that uses node native integration breaks until it and all its dependencies are recompiled for the right version of node.

I hadn't thought of this

Their code is complex in ways the CLI makes unnecessary. As a result they are unable to be responsive to user suggestions and requests.

I believe the extension has been developing over time and handles a lot of cases because of the next point

It depends on the presence of the Arduino IDE (I think they're using it to compile)

Yes, they use kind of a "hidden" feature which is the internal arduino-builder. With the new CLI we're trying to replace its functionalities to make it a more modern tool which allows advanced users (and automation enthusiasts) to do more in a more structured (hopefully elegant) way. Of course we also have to maintain a lot of legacy behaviour which makes us sometimes introduce regressions people are not too happy about, that is why we still consider this tool an Alpha

There is absolutely no automation of paths for libraries. The vast majority of Arduino users are new to programming and are not ready to troubleshoot pathing problems for the VS Code C++ language support in order to get Intellisense code completion and edit-time static checking.

this is a rather tough one we are working on for the new IDE, and Language Servers have rather complex workflows. Because of how Arduino tends to simplify a user's life by automating, collecting, assembling, pre-compiling, some things are not easy to achieve, but we're convinced we're close to a solution that will allow correct library and source indexing for code assist

Serial monitor: if the ino file sets a speed for Serial then that is the speed the serial monitor should use. It's really not hard to look for Serial.begin(([0-9]+)) but they don't.

This is something we also don't do in our own IDE, and although I can see it could be useful it requires code scanning on a window basis, don't set your hopes high for this :D

If there is exactly one board detected then autoselect it. But they don't.

with this we go back to scanning serial ports on the machine for connected serial devices, which can be easy on windows but on Linux/Mac you'd have several devices only one of which might be a bootloader-ready board. Check out my arduino-cli board list output, for example, and consider I have no board attached

Port                            Type        Board Name FQBN Core
/dev/cu.Bluetooth-Incoming-Port Serial Port Unknown
/dev/cu.Bosone-SPPDev           Serial Port Unknown
/dev/cu.Bosone-SPPDev-1         Serial Port Unknown
/dev/cu.Bosone-SPPDev-2         Serial Port Unknown
/dev/cu.Bosone-SPPDev-3         Serial Port Unknown
/dev/cu.Bosone-SPPDev-4         Serial Port Unknown
/dev/cu.Bosone-SPPDev-5         Serial Port Unknown
/dev/cu.Bosone-SPPDev-6         Serial Port Unknown
/dev/cu.JBLFlip5-vCOMM          Serial Port Unknown

the CLI or an IDE have no way to know wether this is an Arduino compatible board.
The same is true for our IDE(s)

If you don't specify a satisfied path for arduino-cli then I can download and install it and likewise for cores and libs. They don't.

I don't think I get this point :)

[PeterWone]

I guess I'm pitching at a seamless UX for the inexperienced, and a convenient one. My ambition is that if you install my extension, it will sort out all of the dependencies for you. A more finished, "consumer" experience. Some of these observations don't apply to people who use unix. I put it to you that (a) they are more technically sophisticated and don't need help and (b) they are a minority. I could be wrong about (b) but I have no way to check. At any rate if there is only one detected board and it is serial (usb) then the user is almost certainly using Windows and it is almost certainly the right device to flash.

[ubidefeo]

I guess I'm pitching at a seamless UX for the inexperienced, and a convenient one. My ambition is that if you install my extension, it will sort out all of the dependencies for you. A more finished, "consumer" experience.

I see where you come from, but why would your target user use VS Code rather than our future IDE which is going to tackle pretty much what you're aiming for? (also it's based on Theia/VS Code)
Have you tried it?

Some of these observations don't apply to people who use unix. I put it to you that (a) they are more technically sophisticated and don't need help and (b) they are a minority. I could be wrong about (b) but I have no way to check.

You're probably right :D

At any rate if there is only one detected board and it is serial (usb) then the user is almost certainly using Windows and it is almost certainly the right device to flash.

In the majority of cases this is probably true.
I'd invite you to use our Alpha IDE, which currently has some broken things (mostly in Language Server space, we're reworking that part out) https://github.com/arduino/arduino-pro-ide

[ubidefeo]

@PeterWone , download the Nightly Build which has the latest fixes.
We'll release new features in a few weeks

[matthijskooijman]

but we're convinced we're close to a solution that will allow correct library and source indexing for code assist

@ubidefeo, can you comment on the direction of this solutoin? I'm interested also from the perspective of #849, about generating a compilation database, which is one (maybe partial) solution to this problem.

[PeterWone]

why would your target user use VS Code rather than our future IDE

Versatility. People sometimes do things other than programming Arduinos.

It is hard to convey the versatility of VS Code to someone unfamiliar with its ecosystem. It is far and away the most popular web development tool on three platforms. Just the things that I use it for personally include

• direct and very good SCM integration defaulting to git with sophisticated diff and merge tooling
• integration with SQL Server, MySQL, Oracle etc
• language support (syntax colouring, autocompletion, parameter hints etc) for C, C++, C#, JS, TS, half a dozen dialects of SQL, HTML, CSS, all the CSS preprocessors and markdown
• step through debug support designed for extension. Someone just released an extension that integrates gdb so now you can set breakpoints and step through code and inspect variables for anything that supports gdb. There are debugger integration extensions for Chrome, Edge and Firefox, and obviously for dotnet core.
• Live Share which lets multiple developers connect to a shared edit/debug session for collaborative debugging. You can "follow" a particular user for an experience similar to screen sharing except you can both edit at the same time, or not follow and work on totally different files. Also supports voice plus text chat although I don't use that. Only the host system needs the source and the dependencies.

Without wishing to disrespect your efforts, you cannot hope to compete with that-- but you don't need to. I'm not kidding when I say I think you'd be better off producing language support for VS Code.

[ubidefeo]

@matthijskooijman the team is investigating exactly that, I always keep an eye on your interactions ;)
We're working on integrating a language server as a VS Code Extension in Theia and people who are more in the know that I can ever be are exploring generating a compile db to feed to clangd.
A little patience, we have a lot on our plate ;)

[ubidefeo]

@PeterWone I know very well what you mean.
I use VS Code off and on for other things, but most of my workflow is based on Sublime Text and CLI.
Talking about gdb integration I currently use VS Code to debug Arduino code running on SAMD devices and it's great :)

I take no offence in your comment, and we don't dare competing with VS Code. We are looking at several opportunities and exploring different paths.
Most of the components that turn Theia into the new Arduino IDE are getting a lot of thinking from our team, but because we are still in this phase we cannot fully reveal what we're looking into not to create any expectations.
I would hate to make promises and not being able to maintain them, and I'm convinced that in this stage, for a little more time, we can do a better job focusing on our short term goals and release updates to our tools that keep our (alpha/beta) users happy and engaged.
We take every interesting suggestion and proposal very seriously, and this issue is already pinned to our board.
I hope you'll want to keep engaging and follow development closely

[matthijskooijman]

We're working on integrating a language server as a VS Code Extension in Theia and people who are more in the know that I can ever be are exploring generating a compile db to feed to clangd.

Cool, thanks for sharing :-)

[PeterWone]

@ubidefeo - I wonder whether you would mind sharing email addresses to answer occasional questions while I'm figuring out integration with VS Code. Questions like this one: does each core bring its own complete toolchain or are there platform dependencies that must be pre-installed like eg gcc or msvc?

Questions like this are a bit out of scope for arduino-cli issues. If you are interested in corresponding my email is peter dot wone at outlook dot com.

[ubidefeo]

hey @PeterWone
communicating in a private way would hinder productivity and possibly generate the need to curate content back into GH issues, I'd like to keep this kind of conversations open to a larger audience, especially because we have some excellent collaborators in this repo and the IDE's who can contribute with questions, challenges and clarifications.

Questions like this one: does each core bring its own complete toolchain or are there platform dependencies that must be pre-installed like eg gcc or msvc?

You can have dependencies and a lot more. Our documentation wizard @per1234 has been compiling a great reference and there's everything you need to know. Your question will probably find an answer here https://arduino.github.io/arduino-cli/latest/platform-specification/#core-reference

Questions like this are a bit out of scope for arduino-cli issues

@per1234 is this the right place for these questions? If not, which one is?

[PeterWone]

You are right, the docs fellow is almost certainly the right point of contact for clarifications of the documentation. Thanks.

[ubidefeo]

@PeterWone when you have questions you can't find answers to in that doc just ask around here, or maybe @per1234 prefers it to be posted to the forums somewhere?