Are librarymanager and boardmanager links still clickable with IDE v2?

[PaulZC]

Describe the problem

With IDE v1.8 Library Manager, we could include clickable links in examples which would open the library manager or board manager automatically.

With IDE v2, the only option seems to be to follow the link in a web browser, which then fails:

My question is: is there a new format for clickable library and board manager links in examples?

Thanks!
Paul

To reproduce

Add a clickable link to an example and click it:

//Click here to get the library: http://librarymanager/All#SparkFun_u-blox_GNSS

Expected behavior

The library manager used to open (with v1.8) - with v2 it does not... (Sad times!)

Additional context

No response

Issue checklist

• I searched for previous reports in the issue tracker
• My report contains all necessary details

[kittaakos]

I need the spec on how this should work in IDE2. @cmaglie or @per1234, could you please link how the pathname and hash should be translated to a library/boards manager query? Thank you!

Update:

• Should this work only in .ino files,
• .ino + .pde,
• any additional files,
• only in source files,
• in header files,
• or the combination of any from above.

[per1234]

Here is the specification @kittaakos:

https://arduino.github.io/arduino-cli/dev/sketch-specification/#libraryboards-manager-links

So path will have a value of either librarymanager or boardsmanager.
The hash is the search term that should be pre-filled in Library Manager or Boards Manager.

I guess it should be supported in any files. Arduino IDE supports it in .ino, header, and source files, but there is no precedent for Markdown and AsciiDoc because Arduino IDE 1.x doesn't open those. But I don't see any harm in supporting it in those files as well. So I would go with whatever is the easiest to implement and maintain.

[kittaakos]

Here is the specification @kittaakos:

https://arduino.github.io/arduino-cli/dev/sketch-specification/#libraryboards-manager-links

So path will have a value of either librarymanager or boardsmanager. The hash is the search term that should be pre-filled in Library Manager or Boards Manager.

Thank you! Unfortunately, the spec does not cover @PaulZC's request:

http://librarymanager/All#SparkFun_u-blox_GNSS

First,librarymanager and boardsmanager are the authority of the URL. Here, pathname is All and not librarymanager/boardsmanager. I assume that is the equivalent of the Type dropdown from the IDE2.

Secondly, should the URL resolver support Topic for the library manager? If yes, how?

[PaulZC]

Hi @kittaakos (& @per1234 ),

Thank you for looking at this.

Something to think about, regarding the hash, is how underscores are interpreted. I believe underscores should be interpreted as spaces when performing the search. We'd like to get back to the place where //http://librarymanager/All#SparkFun_u-blox_GNSS finds the SparkFun u-blox GNSS library.

Very best wishes,
Paul

[per1234]

pathname is All and not

I forgot about that. I left it out of the sketch specification because it never worked (it might appear to work from the "All" example, but that is the default selection of the menu). You can throw anything you like in that path and "All" will still be selected in Arduino IDE 1.x.

But it is in use out in the wild so the URIs with path must at least be supported.

I assume that is the equivalent of the Type dropdown from the IDE2.

It seems like it from the DropdownUpdatableLibrariesItem/DropdownUpdatableCoresItem names in the code, but I can't follow it from there so @cmaglie or @facchinm would have to verify:

arduino/Arduino@b99ab40

should the URL resolver support Topic for the library manager?

The real use case for this is to help the user to install a specific library dependency of a sketch. The problem is that the keyword search might return multiple libraries, so it helps, but still leaves some room for ambiguity. Adding Topic filtering helps with that, but doesn't solve it. For example, if I was to do something like this:

// Install the Servo library via Library Manager
// if using the Arduino IDE, click here: http://librarymanager/All/Device%20Control#Servo

The user is still going to get a lot of results. I think the sorting for exact match proposed in #1106 would be much more helpful for this use case than the Topic filter.

---

I haven't seen any examples of use cases of these URIs that would benefit from the intended use of the Topic filter. For example:

// If you would like to see a list of display libraries created by SparkFun or for SparkFun products, click here:
// http://librarymanager/All/Display#SparkFun

Do you see a use case for being able set the Topic filter via the URI @PaulZC?

[PaulZC]

The real use case for this is to help the user to install a specific library dependency of a sketch.

Agreed. That's absolutely it. The user can copy a code snippet from a SF Hook-Up Guide and - with one click - install the relevant library or board package.

Do you see a use case for being able set the Topic filter via the URI?

Sure. That would be really nice-to-have. The Sensors category would be a huge one for us. Thanks!!

[kittaakos]

I understand the importance of backward compatibility, and I can make IDE2 resolve the links as IDE 1.x, but the specification should be improved in the long run. It has flaws.

• Why http scheme?
• Why not use a query string for the query (instead of enumerating Type and Topic). The lib manager would force users always to include the Type (All) before the Category.
• Query strings must be URL encoded instead of using any magical _ to ' ' mapping.

Something like this could be better than the 1.x version:

arduino://manager/library/?topic=Sensor&type=All&query=SparkFun%20u-blox%20GNSS

IDE2 could help to generate such URLs.

[per1234]

Something like this could be better than the 1.x version

I agree. The old system was poorly designed.

magical _ to ' ' mapping.

There isn't any mapping in the link system. Those underscores end up in the search field. It is the Arduino IDE 1.x Library Manager search that is able to find the libraries even with the underscores.

[kittaakos]

There isn't any mapping in the link system. Those underscores end up in the search field.

Thank you for the clarification 👍

[kittaakos]

@PaulZC, could you please help us with the verification? Here is the PR, here is the doc how to get the beta build from GH. If something is missing from the supported cases, let us know. Thank you!

[PaulZC]

Hi @kittaakos ,

This is excellent! Thank you for working on it so quickly. But there are two gremlins I can see:

Gremlin1:

With the old IDE search tool, underscores were - I think - equivalent to spaces. With your fix, that's no longer true... See code and screenshots below.

Dilemma! Do we (A) do it the right way and go with your changes as-is. Or do we (B) persuade you to add a nasty work-around so that the search sees underscores as equivalent to spaces, and vice-versa?

I'm thinking option (A) is probably the right way to do this. Even though it means we've got to go through ALL of our code snippets in MANY places and change underscores to %20...

Thoughts please?

Gremlin2:

Once you have selected a link that includes a Topic, the Topic 'sticks' when you go back to a plain "All" (Type All, no Topic). You have to manually change (e.g.) Topic->Display back to Topic->All to see the non-Display results.

I suggest defaulting Topic to All if no Topic is specified.

Thanks again,
Paul

void setup() {}
void loop() {}
// Additional Board Manager URL:
// https://raw.githubusercontent.com/sparkfun/Arduino_Apollo3/main/package_sparkfun_apollo3_index.json
//
// http://boardsmanager/All#SparkFun_Apollo3
// http://boardsmanager/All#SparkFun%20Apollo3
// http://librarymanager/All#SparkFun_u-blox_GNSS
// http://librarymanager/All#SparkFun%20u-blox%20GNSS
// http://librarymanager/All/Display#SparkFun
// http://librarymanager/All/Sensors#SparkFun
// http://librarymanager/Updatable/Display#SparkFun

[kittaakos]

I appreciate your help, @PaulZC.

With the old IDE search tool, underscores were - I think - equivalent to spaces. With your fix, that's no longer true... See code and screenshots below.

From #1442 (comment):

magical _ to ' ' mapping.

There isn't any mapping in the link system. Those underscores end up in the search field. It is the Arduino IDE 1.x Library Manager search that is able to find the libraries even with the underscores.

I see. What do you think, @per1234? Could you help figure out with the community how this should work? Since IDE2 does not run any search, we might need to adjust the CLI part too.

add a nasty work-around so that the search sees underscores as equivalent to spaces, and vice-versa?

If it works in IDE 1.x it should also work with IDE2.

In the long run, I believe we have to come up with a better spec and deprecate the IDE 1.x behavior. Once the 1.x URL spec is deprecated, it must still work in IDE2, but IDE2 should reveal it as a warning and provide a 'quickFix' textDocument/codeAction to convert the old format to the new one. So no manual update will be required by the users.

Once you have selected a link that includes a Topic, the Topic 'sticks' when you go back to a plain "All" (Type All, no Topic). You have to manually change (e.g.) Topic->Display back to Topic->All to see the non-Display results.

I suggest defaulting Topic to All if no Topic is specified.

It's a bug. Thank you for spotting it. I update the PR.

[PaulZC]

Ah... We do need the 'quickFix'. The %20 does not work on IDE 1.8.19:

[kittaakos]

Currently blocked by arduino/arduino-cli#1895.

[kittaakos]

I updated the PR. As described here, IDE2 escapes the _ with spaces ( ). Note that the <input> in the library/boards manager view must use spaces instead of underscores. Otherwise, it won't work.

See it in action.

IDE2:

escape_underscore_ide2.mp4

IDE 1.x:

escape_underscore_ide1.mp4

Other than this hack, IDE2 cannot do much without furher CLI support.

[cmaglie]

for reference here's how the Arduino IDE 1.8.x do the search, I'm going to implement the same search:

  private boolean filterCondition(ContributedLibraryReleases lib) {
    if (selectedCategoryFilter != null && !selectedCategoryFilter.test(lib)) {
      return false;
    }

    ContributedLibrary latest = lib.getLatest();
    String compoundTargetSearchText = latest.getName() + " "
                                      + latest.getParagraph() + " "
                                      + latest.getSentence();
    if (latest.getProvidesIncludes() != null) {
      compoundTargetSearchText += " " + latest.getProvidesIncludes();
    }
    if (!stringContainsAll(compoundTargetSearchText, selectedFilters)) {
      return false;
    }

    return true;
  }

  /**
   * Check if <b>string</b> contains all the substrings in <b>set</b>. The
   * compare is case insensitive.
   *
   * @param string
   * @param filters
   * @return <b>true<b> if all the strings in <b>set</b> are contained in
   *         <b>string</b>.
   */
  private boolean stringContainsAll(String string, String filters[]) {
    if (string == null) {
      return false;
    }

    if (filters == null) {
      return true;
    }

    for (String filter : filters) {
      if (!string.toLowerCase().contains(filter.toLowerCase())) {
        return false;
      }
    }

    return true;
  }

[kittaakos]

I have updated the PR to include a lib search change from the CLI. All is looking good for the libraries. IDE2 is waiting for a similar change for the platforms, then IDE2 can provide the same behavior to users as IDE 1.x. Please bear with us.

[PaulZC]

Thank you @kittaakos & @cmaglie ,

Please bear with us

No problem!

[PaulZC]

Thank you @kittaakos @per1234 @cmaglie - #1481 works perfectly for me.
Sincere best wishes,
Paul