Support multiple version of a given library

[lmihalkovic]

The thought here is for LIB MANAGER to allow several version of a given library and let users choose when they want to move their code up/down.

At the moment libraries are stored in something like this

libraries
   libABC
       file1
       file2
   libDEF
       file1
       file2

Supporting something like the following would allow different versions to coexist together without forcing all user sketches to share a single version

libraries
   libABC
       1.0.1
           file1
           file2
       1.0.8
           file1
           file2
   libDEF
       2.0.0
           file1
           file2

#4064 has more info

[ffissore]

How is user supposed to choose which library version to use?

[lmihalkovic]

The is the best question...

There is a conflict between the amount of work required to implement, and the amount of work for the users. Would be good to find a compromise somewhere in the middle...

The lib index code is all in one spot.. so the implementation on the side side should should spread around. The leaves the user... for every user happy with use-latest-unless-I-tell-you, there will be others that have a compelling reason to say it is not good for them.
I was thinking that something like the following heuristic would allow a gradual implementation over time

• global configurable IDE setting : "Multi Lib version support: [ ]Always latest [ ]Always oldest"
• behavior modifiable per sketch via a general choice for this sketch (inside the dependencies.txt I proposed)

{ 
  // This is the very minimal file that just addresses this
  resolution {
    versions: OLDER_FIRST // or  LATEST_FIRST
  }  
}

With the minimal amount of work (and some VERY MINIMAL work on the libmanager so that it does not crash on the numbered folders), a started implementation can be rolled in (users can manually create the folders to start using the feature).

• then in a separate phase, the lib manager can be modified to assist with managing the folders.
• then in a separate phase the dependencies.txt file can be enhanced with EXPLICIT dependencies

{ 
  ' this is a simplified JSON file following the Ninja format 
  depends [
    {
      name: Adafruit_NeoPixels
      version: 1.0.8              // in the short term this can be ignored if you don't want to allow side by side versions to exist
      source: SKETCHBOOK      // this is part of the link order proposal from #4064  -  done separately
    }
  ]
  resolution {
     order: [BUILTIN, PLATFORM_PARENT, PLATFORM_TARGET, LIBMANAGER, SKETCHBOOK, SKETCH]
    conflicts: WARN 
    versions: OLDER_FIRST 
  }  
}

• then in a separate phase, a per-sketch config screen can be implemented (either template+validator as a tab the editor, or right click in the outline-view, or popup inside the editor.
  'template+validator' basically means treating the file like a normal file, like I currently do for the lib specific descriptors:
  

When I write 'separate phase' it is just to show that these are tasks that allow a gradual delivery of the complete scheme, rather than a all or nothing feature.

[ffissore]

Apparently, you missed the point: where and what the user clicks when something goes bad?
For example: what if a user opens an old sketch from a custom sketchbook and the "dependent" library at the specified version is gone, while the same library but with a different version is present?

[NicoHood]

Then you could search via library manager for an old version (because you saved the url before)
or you just post an error. Because if you send a sketch with IDE 1.0.5 dependenc to someone else you have the same problem. And if a sketch relies on an old library you have the problem too. with ot without this feature.

But if you saved both versions you can have multiple sketches work with multiple versions of a library. This makes it easier to provide dev versions. Then the user dont have to fix on a very old library version, because his main project requires it and his "try and error debug" sketch want to try out new features.

[lmihalkovic]

Apparently, you missed the point: ...

I have yet to encounter a situation where this kind of start leads to productive outcomes.. but let's try.

The first post was leaving some details out... let me retrace my steps, trying to be more specific.

Everything hinges on retroffiting the UI with 2 new internal properties. I only explicitly referred one, but the 'dependencies.txt' snippet I pasted shows lets you infer the existence of the second one. I was and still am of the opinion that these should be accessible with new config panel entries. These are the settings in question (they are represented as check boxes for simplicity, but they could as well be drop-downs):

Multi library version default selection: [ ]Always latest [ ]Always oldest
Library version conflicts:  [ ]WARNING  [ ]ERROR

These setting drive how the IDE handles the scenario you described. In the absence of sketch specific instructions:

• ALWAYS_LATEST means the IDE will take any matching lib (based solely on name as is the case now), and use the most recent version in all cases regardless of how versions exist
• ALWAYS_OLDEST is for a more defensive approach, where the IDE will simply use the smallest version number, regardless of how many other versions are accessible
  The second setting (what I had labelled conflicts in dependencies.txt) is to tell how the user wants to be informed, with either a WARNING level message before the compilation starts, or by stopping the compilation. At the minimum, two visually distinct simple messages with no attempt to stop the compilation would be simpler to implement, and potentially enough?!

This is for the very bare minimum support... minimum workload to implement, low level of commitment for the future, and quite extensible.

Then we get into the more elaborate support for the feature. The global behavior is now customizable per sketch via entries in dependencies.txt. Again here, a minimum first step is to simply have the same properties (see my first snippet) configured manually, nothing else... However extended capabilities can be provided by storing a version number requirement for the libs. This level of feature coverage can still be done with minimal work in the IDE as well as in the arduino-builder.

The next level of functionality is where the more benefits come for users, with

• the ability to manage multiple versions in the Lib Manager window (I thought about a simple scheme again requiring minimal changes in order to allow users to delete specific versions of a lib - but again... can never be done in the IDE is deemed to risky - more can be said here, but there is no point yet if we do agree on some basics first)
• the availability of some editing capabilities in the IDE

Again trying to keep the implementation workload light, my snapshot hints at a very simply initial approach that would treat dependencies.txt like I am currently treating keywords.txt and library.properties when editing the code for a library: a simple text editor tab with some sanity checking to prevent silly situations. This capability relies on code I wrote that makes it simple to specialize Editor.java based on the nature of the file being edited (I shared some of that today on a different issue and was thinking about sharing the rest via PR).

To be exhaustive on this last topic, I believe there is a case to be made for externalizing the capability to alter the editor behavior in the form of JavaScript code, the idea being that it would be easier to perform some small changes if there is no need to recompile the entire IDE, not to mention that users could do it themselves (I will try it out soon).

Hope this is clearer.

[lmihalkovic]

No matter what, this is just a suggestion for how to support multiple versions. I do not think it is the only way to do so, but they are a compromise, and IMHO not the worst possible choice. Another alternative could be to have a 'link' (either a real symlink or something like the .url files on windows) that 'points' to the 'active' version of a lib (managed manually or in lib mgr). Another alternative is to name the folders Adafruit_NeoPixel-1.0.3, Adafruit_NeoPixel-1.0.5 directly, and keep the straight Adafruit_NeoPixel name for the currently active lib. Yet another way would be following the OSX scheme.

Regardless of how, IMHO side-by-side versions of the same lib is useful. After all .NET, Un*x, OSX, ... all support it. I was just trying to share some idea for providing user value at the lowest implementation cost (ergo soon).

[ffissore]

You're too focused on making the implementation simple and are missing the most important goal: provide a tool that is easy to use for novices. Indeed, allowing multiple versions of libraries, making the IDE guess which one to use automagically, add yet-another checkbox for trying to control its behaviour, all this makes it harder to use it
What you're talking about is commonly achieved using different sketchbooks. Which is also a very good thing since it promotes clean and nice workspaces: allowing multiple library versions instead will promote creating sketchbooks cluttered with folders
Closing as wontfix

[ffissore]

I love your enthusiasm @lmihalkovic but I think you're using it wrong. Please think about final users instead of which java file to edit.

[lmihalkovic]

Looking at some of the things you have done and some of the many complaints you have closed in the past from other users, it is almost comical to read this

You're too focused on making the implementation simple and are missing the most important goal: provide a tool that is easy to use for novice users

Reminds me of what happens in declining empires when successive generations inherit things too big for their shoulders.

[lmihalkovic]

linked to

• lib manager overrides core libraries included with 3rd party boards #4064 (current version issue after an update)
• Can't verify/upload with directory junctions in the Arduino libraries folder #4402 (creative ways people have to use multiple versions)

[KiLLeRRaT]

I too think this is necessary. For ideas or insperation, look at how npm supports multiple versions of packages, or how nuget does it.