IDE startup takes too long when a soft link into big directory structures exist in the "sketchbook location" directory

[Ho-Ro]

Describe the bug
The IDE startup takes forever when links into complex directories with thousands of subdirectories exist in "sketchbook location" directory structure.

To Reproduce
Steps to reproduce the behavior:

1. Go to the sketchbook storage directory (in my case cd ~/Arduino/Projects).
2. Create a link into a complex directory structure with circular links.
3. Start the ide (beta5).
4. The IDE opens, but the splash screen is displayed forever, warnings are shown:

root WARN Could not load sketch from ...
root WARN Could not load sketch from ... 
...
root WARN Could not load sketch from ...
root WARN Frontend EditorNavigationContribution.onStart is slow, took: 450821.8 ms

5. Startup takes almost 8 minutes.
6. After removing the critical link the IDE starts much faster.

Expected behavior
The splash screen closes fast, and the IDE works normally - independent of the project directory structure.

Desktop

• OS: debian stable with KDE
• Version: arduino-ide beta 5

EDIT: Especially the circular links that I use for structured documentation purposes (e.g. TOP->../.. or A->B and B->A or even more complex with three and four hops) slow down the startup. Obviously the IDE tries to traverse the whole tree and visits the directories over and over again until the max. path length is reached - then it stops. During this traversal it spits out thousands of warnings :(

2nd EDIT: This problem did not occur in beta.3, beta.4 could not be tested on my Debian stable system due to #259.

3rd EDIT: Related to beta.4 issue #269?

[willie68]

Is there any improvment in sight?
Todays measurement: (i9, 32GB, M2 SSD)
2.1.0 time to get the first sight after start: 45 sec, time until i can code: 1:45
1.8.19: time to get the first sight after start: 6 sec, time until i can code: 0:45
Both with the same workspace/sketchbook.
(All measured after a reboot, so both without chached files)

and for fun: VS Code (with activated support of JS, Java and Golang, all in one project)
time to get the first sight after start: 3 sec, time until i can code: 0:15

It's even getting worther than ever...
Don't get me wrong, I've been actively testing 2.x since the first public beta because I like the functionality. But that's just a no-go for me as a hobbyist. And for my students (arduino4kids project) these loading times are simply incomprehensible and disrupt the flow of learning.

[ubidefeo]

@willie68
it really depends on how big/deep we're talking and what the purpose of this might be.
I have 214 Sketches in the sketchbook folder.
The library folder contains 263 libraries.
IDE 2.1 starts in 5 seconds and it's ready to code.

I don't use symlinks to folders because it's not the way you're supposed to work in the Arduino context,
and also using symlinks creates issues when syncing file-systems.
Also... why would one not try to avoid circular links? 🤔

What is your specific context?
I'm trying to identify a need in order to investigate a possible solution :)

Thanks
u.

[Ho-Ro]

because it's not the way you're supposed to work in the Arduino context

But it is my way to organize my work, e.g. symlink to a doc directory with datasheets that point to some papers in another section that points to the datasheets.
One of this circles is enough to confuse the 2.0 while 1.x is fine...

and also using symlinks creates issues when syncing file-systems.

Not necessary - set "do not follow symlinks".

[ubidefeo]

Is the symlink inside your sketch or in your sketchbook folder?

[ubidefeo]

I created a symlink into my sketchbook and linked it to my system drive root, which itself has some symlinks being my whole OS.
I am not able to reproduce the issue.

if you can post the result of a tree operation on your folder maybe I have more elements

[willie68]

I don't think symbol links are the problem. I have 65 folder in the sketchbook, some with subfolder for different projects. (Mainly because many projects live in git repos) no symbol links! I think round about 200 projects. But the libraries folder is crowded. 265 libs, but mostly named arduino-xxxxxx which are older versions of the libs. And that will cause the problem. The ide has to scan every libs folder for the infos of keywords and so on. And the lib manager lacks the function of deleting older version. And its a mess when you tried to do it manually. The other part is, scanning the folder seems to be a syncron task. If you implement that to async this would help much. That's how professional IDE do such scanning.

[ubidefeo]

oh, now I get it
legacy libraries need a lot more work, that is a known problem.
scanning the libraries is also required to populate the examples menu, and that's something that professionals IDEs don't need to do.
This IDE is targeted to Arduino projects and we have a lot of legacy to respect in order not to break things.
That said, optimising it and making it better is a clear goal of ours :)

Our backlog is large, and the team is small...
Bear with us 🙏🏼

[willie68]

I'm willing to help, I'm a pro, but my problem is that one has switched to Elektron and JS. That's not my preferred language. I'm more of a backend guy, Java, Golang, C++...
I'm just doing a few smaller things in JS, little Vue sites and stuff like that, but no bigger projects.

[Ho-Ro]

For me the 2.1.0 improves the behaviour, startup time until I can start working is 15 seconds - faster than 2.0.0 and much faster than the .rc versions.

ok for me -> close

Thx to the whole team for your effort!

[per1234]

@willie68 is there a soft link into a big directory structure under your sketchbook folder (the path configured via the "Sketchbook location" IDE preference)?

@Ho-Ro do find that there has been a regression in the IDE in regards to this issue since the time of your previous report that the problem was resolved?

[Ho-Ro]

o find that there has been a regression in the IDE in regards to this issue since the time of your previous report that the problem was resolved?

I tested again after a fresh boot of my system and it took again a very long time. The next try was much faster due to the already buffered disk content. Version 1.x is still faster and more responsive.