Skip to content

Please prevent from scanning hidden directories #327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
d-a-v opened this issue Jun 30, 2019 · 0 comments · Fixed by arduino/arduino-cli#366
Closed

Please prevent from scanning hidden directories #327

d-a-v opened this issue Jun 30, 2019 · 0 comments · Fixed by arduino/arduino-cli#366
Labels
conclusion: resolved Issue was resolved topic: code Related to content of the project itself type: enhancement Proposed improvement

Comments

@d-a-v
Copy link

d-a-v commented Jun 30, 2019

Hi,

I have lots of git branches in my hardware .git/ directories (submodules too) and arduino-builder is now losing itself for too much time in them.

Details:

With -verbose -debug-level 10 the last message is

===debug ||| Ts: {0} - Running: {1} ||| [1561932660 WipeoutBuildPathIfBuildOptionsChanged]

I then analysed with linux strace what was happening and found that all time was spent scanning my .git/ directories in my hardware directory (I have lots of testing branches and not a very fast computer).
Changing permissions of the git directories solved the issue:

find . -name .git -exec chmod 000 {} \;

Changing them back to 700 made the issue appear again.

FWIW, another dirty workaround is:

while true; do rm -f /tmp/*/build.options.json; sleep 1; done

Is it an acceptable generic (kind-)request to prevent from scanning all directories whose names are starting with a dot . (''hidden'' directories under unix) (or at least .git/) ?
earlephilhower added a commit to earlephilhower/arduino-builder that referenced this issue Jul 1, 2019
@earlephilhower
Don't scan into hidden directories (.git, etc)
77c4be3 
When checking for updated core files, don't scan inside UNIX hidden
directories like .git.  Scanning inside a GIT tree for a large project
with many branches could take many minutes of runtime and multiple GB of
RAM.  This was seen in the Arduino core for the ESP8266, but is probably
also an issue for other core development teams.

Fixes arduino#327
earlephilhower added a commit to earlephilhower/arduino-builder that referenced this issue Jul 1, 2019
@earlephilhower
Don't scan into hidden directories (.git, etc)
5dadfe8 
When checking for updated core files, don't scan inside UNIX hidden
directories like .git.  Scanning inside a GIT tree for a large project
with many branches could take many minutes of runtime and multiple GB of
RAM.  This was seen in the Arduino core for the ESP8266, but is probably
also an issue for other core development teams.

Fixes arduino#327

Signed-off-by: Earle F. Philhower, III <[email protected]>
earlephilhower added a commit to earlephilhower/arduino-builder that referenced this issue Jul 1, 2019
@earlephilhower
Don't scan hidden directories (.git, etc)
2eed815 
When checking for updated core files, don't scan inside UNIX hidden
directories like .git.  Scanning inside a GIT tree for a large project
with many branches could take many minutes of runtime and multiple GB of
RAM.  This was seen in the Arduino core for the ESP8266, but is probably
also an issue for other core development teams.

Fixes arduino#327

Signed-off-by: Earle F. Philhower, III <[email protected]>
@earlephilhower earlephilhower mentioned this issue Jul 1, 2019
Closed
earlephilhower added a commit to earlephilhower/arduino-builder that referenced this issue Jul 1, 2019
@earlephilhower
Don't scan hidden directories (.git, etc)
44f482a 
When checking for updated core files, don't scan inside UNIX hidden
directories like .git.  Scanning inside a GIT tree for a large project
with many branches could take many minutes of runtime and multiple GB of
RAM.  This was seen in the Arduino core for the ESP8266, but is probably
also an issue for other core development teams.

Fixes arduino#327

Signed-off-by: Earle F. Philhower, III <[email protected]>
earlephilhower added a commit to earlephilhower/arduino-cli that referenced this issue Aug 28, 2019
@earlephilhower
Don't scan hidden directories (.git, etc.)
f5cb7d3 
Port of arduino/arduino-builder#328

When checking for updated core files, don't scan inside UNIX hidden
directories like .git.  Scanning inside a GIT tree for a large project
with many branches could take many minutes of runtime and multiple GB of
RAM.  This was seen in the Arduino core for the ESP8266, but is probably
also an issue for other core development teams.

Fixes arduino/arduino-builder#327

Signed-off-by: Earle F. Philhower, III <[email protected]>
@earlephilhower earlephilhower mentioned this issue Aug 28, 2019
Merged
earlephilhower added a commit to earlephilhower/arduino-cli that referenced this issue Aug 30, 2019
@earlephilhower
Don't scan hidden directories in arduino-builder
ff7f0d1 
Arduino-builder scans the core to determine if core config files have
changed between invocations.  Unfortunately, it goes into SCCS dirs
such as .git (which can have 100,000+ files).  This is wasted effort
and can cause massive amounts of runtime and memory use when core
developers are building in their git clones.

Use a new function already added by the builder/utils.go to determine
if a file or folder is SCCS, and if so skip it in the rebuild-required
check.

Fixes arduino/arduino-builder#327

Signed-off-by: Earle F. Philhower, III <[email protected]>
rsora pushed a commit to arduino/arduino-cli that referenced this issue Sep 5, 2019
@earlephilhower
Don't scan hidden directories in arduino-builder (.git, etc.) (#366)
84172d8 
* Don't scan hidden directories in arduino-builder

Arduino-builder scans the core to determine if core config files have
changed between invocations.  Unfortunately, it goes into SCCS dirs
such as .git (which can have 100,000+ files).  This is wasted effort
and can cause massive amounts of runtime and memory use when core
developers are building in their git clones.

Use a new function already added by the builder/utils.go to determine
if a file or folder is SCCS, and if so skip it in the rebuild-required
check.

Fixes arduino/arduino-builder#327

Signed-off-by: Earle F. Philhower, III <[email protected]>

* Add comment to force rebuild
rsora pushed a commit to arduino/arduino-cli that referenced this issue Sep 5, 2019
@earlephilhower
Don't scan hidden directories in arduino-builder (.git, etc.) (#366)
3be2287 
* Don't scan hidden directories in arduino-builder

Arduino-builder scans the core to determine if core config files have
changed between invocations.  Unfortunately, it goes into SCCS dirs
such as .git (which can have 100,000+ files).  This is wasted effort
and can cause massive amounts of runtime and memory use when core
developers are building in their git clones.

Use a new function already added by the builder/utils.go to determine
if a file or folder is SCCS, and if so skip it in the rebuild-required
check.

Fixes arduino/arduino-builder#327

Signed-off-by: Earle F. Philhower, III <[email protected]>

* Add comment to force rebuild
@rsora rsora added the type: enhancement Proposed improvement label Sep 22, 2021
@per1234 per1234 added conclusion: resolved Issue was resolved topic: code Related to content of the project itself labels Sep 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
conclusion: resolved Issue was resolved topic: code Related to content of the project itself type: enhancement Proposed improvement
Projects
None yet
Milestone
No milestone
4 participants
@d-a-v @per1234 @facchinm @rsora