Don't scan hidden directories in arduino-builder (.git, etc.) #366
Conversation
|
|
|
I see this change is already in the new -cli. Thanks! Closing. |
|
Sadly, |
|
Reopening as a WIP. I'll need to find the core scanner as well and apply the same sort of logic that's being done on the sketch directory. |
earlephilhower
changed the title
|
Can you rebase on |
|
Sure thing. I'll do the rebase by EOD my time. I also noted in the Sketch file-filtering routine that |
Arduino-builder scans the core to determine if core config files have changed between invocations. Unfortunately, it goes into SCCS dirs such as .git (which can have 100,000+ files). This is wasted effort and can cause massive amounts of runtime and memory use when core developers are building in their git clones. Use a new function already added by the builder/utils.go to determine if a file or folder is SCCS, and if so skip it in the rebuild-required check. Fixes arduino/arduino-builder#327 Signed-off-by: Earle F. Philhower, III <[email protected]>
earlephilhower
force-pushed
the
skiphidden
branch
from
f5cb7d3 to
ff7f0d1
Compare
earlephilhower
changed the title
|
@masci, thanks for updating the tag. I had to get a new Go build environment before I could rebase/fix the PR. I can confirm @d-a-v's observation that git head of both repos does result in the 4.5GB RAM and 3:00++ min runtime, and that this PR as-is fixes the issue in my testing. The new PR simply hooks the scans that arduino-builder already applies when looking at source folders to be scanned when checking that the Arduino |
|
CI failure looks like a Codecov token issue, not related the the PR (I hope!). |
|
@masci Is it possible re-trigger CI for this PR ? |
|
git seems to be in an inconsistent state, checkout is failing with this message: https://github.com/arduino/arduino-cli/pull/366/checks?check_run_id=209807044#step:3:346 This seems to happen to others as well, see actions/checkout#23 but I'm afraid there's nothing I can do to fix this. @earlephilhower can you try adding another commit to your branch, or pick the commit into a brand new branch? (no idea if this will change anything, just speculation) |
|
Thanks, @masci. I've pushed in a no-op change and am seeing the GH actions working now. We'll see in a few minutes if we're all good. |
|
@earlephilhower we're good now. The remaining failure is bad copypasta (fat fingers are mine), this line https://github.com/arduino/arduino-cli/blob/master/.github/workflows/test.yaml#L80 should be added here as well https://github.com/arduino/arduino-cli/blob/master/.github/workflows/test.yaml#L66 |
|
Is the copypasta fix something you want in this PR? Seems like it's better to have it in a separate push for sanity's sake, then I would just merge upstream/master and the commit logs would be comprehensive. |
|
@earlephilhower you can rebase on |
|
Just a ping to say the WIP label can be removed at your convenience. With your fix it's building and running cleanly now on my system and in CI. |
* Don't scan hidden directories in arduino-builder Arduino-builder scans the core to determine if core config files have changed between invocations. Unfortunately, it goes into SCCS dirs such as .git (which can have 100,000+ files). This is wasted effort and can cause massive amounts of runtime and memory use when core developers are building in their git clones. Use a new function already added by the builder/utils.go to determine if a file or folder is SCCS, and if so skip it in the rebuild-required check. Fixes arduino/arduino-builder#327 Signed-off-by: Earle F. Philhower, III <[email protected]> * Add comment to force rebuild
|
Many thanks for the fix, and for merging it quickly. Performance increase is just tremendous. |
Port of arduino/arduino-builder#328
When checking for updated core files, don't scan inside UNIX hidden
directories like .git. Scanning inside a GIT tree for a large project
with many branches could take many minutes of runtime and multiple GB of
RAM. This was seen in the Arduino core for the ESP8266, but is probably
also an issue for other core development teams.
Fixes arduino/arduino-builder#327
Signed-off-by: Earle F. Philhower, III [email protected]