Add file downloader cache to make faster the library/boards manager #9023

mattiabertorello · 2019-06-28T07:03:37Z

Add the cache for the library and boards files
Fix the delete of the files when something fail during the signature check
Add this configuration: http.connection_timeout_ms with 5000 as a default value, cache.enable with true as a default value
Add logging on local file Arduino15/logs/application.log (rotation policy: compress if the log is bigger than 50MB, delete when there are more than 20 compressed log files. See app/src/log4j2.xml)

Fixed issues: #8936, #6333, #7690 (comment)
Mitigated: #8988

The cache will consider the "Cache-Control" and the ETag headers.
So if the server delivers a file, setting a Cache-Control="private, max-age=14400" http header, the file will be cached for 4 hours without make any http request. When the cached file expires the download will make a HEAD request to check for ETag header changes, before making an actual download request.
Trying to emulate the browser behavior: https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching

The metadata of the cached files will be store in the Arduino15/cache/cache.json
example:

{
  "files" : [ {
    "eTag" : "a778ceeb5945afa31275495b8ccd5434",
    "remoteURL" : "https://downloads.arduino.cc/packages/package_index.json.sig",
    "localPath" : "/Users/mattiabertorello/Library/Arduino15/cache/downloads.arduino.cc/packages/package_index.json.sig",
    "md5" : "MD5:a778ceeb5945afa31275495b8ccd5434",
    "createdAt" : "2019-07-03T15:46:58.877",
    "cacheControl" : {
      "maxAge" : 14400,
      "public" : false,
      "private" : true,
      "mustRevalidate" : false,
      "noCache" : false,
      "noStore" : false,
      "noTransform" : false,
      "proxyRevalidate" : false,
      "smaxAge" : -1
    },
    "expires" : "2019-07-03T19:46:58.877"
  }]
}

per1234 · 2019-06-29T04:32:01Z

I did a test using a throwaway Boards Manager JSON file I published here:
https://gist.github.com/per1234/ed63fa1f88e7cff876620053378e895b/raw/80bc2b996db7fd878dfbe86dd2024490f57fecc6/package_foo_index.json
This file intentionally has a bad package file URL for the 1.0.4 release so installation is expected to fail. With Arduino IDE 1.8.10 Hourly Build 2019/05/21 09:33 I get this clear error message:

 Error downloading https://MCUdude.github.io/MicroCoreFoo/MicroCore-1.0.4.tar.bz2
java.lang.RuntimeException: java.lang.Exception: Error downloading https://MCUdude.github.io/MicroCoreFoo/MicroCore-1.0.4.tar.bz2
	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:175)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Error downloading https://MCUdude.github.io/MicroCoreFoo/MicroCore-1.0.4.tar.bz2
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:139)
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:73)
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:55)
	at cc.arduino.contributions.packages.ContributionInstaller.install(ContributionInstaller.java:101)
	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:172)
	... 1 more
Caused by: java.io.IOException: Received invalid http status code from server: 404
	at cc.arduino.utils.network.FileDownloader.downloadFile(FileDownloader.java:205)
	at cc.arduino.utils.network.FileDownloader.download(FileDownloader.java:132)
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:137)
	... 5 more

But with build 871 I get this:

 Error downloading https://MCUdude.github.io/MicroCoreFoo/MicroCore-1.0.4.tar.bz2
java.lang.RuntimeException: java.lang.Exception: Error downloading https://MCUdude.github.io/MicroCoreFoo/MicroCore-1.0.4.tar.bz2
	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:175)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Error downloading https://MCUdude.github.io/MicroCoreFoo/MicroCore-1.0.4.tar.bz2
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:142)
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:76)
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:58)
	at cc.arduino.contributions.packages.ContributionInstaller.install(ContributionInstaller.java:103)
	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:172)
	... 1 more
Caused by: java.nio.file.FileSystemException: C:\Users\per\AppData\Local\Arduino15\staging\packages\MicroCore-1.0.4.tar.bz2: The process cannot access the file because it is being used by another process.

	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
	at java.nio.file.Files.delete(Files.java:1126)
	at cc.arduino.utils.network.FileDownloader.downloadFile(FileDownloader.java:189)
	at cc.arduino.utils.network.FileDownloader.download(FileDownloader.java:129)
	at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:140)
	... 5 more

Note the error message now says "The process cannot access the file because it is being used by another process." instead of "Received invalid http status code from server: 404".

C:\Users\per\AppData\Local\Arduino15\staging\packages\MicroCore-1.0.4.tar.bz2 didn't exist on my computer before the installation attempt. It is a 0 byte file that was created during the installation. The error message "The process cannot access the file because it is being used by another process." makes me think of a conflict with an antivirus software's on-access file scanner, but this is not the case because I tried with the antivirus disabled and it still happens. I feel like this error message could be very misleading to someone trying to troubleshoot this problem because it acts as a distraction from the real problem that the file couldn't be downloaded, and also blocks the helpful HTTP response code from being shown.

I also tried with build 871 running in portable mode, with the same error message.

On my Linux machine, I get the "Received invalid http status code from server: 404" error from build 871.

- The file will be close before delete it - Some refactoring of the downloader cache

facchinm · 2019-07-01T10:56:02Z

@ArduinoBot build this please

mattiabertorello · 2019-07-01T15:21:12Z

I should fix the problem but I can't test on windows, could you test it ? @per1234
Thanks

per1234 · 2019-07-02T11:34:58Z

@mattiabertorello looks good. Thanks!

feikname · 2019-07-03T00:47:52Z

Does this solve #6333?

facchinm · 2019-07-03T14:16:02Z

@ArduinoBot build this please

ArduinoBot · 2019-07-03T14:38:04Z

✅ Build completed.

Please test this code using one of the following:

⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-873-linux32.tar.xz
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-873-linux64.tar.xz
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-873-windows.zip
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-873-macosx.zip
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-873-linuxarm.tar.xz

ℹ️ The linuxarm build is still experimental and may not be always available.

rsora · 2019-07-03T15:24:07Z

@feikname yes, this PR should address also your issue.
It would be great if you can help us in testing the preview binary available in the comment below (this one: #9023 (comment)) and attach in a comment here the logs (available in the Arduino15/logs/application.log folder as described in the PR summary) if you experience any sort of issues.
Thanks a lot!

edit : changed build download link due to some bugfixing

…nameUtils

Add the possibility to configure them `http.max_redirect_number`

and will be downloaded only one time

rsora · 2019-07-11T13:23:00Z

@per1234 consider the signature whitelisting a foundation for implementing in near future a 3rd party indexes security mechanism without breaking retro-compatibility.

Currently the signature check mechanism uses a public and private keys and consists in signing the Arduino's package_index.json server side with our private key, and checking the signature in the IDE using the Arduino's public key bundled in the IDE.

We are investigating in ways to make more secure also 3rd party cores indexes, maybe starting using our own key end ending up having some sort of CA mechanism similar to the one used by web browsers. But this is not so easy to implement so we implemented only the first bits of the feature.

I hope that this clarifies your doubts, otherwise, feel free to ask for details :)

red-scorp · 2019-07-12T06:56:37Z

Germany. Same error https://downloads.arduino.cc/packages/package_index.json.sig file signature verification failed. File ignored.
I put 882 in portable mode and this is my log:
application.log

rsora · 2019-07-15T15:36:00Z

@ArduinoBot build this please

ArduinoBot · 2019-07-15T15:57:57Z

✅ Build completed.

Please test this code using one of the following:

⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-885-linux32.tar.xz
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-885-linux64.tar.xz
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-885-windows.zip
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-885-macosx.zip
⬇️ http://downloads.arduino.cc/javaide/pull_requests/arduino-PR-9023-BUILD-885-linuxarm.tar.xz

ℹ️ The linuxarm build is still experimental and may not be always available.

…E 1.8.10 arduino/Arduino#9023 changed the Arduino IDE CLI's output. Due to this, when no boardsmanager.additional.urls preference is set, the output from Arduino IDE 1.8.10 (and onwards likely) caused priorBoardsmanagerAdditionalURLs to be set to "Set log4j store directory /home/travis/.arduino15". The solution was to check the exit status of the arduino --get-pref command and if it is 4 (Preference passed to --get-pref does not exist), overwrite the spurious value.

Mattia Bertorello added 2 commits June 28, 2019 09:02

Add the file downloader cache to make faster the library/boards manager

6592c42

Add slf4j, optimize some code and fix reported lint problem

8ca093b

mattiabertorello force-pushed the add-file-cache branch from 4b52a98 to 980df58 Compare June 28, 2019 15:30

Fix possible empty files during the download of the package index

d089323

mattiabertorello force-pushed the add-file-cache branch from 980df58 to d089323 Compare June 28, 2019 15:44

Mattia Bertorello added 3 commits June 28, 2019 17:49

Remove notNull annotation that failed the build

e6f0912

Convert CircularRedirectException in IOException

d3e7122

Add slf4j option to show the datetime in the logs

934df81

fix the misleading exception throw on windows

207128d

- The file will be close before delete it - Some refactoring of the downloader cache

arduino deleted a comment from ArduinoBot Jul 1, 2019

facchinm added this to the Release 1.8.10 milestone Jul 1, 2019

Add log4j dependencies

53be417

Mattia Bertorello added 4 commits July 3, 2019 15:25

Add log4j configuration

2596ece

Split download and check signature, add check signature to library index

5157688

Refactoring HttpConnectionManger and request-id

00818af

Add cache.json file and improve stability

a7d395f

mattiabertorello force-pushed the add-file-cache branch from f21584b to a7d395f Compare July 3, 2019 13:30

Change key from connection_timeout to connection_timeout_ms preferences

5dba31b

rsora mentioned this pull request Jul 3, 2019

package_index.json file signature verification failure for some users #8988

Open

arduino deleted a comment from ArduinoBot Jul 3, 2019

Mattia Bertorello added 2 commits July 3, 2019 18:02

Refactoring FileDownloaderCache

e1e4fb3

Use verifyDomain also for the library index

1bc994e

Mattia Bertorello added 3 commits July 11, 2019 14:32

Add commons-io dependency and replace the extract file name with File…

58fc5a5

…nameUtils

Increase the redirect to follow to 20

dde5668

Add the possibility to configure them `http.max_redirect_number`

Do not cache the core or the library because are too big

a8c7184

and will be downloaded only one time

mattiabertorello force-pushed the add-file-cache branch from eb143bd to 425affe Compare July 11, 2019 14:18

Reduce download method complexity of FileDownloader class.

1bfdf83

mattiabertorello force-pushed the add-file-cache branch from 425affe to 1bfdf83 Compare July 11, 2019 14:36

mattiabertorello force-pushed the add-file-cache branch 2 times, most recently from 024dab7 to 8b90f1a Compare July 12, 2019 11:35

Mattia Bertorello added 2 commits July 12, 2019 14:36

Delete cached file if the signature verify fail

2d04282

Change builder domain with https and add logging

85e91ef

mattiabertorello force-pushed the add-file-cache branch from 8b90f1a to 85e91ef Compare July 12, 2019 12:36

Mattia Bertorello added 2 commits July 12, 2019 14:53

Complete disable cache if the file is not a *_indexes

4da41f7

Fix linter problems

636f930

mattiabertorello force-pushed the add-file-cache branch from 34073d7 to 636f930 Compare July 12, 2019 13:41

Parse the old library index file also when the signature verify fail

3c5dbe6

facchinm mentioned this pull request Jul 18, 2019

TEST: fix proxies not being applied #8436

Closed

Merge branch 'master' into add-file-cache

9ce5101

facchinm merged commit 324a9bc into arduino:master Jul 18, 2019

per1234 mentioned this pull request Jul 30, 2019

Uninstall does not remove lib binaries arduino/arduino-cli#285

Closed

This was referenced Sep 18, 2019

Proxy configuration not saved #7573

Closed

How to use an (avr) core from git? #9237

Open

Downloaded boards disappeared #8936

Closed

List of libraries is re-downloaded everytime that the Library Manager is opened #6333

Closed

per1234 mentioned this pull request Aug 9, 2020

Please stop the library manager from updating the list of libraries EVERY time #10615

Open

per1234 mentioned this pull request Dec 24, 2020

Proxy error, getting Boards and Libraries updates #5887

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add file downloader cache to make faster the library/boards manager #9023

Add file downloader cache to make faster the library/boards manager #9023

mattiabertorello commented Jun 28, 2019 •

edited

Loading

per1234 commented Jun 29, 2019

facchinm commented Jul 1, 2019

mattiabertorello commented Jul 1, 2019

per1234 commented Jul 2, 2019

feikname commented Jul 3, 2019

facchinm commented Jul 3, 2019

ArduinoBot commented Jul 3, 2019

rsora commented Jul 3, 2019 •

edited

Loading

rsora commented Jul 11, 2019

red-scorp commented Jul 12, 2019

rsora commented Jul 15, 2019

ArduinoBot commented Jul 15, 2019

Add file downloader cache to make faster the library/boards manager #9023

Add file downloader cache to make faster the library/boards manager #9023

Conversation

mattiabertorello commented Jun 28, 2019 • edited Loading

per1234 commented Jun 29, 2019

facchinm commented Jul 1, 2019

mattiabertorello commented Jul 1, 2019

per1234 commented Jul 2, 2019

feikname commented Jul 3, 2019

facchinm commented Jul 3, 2019

ArduinoBot commented Jul 3, 2019

rsora commented Jul 3, 2019 • edited Loading

rsora commented Jul 11, 2019

red-scorp commented Jul 12, 2019

rsora commented Jul 15, 2019

ArduinoBot commented Jul 15, 2019

mattiabertorello commented Jun 28, 2019 •

edited

Loading

rsora commented Jul 3, 2019 •

edited

Loading