IDE uses invalid Accept: header for board downloads

[DataGhost]

Real issue: https://bugs.openjdk.java.net/browse/JDK-8163921
Unfortunately, that hasn't been touched in two years and there is no easy way for me to bump that bug report, create a new one or get in touch with someone who can.

I forked the Arduino board definitions to my own personal Gitlab instance so I can easily keep my changes on top of upstream and have an easy update mechanism from the IDE. I put the package_index.json in a repository, accessible through .../-/raw/...file.json, and in there I pointed to archives that Gitlab generates from tags, through .../-/archive/.../file.zip. The fetch on raw/ works fine, but when I try to install my board definitions, Gitlab throws a 406 on archive/, and in the Gitlab logs, I see

"*; q=.2" is not a valid MIME type.

I verified that the Arduino IDE also does this on the request for raw/, so for some reason, Gitlab does not care about it on that endpoint, but that's out of scope here.

A simple workaround is to explicitly set a proper Accept: header for every request, even */* would suffice, or just the invalid header from the JDK (below) minus the offending part. This could be done in cc.arduino.utils.network.HttpConnectionManager.makeConnection, for example, but you probably know a better place to do it.

Relevant headers from the request:

User-agent: ArduinoIDE/1.8.12 (Windows 10; 10.0; x86; en) Java/1.8.0_191 (Oracle Corporation)
Accept: text/html, image/gif, image/jpeg, *; q=.2, /; q=.2

Log from the IDE:

Error downloading https://MYGITLAB/-/archive/master/arduino-avr-master.zip
java.lang.RuntimeException: java.lang.Exception: Error downloading https://MYGITLAB/-/archive/master/arduino-avr-master.zip
at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:175)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Error downloading https://MYGITLAB/-/archive/master/arduino-avr-master.zip
at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:149)
at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:83)
at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:60)
at cc.arduino.contributions.packages.ContributionInstaller.install(ContributionInstaller.java:105)
at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:172)
... 1 more
Caused by: java.io.IOException: Received invalid http status code from server: 406
at cc.arduino.utils.network.FileDownloader.openConnectionAndFillTheFile(FileDownloader.java:239)
at cc.arduino.utils.network.FileDownloader.downloadFile(FileDownloader.java:182)
at cc.arduino.utils.network.FileDownloader.download(FileDownloader.java:129)
at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:147)
... 5 more

[matthijskooijman]

Thanks for the report, looks good to fix.

I guess it would be good to send more specific Accept headers (e.g. application/json and application/zip, or whatever we need exactly and what their mimetypes are).

Related, showing a nicer error when a download fails might also be good.

[DataGhost]

Yeah, though in most cases it doesn't seem to matter in the IDE. But it would be more correct, yes.

Subtle nuance on my report, though, the ".2" part is also invalid, changing that to "0.2" makes it valid and accepted. So just dropping "*; q=.2" will not fix the issue. Just found this out while modifying the Accept header in my Apache reverse proxy. I rewrote the last two entries into "*/*; q=0.2" and that works.

# Java JDK fix (bug 8163921, arduino bug 10264)
RequestHeader edit Accept " \*; q=\.2, \*/\*; q=\.2" " */*; q=0.2"

[matthijskooijman]

Yup, the relevant bits of the spec are:

Accept = #( media-range [ accept-params ] )
accept-params  = weight *( accept-ext )
weight = OWS ";" OWS "q=" qvalue
qvalue = ( "0" [ "." *3DIGIT ] ) / ( "1" [ "." *3"0" ] )

(note that OWS means optional whitespace)
IOW, a q= should be between 0.000 and 1.000 (inclusive), and the leading digit is mandatory.

See also https://tools.ietf.org/html/rfc7231#section-5.3.1