Boards Manager URLs incorrect HTTP GET Range header field

[kosza]

I am using Arduino 1.8.3.
I added http://arduino.esp8266.com/stable/package_esp8266com_index.json to Boards Manager URLs and got a downloading error.

I checked the traffic and discovered that the HTTP request is incorrect: Range: bytes=26000-
It should be Range: bytes=0- or it shouldn't be.

Here is the http request and reply:
GET /versions/2.3.0/package_esp8266com_index.json HTTP/1.1
User-agent: ArduinoIDE/1.8.3 Java/1.8.0_121
Range: bytes=26000-
Host: arduino.esp8266.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, /; q=.2
Connection: keep-alive

HTTP/1.1 416 Requested Range Not Satisfiable
X-Powered-By: Express
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Last-Modified: Sat, 06 May 2017 03:02:19 GMT
ETag: W/"6590-15bdbb53a78"
Content-Type: text/html; charset=utf-8
Content-Range: bytes */26000
X-Content-Type-Options: nosniff
Content-Length: 459
Date: Thu, 17 Aug 2017 06:23:05 GMT
Connection: keep-alive

RangeNotSatisfiableError: Range Not Satisfiable
   at SendStream.error (/home/nodeuser/apps/download/node_modules/express/node_modules/send/index.js:275:31)
   at SendStream.send (/home/nodeuser/apps/download/node_modules/express/node_modules/send/index.js:618:19)
   at onstat (/home/nodeuser/apps/download/node_modules/express/node_modules/send/index.js:677:10)
   at Object.oncomplete (fs.js:107:15)

Regards,
Kosza

[facchinm]

mmmh, wrong project?
All the node output is surely not part of Arduino. Are you sure you are using IDE 1.8.3

Sorry I totally misread the post

[facchinm]

Hi @kosza , can you paste here the output from the IDE console when the download fails?
I'm unable to reproduce it with the same link, so I think it could be caused by some network problems on your side (or related with the json providers).

[kosza]

Only this:
Error downloading http://arduino.esp8266.com/stable/package_esp8266com_index.json

[facchinm]

But if you click the link you are able to see the json, right? What happens if you add, for example, https://adafruit.github.io/arduino-board-index/package_adafruit_index.json ?

[kosza]

From browser I see the correct json.
The problem is that the Arduino try to download the file from 26000th byte.
It is interesting that your Arduino can download. May be there is a configuration but I didn't find it.
Adafruit's json downloaded correctly. The Range was bytes=0-.

Any ideas?

[facchinm]

Only one idea; maybe there is a stray file from a previous download attempt.
You could try removing that file from Arduino15 folder (on Windows: C:\Users\yourUsername\AppData\Local\Arduino15 , on osx /Users/youruser/Library/Arduino15, on Linux ~/.arduino15 ). In that folder you should find some json files (and their temporary copies and signatures). Remove them all and restart the IDE and let me know if it worked 😉

[kosza]

You have right. I have deleted the jsons and now IDE downloaded the whole file.
Before I deleted the files the esp's json file was 26000 bytes. So it means the IDE tried to download the new file contents. Why didn't IDE download file from the beginning?

Thanks for your help.

[facchinm]

This is a good question. There is some code here

Arduino/arduino-core/src/cc/arduino/contributions/DownloadableContributionsDownloader.java

Line 68 in 31a9029

if (!Files.isRegularFile(outputFile, LinkOption.NOFOLLOW_LINKS) || (Files.size(outputFile) < contribution.getSize())) {

to handle resumed downloads, probably we should add a check to automatically delete the file if the download starts failing.

[kosza]

I checked DownloadableContributionsDownloader.java, I think it is ok.

// Open file and seek to the end of it
file = new RandomAccessFile(outputFile, "rw");
initialSize = file.length();
file.seek(initialSize);

I think you can remove the following code from line 165 and 186 in Arduino/arduino-core/src/cc/arduino/utils/network/FileDownloader.java
connection.setRequestProperty("Range", "bytes=" + initialSize + "-");
And of course also need to remove line 149 in same.
file.seek(initialSize);

BTW
I did many test. All result were that IDE downloaded the file from 0th byte and replaced the existed file. (I decrease/ingreased the size of the file, I deleted the file.)

[facchinm]

Yep, it enters that bogus condition only in a few very particular conditions... Since jsons are pretty small, I'd download them from 0 everytime to make sure this problem never happens again (instead, I'd leave the feature for bigger files like cores and toolchains)

[jayaramanrp]

Only one idea; maybe there is a stray file from a previous download attempt.
You could try removing that file from Arduino15 folder (on Windows: C:\Users\yourUsername\AppData\Local\Arduino15 , on osx /Users/youruser/Library/Arduino15, on Linux ~/.arduino15 ). In that folder you should find some json files (and their temporary copies and signatures). Remove them all and restart the IDE and let me know if it worked 😉

it worked !!!!!1

[ReeceRobinson]

This is still a problem at least on my Mac with arduino 1.8.7. Library Manager freezes when trying to add a library.

[Minesbuildlab]

Can some somebody help me with the issue showed above. I tried removing all of the files and it still does nothing to help download the packages. In my IDE in the board manager... It refuses to download anything.
Error downloading https://downloads.arduino.cc/packages/package_index.json java.lang.RuntimeException: java.lang.Exception: Error downloading https://downloads.arduino.cc/packages/package_index.json at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onUpdatePressed$1(ContributionManagerUI.java:150) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.Exception: Error downloading https://downloads.arduino.cc/packages/package_index.json at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:139) at cc.arduino.contributions.packages.ContributionInstaller.download(ContributionInstaller.java:333) at cc.arduino.contributions.packages.ContributionInstaller.downloadIndexAndSignature(ContributionInstaller.java:307) at cc.arduino.contributions.packages.ContributionInstaller.updateIndex(ContributionInstaller.java:285) at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onUpdatePressed$1(ContributionManagerUI.java:146) ... 1 more Caused by: java.net.SocketException: Network is unreachable: connect at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) at cc.arduino.utils.network.FileDownloader.downloadFile(FileDownloader.java:181) at cc.arduino.utils.network.FileDownloader.download(FileDownloader.java:132) at cc.arduino.contributions.DownloadableContributionsDownloader.download(DownloadableContributionsDownloader.java:137) ... 5 more

[ArduinoSk]

Only one idea; maybe there is a stray file from a previous download attempt.
You could try removing that file from Arduino15 folder (on Windows: C:\Users\yourUsername\AppData\Local\Arduino15 , on osx /Users/youruser/Library/Arduino15, on Linux ~/.arduino15 ). In that folder you should find some json files (and their temporary copies and signatures). Remove them all and restart the IDE and let me know if it worked

worked, thank you