HTTP client gets randomly stuck, with its stream's available() returning 0 almost forever

[teo1978]

I have the following code (I have simplified it to post it here) that downloads a file from an URL and streams it to some physical output.

I read the downloaded data a few bytes at a time and use it as I download it beacause it's to big to store into the RAM (around 250 kB).

  uint8_t httpbuf[128] = {0};
  
  HTTPClient http;
  
  String url="http://some.url/";
  
  int httpcode=200;
  int len=0;
  
  http.begin(url);
  httpcode=http.GET();
  WiFiClient* httpstream = http.getStreamPtr();
  if (httpcode==200) {
    len=http.getSize();
  
    while (len > 0 || len ==-1) {
      int gotbytes=0;
    
      if (!http.connected()) {
        break;
      }
    
      size_t avl=httpstream->available();
      
      if (avl) {
        gotbytes = httpstream->readBytes(httpbuf, ((avl>sizeof(httpbuf))?sizeof(httpbuf):avl));
        Serial.println("Got some bytes");
      }
      else {
        Serial.println("No bytes available");
        delay(100); // can change this to any value, the result doesn't change.
      }
      if (gotbytes>0) {
        if (len>0) len=max(0, len-gotbytes);
        
        // use the data in httpbuf to generate some sort of physical output
        
      }
    }
    
    
  }
  else {
    // http error
  }

What happens is that often, randomly, httpstream->available() starts returning always 0 for very long periods of time.

This is not a network issue or a server issue, as I have never, ever observed anything like this on any kind of http client before.

The expected output would be lots of Got some bytes and some No bytes available from time to time, eventually resulting in downloading the whole file.

The observed result is either:

1. I start getting dozens of consecutive No bytes available, with some Got some bytes from time to time, so the download is extremely slow, but eventually succesfully finishes, or it suddenly "wakes up" and resume normal speed (i.e. I stop getting "No bytes available" and get mostly "got some bytes" until the end)
   or
2. I start getting No bytes available forever, until eventually the connection gets closed, so http.connected() return false and breaks the loop, without finishing the download.

P.S. until this is fixed, suggestions for a workaround would be highly appreciated.

[devyte]

@teo1978 when you opened this issue, you were presented with an issue template that contains specific instructions and requested info. You have ignored this.
I am keeping this open only because it rings a bell in my mind, though I can't place why yet.
Please edit your post, and make the sketch Complete (i.e.: it can be built and loaded directly on a board) and Verifiable (add a specific url that shows the problem). Please make sure to keep it Minimal, i.e.: do NOT post your entire project.

[teo1978]

when you opened this issue, you were presented with an issue template that contains specific instructions and requested info. You have ignored this.

Yep, I have provided the information that I have, and that I had the time to provide.

Whether or not you are willing to do something with it is, of course, entirely up to you.

[devyte]

It's not about what I do with the info, as this is not a service provided to users, but rather a community effort. The issue you encountered affects you, so you're the primary interested party for pursuing a fix. If you don't have the time to follow the minimum instructions in the issue template, then reporting this was just a waste of your own time, as well as mine.
I invite you again to complete the code above, so that someone else can take a look. If you're not willing or able, then please close this.

[teo1978]

It's not about what I do with the info, as this is not a service provided to users, but rather a community effort

Exactly, this is not a service provided to the user and I am not here asking for help. I am reporting an issue hoping this helps the developers (or any contributor) fix it and hence improve the library. I provided as much information as I have and spent as much effort as I am willing to spend. It may be useful or it may be not.

I don't expect anybody to step in right away and start looking at the issue and fix it. I don't think closing the issue would be a sensible thing to do either, but that's up to the criteria of whoever maintains the project.

The issue you encountered affects you, so you're the primary interested party for pursuing a fix

I am an interested party; the abovementioned community, as well as anybody using the library, also are. Who is more interested than anybody else, I don't know and I don't think is of any relevance.

I invite you again to complete the code above

I'll probably do that when I get some time to: after all, as you said I am interested in getting this fixed, or even just in digging into it enough to find a workaround.

Or somebody else might also be willing to do that.

[teo1978]

Ok, so I have done the work you so kindly asked me to do, and here's a complete, minimal (as minimal as I could) and reproducible sketch with a real example URL.

It's an Arduino sketch that I am opening with the Arduino IDE and testing on a Sparkfun ESP8266 "Thing Dev", but should work on other Arduino-compatible ESP8266 board, I guess. I hope that's ok.

NOTE: you need to replace "YOUR_WIFI_SSID" and "YOUR_WIFI_PASSWORD" with the correct values for your wifi network.

The code is at the end of this comment.

To me, it reproduces the issue pretty frequently. Sometimes it behaves as expected, sometimes it exhibits the issue just a little bit, and sometimes to the point that it goes on forever.

(and occasionally it fails to connect with the -1 error code which is supposed to mean "connection refused" which I'm pretty sure is bull****, but that's another story).

The expected output would look like this:

Hello!
Trying to connect to Wifi 'Sixty-Seven' with password '****'
Trying to connect to Wifi 'Sixty-Seven' with password '****'
Trying to connect to Wifi 'Sixty-Seven' with password '****'
Trying to connect to Wifi 'Sixty-Seven' with password '****'
...connected succesfully
... Loading file from url: http://opentarget.org/test/esp8266/1/test.php
... got HTTP code 200; size is 245760
10070 bytes downloaded; 235690 remaining
20150 bytes downloaded; 225610 remaining
30262 bytes downloaded; 215498 remaining
40342 bytes downloaded; 205418 remaining
50422 bytes downloaded; 195338 remaining
60502 bytes downloaded; 185258 remaining
70614 bytes downloaded; 175146 remaining
80694 bytes downloaded; 165066 remaining
90750 bytes downloaded; 155010 remaining
100830 bytes downloaded; 144930 remaining
110942 bytes downloaded; 134818 remaining
120942 bytes downloaded; 124818 remaining
131030 bytes downloaded; 114730 remaining
141110 bytes downloaded; 104650 remaining
151190 bytes downloaded; 94570 remaining
161270 bytes downloaded; 84490 remaining
171382 bytes downloaded; 74378 remaining
181462 bytes downloaded; 64298 remaining
191542 bytes downloaded; 54218 remaining
201654 bytes downloaded; 44106 remaining
211734 bytes downloaded; 34026 remaining
221814 bytes downloaded; 23946 remaining
231894 bytes downloaded; 13866 remaining
242006 bytes downloaded; 3754 remaining
Done downloading file. 245760 bytes downloaded.

Occasionally, in the middle of that, you may expect to get very few occurrences of this:

... !! been waiting for available bytes for more than 2 seconds!!

When the issue happens, you get something like this:

Hello!
Trying to connect to Wifi 'Sixty-Seven' with password '****'
Trying to connect to Wifi 'Sixty-Seven' with password '****'
Trying to connect to Wifi 'Sixty-Seven' with password '****'
Trying to connect to Wifi 'Sixty-Seven' with password '****'
...connected succesfully
... Loading file from url: http://opentarget.org/test/esp8266/1/test.php
... got HTTP code 200; size is 245760
10070 bytes downloaded; 235690 remaining
20150 bytes downloaded; 225610 remaining
30262 bytes downloaded; 215498 remaining
40342 bytes downloaded; 205418 remaining
50422 bytes downloaded; 195338 remaining
60502 bytes downloaded; 185258 remaining
70558 bytes downloaded; 175202 remaining
80670 bytes downloaded; 165090 remaining
90726 bytes downloaded; 155034 remaining
100806 bytes downloaded; 144954 remaining
110862 bytes downloaded; 134898 remaining
120942 bytes downloaded; 124818 remaining
131054 bytes downloaded; 114706 remaining
141134 bytes downloaded; 104626 remaining
... !! been waiting for available bytes for more than 2 seconds!!
149478 bytes downloaded; 96282 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
151086 bytes downloaded; 94674 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
153230 bytes downloaded; 92530 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
154838 bytes downloaded; 90922 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
156446 bytes downloaded; 89314 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
158590 bytes downloaded; 87170 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
160198 bytes downloaded; 85562 remaining
170254 bytes downloaded; 75506 remaining
180366 bytes downloaded; 65394 remaining
190446 bytes downloaded; 55314 remaining
200526 bytes downloaded; 45234 remaining
... !! been waiting for available bytes for more than 2 seconds!!
210046 bytes downloaded; 35714 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
212190 bytes downloaded; 33570 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
213798 bytes downloaded; 31962 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
215942 bytes downloaded; 29818 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
217550 bytes downloaded; 28210 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
219158 bytes downloaded; 26602 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
221302 bytes downloaded; 24458 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
222910 bytes downloaded; 22850 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
225054 bytes downloaded; 20706 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
226662 bytes downloaded; 19098 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
228806 bytes downloaded; 16954 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
230414 bytes downloaded; 15346 remaining
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!
... !! been waiting for available bytes for more than 2 seconds!!

(and after literally several minutes it may even manage to finish downloading!!)

Regarding the code, I know it could be cleaner. The reason i have not made it even simpler is that I need to print some debugging information of what's happening, but at the same time I want to avoid speeding down the download significantly because of all the serial output, in case speeding down the client interferes with the download itself. So I try to print messages only when either I have accumulated a few downloaded kB's, or a couple of seconds have passed with no available data.

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

#define MYDEBUG;

#ifdef MYDEBUG
  char mybuffer[1024];
#endif


String client_ssid="YOUR_WIFI_SSID";
String client_password="YOUR_WIFI_PASSWORD";


void setup() 
{
  Serial.begin(115200);
  myDebug("Hello!");



  WiFi.mode(WIFI_STA);
  WiFi.begin(client_ssid.c_str(), client_password.c_str());

  for (int i=0; i<20; i++) {
    myDebug("Trying to connect to Wifi '%s' with password '%s'", client_ssid.c_str(), client_password.c_str());
  
    if (WiFi.status()==WL_CONNECTED) {
      break;
    }
    delay(1000);
  }
  if (WiFi.status()!=WL_CONNECTED)  {
    myDebug("ERROR: Cannot connect to WiFi!!!");
    return;
  }
  
  myDebug("...connected succesfully");
  
  HTTPClient http;
  
  String url="http://opentarget.org/test/esp8266/1/test.php";

  uint8_t httpbuf[128] = {0};

  myDebug("... Loading file from url: %s", url.c_str());
  int httpcode=0;
  int len=0;
  int bytesdownloaded=0;
  http.begin(url);
  httpcode=http.GET();
  WiFiClient* httpstream = http.getStreamPtr();
  if (httpcode==200) {
    len=http.getSize();
  }
  myDebug("... got HTTP code %d; size is %d", httpcode, len);
  int ticks=0;
  int ticks0bytes=0;
  if (httpcode==200) {
    
    while (len > 0 || len ==-1) {
      int gotbytes=0;
    
      if (!http.connected()) {
        break;
      }
      
      size_t avl=httpstream->available();
      
      if (avl) {
        gotbytes = httpstream->readBytes(httpbuf, ((avl>sizeof(httpbuf))?sizeof(httpbuf):avl));
        bytesdownloaded+=gotbytes;
        ticks0bytes=0;
      }
      else {
        delay(100);
        ticks0bytes++;
        if (ticks0bytes>=20) {
          myDebug("... !! been waiting for available bytes for more than 2 seconds!!");
          ticks0bytes=0;
          ticks=10000;
        }
      }
      
      if (gotbytes>0) {
        if (len>0) len=max(0, len-gotbytes);
        ticks+=gotbytes;
        if (ticks>=10000) {
          myDebug("%d bytes downloaded; %d remaining", bytesdownloaded, len);
          ticks=0;
        }
        
        // Here we would do something with the downloaded bytes
      }
      
    }
    myDebug("Done downloading file. %d bytes downloaded.", bytesdownloaded);
  }
  else {
    myDebug("...HTTP Error %d", httpcode);
  }
}

void loop() {
  
}


void myDebug(const char* format, ...) {
  #ifdef MYDEBUG
    va_list arglist;
    va_start(arglist, format);
    vsprintf(mybuffer, reinterpret_cast<const char*>(format), arglist);
    Serial.println(mybuffer);
    va_end(arglist);
  #endif
}

[devyte]

@teo1978 there we go! That wasn't so hard now was it?
Now just please provide the core version (release version like 2.4.2 or git date/hash, etc), and flash size + config (what size spiffs), and the required info should be close to complete.

[devyte]

Notes:
-Stream O(n2)
-SPIFFS large file
-SPIFFS large size

[teo1978]

That wasn't so hard now was it?

Nope, which is why I din't think you needed me to do it.

Now just please provide...

Ok, will do tomorrow

Notes:

I guess those notes are not for me, are they?

[teo1978]

core version (release version like 2.4.2 or git date/hash, etc)

2.4.0, I got it via the Arduino IDE's Board Manager.

and flash size + config (what size spiffs)

As I said it's this board:
https://learn.sparkfun.com/tutorials/esp8266-thing-development-board-hookup-guide?_ga=2.104455811.26954385.1537716697-914235417.1536518212

That's all I know.

[d-a-v]

Would you please at least try 2.4.2 ?
Though git version is always better when reporting bugs.

[teo1978]

Sorry, I have installed 2.4.2 now and I keep observing the same behavior. (do you? have you tried it??)

Though git version is always better when reporting bugs.

I'd be happy to but I'm not sure how to do that, as I usually install the library through the Arduino IDE's Board Manager.

Not sure if I can just download the git version and simply overwrite the currently installed one with it?

[Pablo2048]

Well the documentation is far from perfect, but did you read this https://github.com/esp8266/Arduino#using-git-version ?

[d-a-v]

@teo1978
This looks pretty much like #4176.
#5126 addresses this issue. We are looking for testers.
Once you have installed the git version, please try with this pull request.

How to try

git fetch origin pull/5126/head:sack
git checkout sack

Revert to master:

git checkout master

In case of updates in the PR, do this before restarting the above.

git checkout master
git branch -D sack

[teo1978]

Sorry, I intended to mark as duplicate of #4176 (maybe I can't do that). I thought I would get that option and if not, a confirmation prompt when closing where I could cancel.

Anyway, it does seem to be that issue. I am testing the PR and will report there.

Thank you.