Downloading a file from URL to a file (LittleFS) returns a HTTPC_ERROR_READ_TIMEOUT error

[BrokeStudio]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git). (I don't know how to do this in Platformio)
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP-12E
• Core Version: 3.0.2
• Development Env: Platformio (with -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_PORT=Serial build flags)
• Operating System: Windows

Settings in IDE

• Module: Generic ESP8266 Module
• Flash Mode: qio (I guess)
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• Reset Method: nodemcu
• Flash Frequency: 26MHz
• CPU Frequency: 80MHz
• Upload Using: Serial
• Upload Speed: 921600

Problem Description

Trying to download a file from a URL directly to a file stream returns an "error(-11): read Timeout" (HTTPC_ERROR_READ_TIMEOUT).
It works fine if doing http.writeToStream(&Serial); instead of http.writeToStream(&f);.
It works fine using core v2.7.4 (platform = espressif8266@2.6.3 in platformio.ini)

I got the same error using:

• core 3.0.0 (platform = espressif8266@3.0.0 in platformio.ini)
• core 3.0.1 ((platform = espressif8266@2.6.3 in platformio.ini)
• core 3.1.0 (platform = espressif8266@3.2.0 in platformio.ini).

Seems to be working with 3.0.1-dev, using this config in platformio.ini:

platform = espressif8266
platform_packages =
    platformio/framework-arduinoespressif8266 @ https://github.com/esp8266/Arduino.git

MCVE Sketch

#include <Arduino.h>
#include <LittleFS.h>
#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

#define FILENAME "/test.txt"
#define URL "http://brokestudio.fr/test.txt"
#define SSID "your_ssid"
#define PASSWORD "your_password"

HTTPClient http;
WiFiClient client;
File f;

void setup() {
  // init serial
  Serial.begin(115200);
  Serial.println();

  // connect to WiFi
  WiFi.begin(SSID, PASSWORD);
  Serial.print(F("Connecting"));

  // Wait for the Wi-Fi to connect
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(1000);
    Serial.print('.');
  }
  Serial.println(F("Connected!"));

  // init file system
  if (!LittleFS.begin())
    Serial.println(F("File system mount failed"));
  else
    Serial.println(F("File system mount successful"));

  // check if test file exists and delete it if needed
  if(LittleFS.exists(FILENAME))
    LittleFS.remove(FILENAME);

  // try to download a file
  int httpCode;
  int error;
  Serial.println(F("Downloading: " URL));
  f = LittleFS.open(FILENAME, "w");
  if (f)
  {
    http.begin(client, URL);
    httpCode = http.GET();
    switch(httpCode)
    {
      case HTTP_CODE_OK:
        error = http.writeToStream(&f);
        Serial.printf("Error code: %i\r\n", error);
        break;

      default:
        Serial.println(F("[HTTP] GET failed, error: "));
        Serial.printf("%i - %s\r\n", httpCode, http.errorToString(httpCode).c_str());
        Serial.println(F(URL));
        break;
    }
    f.close();
  }
  http.end();

  // output file content
  Serial.println(F("Reading file content"));
  Serial.println(F("---"));
  f = LittleFS.open(FILENAME, "r");
  String content = f.readString();
  Serial.println(content);
  f.close();
  Serial.println(F("---"));
}

void loop() {
}

Debug Messages

SDK:2.2.2-dev(38a443e)/Core:3.0.2=30002000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-48-g7421258/BearSSL:6105635

fpm close 1
mode : sta(bc:dd:c2:fd:ed:f6)
add if0
Connecting..scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 3
cnt

connected with BARBE_WIFI_RDC, channel 1
dhcp client start...
..ip:192.168.2.25,mask:255.255.255.0,gw:192.168.2.1
.Connected!
File system mount successful
Downloading: http://brokestudio.fr/test.txt
[HTTP-Client][begin] url: http://brokestudio.fr/test.txt
[HTTP-Client][begin] host: brokestudio.fr port: 80 url: /test.txt
[HTTP-Client][sendRequest] type: 'GET' redirCount: 0
[HTTP-Client] connected to brokestudio.fr:80
[HTTP-Client] sending request header
-----
GET /test.txt HTTP/1.1
Host: brokestudio.fr
User-Agent: ESP8266HTTPClient
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Connection: keep-alive
Content-Length: 0

-----
'HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK
'HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/plain
'HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 12
'HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive
'HTTP-Client][handleHeaderResponse] RX: 'Keep-Alive: timeout=15
'HTTP-Client][handleHeaderResponse] RX: 'Date: Tue, 14 Dec 2021 17:15:13 GMT
'HTTP-Client][handleHeaderResponse] RX: 'Server: Apache
'HTTP-Client][handleHeaderResponse] RX: 'Last-Modified: Tue, 14 Dec 2021 16:44:46 GMT
'HTTP-Client][handleHeaderResponse] RX: 'ETag: "c-5d31de829f9eb"
'HTTP-Client][handleHeaderResponse] RX: 'Accept-Ranges: bytes
'HTTP-Client][handleHeaderResponse] RX: '
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 12
[HTTP-Client][returnError] error(-11): read Timeout
[HTTP-Client][returnError] tcp stop
Error code: -11
[HTTP-Client][end] still data in buffer (12), clean up.
[HTTP-Client][end] tcp keep open for reuse
Reading file content
---

---

[d-a-v]

It seems to be an issue with your webserver.
I got your sketch working in host mode with this line but I get code 503 without (Service Unavailable).

    http.begin(client, URL);
+   http.setUserAgent("curl/7.74.0");
    httpCode = http.GET();

Now you have my IP addresses in your logs. Good work :)

However,
When not setting the user agent and commenting out these lines, then your sketch works.
Please tell us whether you wish to make a pull-request soon to fix ESP8266HTTPClient.

[mcspr]

Seems to be working with 3.0.1-dev, using this config in platformio.ini:
platform = espressif8266
platform_packages =
platformio/framework-arduinoespressif8266 @ https://github.com/esp8266/Arduino.git

...issue already fixed in master after this PR?
#8384
#8386
(at least it looks like it b/c of the writeToStream usage)

actual git repo is in the ~/.platformio/packages/framework-arduinoespressif8266@src-*, you could check whether that commit is there

[BrokeStudio]

I'm not sure what you mean by host mode, can you explain please?

I got your sketch working in host mode with this line but I get code 503 without (Service Unavailable).

I added this line as you advised and switch back to Core 3.0.2:

    http.begin(client, URL);
+   http.setUserAgent("esp8266/1.0.0");
    httpCode = http.GET();

but no success, here is the debug log:

SDK:2.2.2-dev(38a443e)/Core:3.0.2=30002000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-48-g7421258/BearSSL:6105635

fpm close 1
mode : sta(bc:dd:c2:fd:ed:f6)
add if0
Connecting..scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 3
cnt

connected with BARBE_WIFI_RDC, channel 1
dhcp client start...
..ip:192.168.2.25,mask:255.255.255.0,gw:192.168.2.1
.Connected!
File system mount successful
Downloading: http://brokestudio.fr/test.txt
[HTTP-Client][begin] url: http://brokestudio.fr/test.txt
[HTTP-Client][begin] host: brokestudio.fr port: 80 url: /test.txt
[HTTP-Client][sendRequest] type: 'GET' redirCount: 0
[HTTP-Client] connected to brokestudio.fr:80
[HTTP-Client] sending request header
-----
GET /test.txt HTTP/1.1
Host: brokestudio.fr
User-Agent: esp8266/1.0.0
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Connection: keep-alive
Content-Length: 0

-----
'HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK
'HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/plain
'HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 12
'HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive
'HTTP-Client][handleHeaderResponse] RX: 'Keep-Alive: timeout=15
'HTTP-Client][handleHeaderResponse] RX: 'Date: Wed, 15 Dec 2021 11:03:47 GMT
'HTTP-Client][handleHeaderResponse] RX: 'Server: Apache
'HTTP-Client][handleHeaderResponse] RX: 'Last-Modified: Tue, 14 Dec 2021 16:44:46 GMT
'HTTP-Client][handleHeaderResponse] RX: 'ETag: "c-5d31de829f9eb"
'HTTP-Client][handleHeaderResponse] RX: 'Accept-Ranges: bytes
'HTTP-Client][handleHeaderResponse] RX: '
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 12
[HTTP-Client][returnError] error(-11): read Timeout
[HTTP-Client][returnError] tcp stop
Error code: -11
[HTTP] GET failed, error:
200 -
http://brokestudio.fr/test.txt
[HTTP-Client][end] still data in buffer (12), clean up.
[HTTP-Client][end] tcp keep open for reuse
Reading file content
---

---

Also, you can see in the debug log in my initial message that a default user-agent was present.

[BrokeStudio]

Thanks @mcspr ! Looks like it's exactly my problem, I'll test it ASAP.

[BrokeStudio]

OK #8384 fixed the issue, thanks for your help.

[d-a-v]

@BrokeStudio
Sorry for my answer. I did not see the issue because I was testing with master.

host mode, can you explain please?

Host mode / emulation on host is the ability to compile and run some sketch on a computer.
On unix (incl. wsl) for your /path/to/your/sketch/sketch.ino:

cd tests/host
make `/path/to/your/sketch/sketch`