http.POST() limit of payload size? #1872

TheAustrian · 2016-04-06T19:04:52Z

Hi!

I'm trying to send data as an HTTPClient to a server with http.POST() and it works great but the payload gets cut off around the 3000 character mark. Does this have something to do with "application/x-www-form-urlencoded"? It shouldn't, iirc, cause POST shouldn't have a size limit?

HTTPClient http;
http.begin("http://example.com/weather/espdata.php");
http.addHeader("Content-Type", "application/x-www-form-urlencoded");
http.POST(textToSend);
http.end();

I just wanted to know if there is a limit on payload size so I can rule out some other issue with my server. The String I'm sending gets put out normally in the serial monitor, so that shouldn't be the problem.

Thanks!

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Humancell · 2016-05-05T04:18:02Z

Bump!

I'm seeing this same issue ... when my payload crossed the 3k mark I'm now getting a http.POST() return of -3 ... HTTPC_ERROR_SEND_PAYLOAD_FAILED ... but can't find any info on why this is, and what exactly failed?

Is there a buffer that needs to be adjusted, or?

chaeplin · 2016-05-05T10:40:17Z

I think payload size > one packet size --> err.
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp#L376-L380

Humancell · 2016-05-05T18:30:05Z

Ok ... this is what I wondered. I actually created a workaround that I'm using successfully, just in case anyone else requires a similar fix. To do it I had to use WifiClient:

#include <ESP8266HTTPClient.h>

String jsonPayload = "{ my big giant 3.5k byte json payload ....}";
int contentLength = jsonPayload.length();

  // create the request and headers
  String request = "POST " + String(endpointURL) + " HTTP/1.1\r\n" +
    "Host: " + String(endpointHost) + "\r\n" + 
    "Accept: application/json" + "\r\n" + 
    "Content-Type: application/json\r\n" +
    "Content-Length: " + String(contentLength) + "\r\n\r\n";

  // Use WiFiClient class to create TCP connections
  WiFiClient client;

  connectResult = client.connect(endpointHost, atoi(endpointPort));

  // This will send the request and headers to the server
  client.print(request);

  // now we need to chunk the payload into 1000 byte chunks
  int cIndex;
  for (cIndex = 0; cIndex < contentLength; cIndex = cIndex + 1000) {
    client.print(jsonPayload.substring(cIndex, cIndex+1000));
    //Serial.print(jsonPayload.substring(cIndex, cIndex+1000));
  }
  client.print(jsonPayload.substring(cIndex));
  //Serial.println(jsonPayload.substring(cIndex));

I have a little more response checking and other stuff ... but you get the idea. Using this I am now sending pretty big payloads, and it seems to work fine. The 1000 bytes was just an arbitrary guess ... if I have time I might try to adjust this to fine tune the operation.

Obviously there are limitations in this approach as I can't do SSL easily (or can I?) ... but maybe something like this ought to be rolled into the handling of the payload on the call to http.POST in the HTTPClient?

chaeplin · 2016-05-05T19:37:21Z

@Humancell use WiFiClientSecure.h for ssl.
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

Chunk can be large to 1460.

Humancell · 2016-05-05T19:51:55Z

Thanks! Nice bonus! I'll make my changes. :-)

espmut · 2017-05-15T15:58:21Z

I just ran into this issue as well. I believe the limit is just under 2000 Bytes. I hope there is a work around or solution for this. I'd rather use HTTPClient.

melvinpmathew · 2017-05-31T15:45:11Z

Is there a work around for this issue? I need to send 8000bytes on POST>

devyte · 2017-10-17T19:20:25Z

@TheAustrian @Humancell is this issue still valid with latest git?

mcanyucel · 2018-01-29T13:54:29Z

I am trying to upload a csv file with size of around 200 kB and using the method int HTTPClient::sendRequest(const char * type, Stream * stream, size_t size) and get the error HTTPC_ERROR_SEND_PAYLOAD_FAILED. Smaller size files work perfectly so I am assuming there is an unwritten file size limit or kind of a bug. This is mentioned in few stackoverflow questions as well but they have no answer (here.
Any new information, suggestions, progress, workaround?

espmut · 2018-01-29T14:33:33Z

For me, I broke up the post into multiple posts.

…

On Mon, Jan 29, 2018 at 6:54 AM, Mustafa Can Yücel ***@***.*** > wrote: I am trying to upload a csv file with size of around 200 kB and using the method int HTTPClient::sendRequest(const char * type, Stream * stream, size_t size) and get the error HTTPC_ERROR_SEND_PAYLOAD_FAILED. Smaller size files work perfectly so I am assuming there is an unwritten file size limit or kind of a bug. This is mentioned in few stackoverflow questions as well but they have no answer (here <https://stackoverflow.com/questions/41388417/send-payload-error-in-http-post-arduino-esp8266> . Any new information, suggestions, progress, workaround? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1872 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbYZfq_JO_OP1ezEystbN2ttXG2DpOgyks5tPc2ngaJpZM4IBZhy> .

-- Machine Utilization Tracker www.mutcloud.com 602.524.0192

Silberlachs · 2018-01-30T10:09:38Z

I see an issue when receiving rss feeds from a news portal over http GET (get string to be clear)

One RSS page with kind of high priority news gets transmitted normally and the esp can send it to a lcd display.

But whenever I try to get the "normal" news feeds, the esp just doesn't do anything at all.
Maybe it restarted?

The normal rss feed has 14527 Characters, the high priority one has 2676 Characters...

I've gone with the high priority one for now and I can see why nearly 15k characters is a bit overkill for this little thing, but still it's an issue....

mcanyucel · 2018-01-30T12:26:57Z

@espmut How did you manage to divide the file? I used another approach that reads the file into a WiFiClient using MTU_Size chunks but it floods everywhere with stack errors :(

d-a-v · 2018-01-30T16:01:55Z

I could reproduce the error.
Can you retry sending big data without chunking into pieces, with DebugLevel HTTP_CLIENT (in Tools menu, with Serial for DebugPort) ?
I get:

[HTTP-Client][sendRequest] short write, asked for xxxx but got yyyy retry...

2MB data are working flawlessly with all debug options activated (they slow down processing a lot).
Bug is located, fix will come hopefully.

Full sketch:

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

#define ssid "open"
#define password ""

void setup()
{
  Serial.begin(115200);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");

  // Print the IP address
  Serial.println(WiFi.localIP());

}

class FakeStream: public Stream
{
  protected:
    size_t fakesize;

  public:
    FakeStream (size_t size): fakesize(size) { }

    virtual int available() {
      return fakesize;
    }
    virtual int read() {
      if (fakesize) {
        --fakesize;
        return 0x5a;
      }
      return -1;
    }
    virtual int peek() {
      return 0x5a;
    }
    virtual void flush() {
      fakesize = 0;
    };
    virtual size_t write (uint8_t data) {
      (void)data;
      return 1;
    }
    size_t size () {
      return fakesize;
    }
    const char* name () {
      return "5a";
    }
};


void loop()
{
  // put your main code here, to run repeatedly:
  HTTPClient http;
  http.begin("http://10.43.1.254:80/test/upload.php");

  int contentLength = 2000000;

  http.addHeader("Content-Type", "application/octet-stream");
  http.addHeader("Content-Length", String(contentLength));

  Serial.println("a----------------------------------------------");
  FakeStream s(contentLength);
  int httpCode = http.sendRequest("POST", &s, contentLength);
  Serial.println(httpCode);
  Serial.println("b----------------------------------------------");

  while (true)delay(1000);
}

The upload.php (got in some other issue):

<?php
file_put_contents('/var/www/html/test/test.txt', file_get_contents('php://input'));
?>

…or sending may fix esp8266#1872

d-a-v · 2018-01-31T10:37:04Z

@TheAustrian @mcanyucel @espmut can you unbreak-to-one-piece your POST and try this pull-request #4265 ?

@Silberlachs the fix is about outgoing data (not incoming data). If it still does not work for you, then please open a new issue with a minimal sketch to highlight the bug. It will be a lot easier for us to reproduce then fix.

…or sending may fix esp8266#1872

…or sending (#4265) may fix #1872

martinius96 · 2018-02-01T14:53:53Z

Why people use HTTPclient, when they can use normal HTTP request under ESP8266Wifi.h library?

mcanyucel · 2018-02-01T14:59:44Z

@martinius96 Are you referring the WiFiClient? I could not see any http request under the ESP8266Wifi.h library.

martinius96 · 2018-02-01T15:03:26Z

@mcanyucel Yes WIficlient. What is difference between wificlient and HTTPclient (BasicHttpClient.ino file)? It do the same thing..

mcanyucel · 2018-02-01T15:15:34Z

In a manner of speaking. As far as I have seen WiFiClient has a very broad virtual size_t write(const uint8_t *buf, size_t size); method, therefore you have to generate all the POST headers manually. If you are making a post request of form-url-encoded with a short string, it may not be very hard to do that. But if you are attaching a binary file, the post request file identifiers are not very user friendly. moreover you have to read the file in buffer sized chunks and I am not very proficient with cpp so I keep getting stack errors all over the place. I think I could not find a single example that works on the internet that uploads a file on the SD card to a server using WiFiClient,
HttpClient on the other hand exposes more streamlined methods that accept the post parameters and generates the necessary strings automatically. It also handles buffer operations better than me; it does not flood everywhere with stack overflows. You can use a single line int sendRequest(const char * type, Stream * stream, size_t size = 0); which in my opinion better and more appealing than to have this.
In the end it is "not reinventing the wheel"; the SD card operations or OTA updates can be done manually using basic core libraries, but it is easier to use the already developed "advanced" libraries.

d-a-v · 2018-02-01T15:16:03Z

@martinius96 HTTPClient uses WiFiClient to make it easier to deal especially with http requests and answers.

@TheAustrian @mcanyucel @espmut the fix is now in master. Can you test it ?

mcanyucel · 2018-02-01T15:19:35Z

@d-a-v Thank you very much for your extremely quick response. I am a little bit overwhelmed with some other stuff nowadays but hopefully I will be able to test it within a few days or maybe early next week.

devyte · 2018-02-20T01:15:52Z

@mcanyucel @TheAustrian @Humancell could you please confirm the fix? I think this is the only pending for release 2.4.1.

mcanyucel · 2018-02-20T19:39:30Z

@devyte I really wish to try but we have deployed all our MCU boards to field. I have ordered several ESP8266s but it will take some time for them to arrive. If I can find a spare around I will try it.

Humancell · 2018-09-15T03:42:00Z

I'm sorry that I somehow missed your request, and the Github notice back in February!

It was funny, but I ran into this issue again last night, with the payload being too large, and found this thread to check back on the status. I did update to v2.4.2 ... and this does fix the issue that I was seeing! I'm now POSTing bigger payloads just fine!

One other issue that I did see, is that I has to add a line to set:

httpClient->setTimeout(250);

Before reading the response stream. Without this, my code that was running just fine before, (v2.3.0) now seemed to have a default timeout of 4000ms! It would just hang. I added the line above, and now it's fine.

EdoMon93 · 2019-01-25T01:28:55Z

I seem to be suffering a similar problem, however in my case I think it might be related to free heap and fragmentation but have not been able to find any workarounds.
I'm using v2.5.0-beta2 and ArduinoJSON v5.13.4 (this might not be relevant).

So in the code I have arrays with a bunch of data which im trying to send to a backend. Due to the fact that the arrays are of considerable size, im splitting up posts to aprox less then 2kB (It should not be necessary but im running out of heap). I do this by calling this function several times, the code surrounding it retries several times in case of error:

int8_t sendSens(uint16_t arrayStart, uint16_t arrayEnd) {

	if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status
		int httpCode = 0;
		DEBUG_PRINT(F("Free Heap before jsonBuffer: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		uint16_t maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		const size_t bufferSize = 2 * JSON_ARRAY_SIZE(arrayEnd - arrayStart + 1)
				+ JSON_OBJECT_SIZE(4);
		DEBUG_PRINT(F("bufferSize: "));
		DEBUG_PRINTLN(bufferSize);
		if (maxFreeBlock < bufferSize) {
			return MAX_BLOCK_SIZE_TOO_SMALL; //TOO FRAGMENTED
		}
		DynamicJsonBuffer jsonBuffer(bufferSize);
		JsonObject& root = jsonBuffer.createObject();
		root["cmd"] = "insert";
		root["device"] = DEVICE;
		JsonArray& sens = root.createNestedArray("sens");
		for (int i = arrayStart; i < arrayEnd; i++) {
			sens.add(dataSens[i]);
		}
		JsonArray& sensTime = root.createNestedArray("sensTime");
		for (int i = arrayStart; i < arrayEnd; i++) {
			sensTime.add(dataSensTime[i]);
		}
#ifdef DEBUG_MODE
		root.printTo(Serial);
#endif
		size_t len = root.measureLength() + 1;
		DEBUG_PRINT(F("\njson Length: "));
		DEBUG_PRINTLN(len);
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINT(F("Max Block: "));
		DEBUG_PRINTLN(maxFreeBlock);
		if (maxFreeBlock < len) {
			return MAX_BLOCK_SIZE_TOO_SMALL; //TOO FRAGMENTED
		}
		DEBUG_PRINT(F("Free Heap before postRequest malloc: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		char *postRequest = (char *) malloc(len);
		root.printTo(postRequest, len);
		DEBUG_PRINT(F("postRequest: "));
		DEBUG_PRINTLN(postRequest);
		DEBUG_PRINT(F("Free Heap before jsonBuffer.clear: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		jsonBuffer.clear();
		ESP.wdtFeed();
		DEBUG_PRINT(F("Free Heap before http.begin: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		WiFiClient client;
		HTTPClient http;
		http.setTimeout(2000);
		http.begin(F("http://mywebpage.com/dataReciever.php")); //Specify request destination
		http.addHeader(F("Content-Type"), F("application/json")); //Specify content-type header
		DEBUG_PRINT(F("Free Heap before http.POST: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		httpCode = http.POST(postRequest); //Send the request
		DEBUG_PRINT(F("Free Heap after http.POST: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		DEBUG_PRINTLN(F("JSON Data Sent"));
		DEBUG_PRINT(F("Response: "));
		DEBUG_PRINTLN(httpCode);
		ESP.wdtFeed();
		if (httpCode == 200) {
			DEBUG_PRINT(F("Response Payload: "));
			DEBUG_PRINTLN(http.getString());
			DEBUG_PRINT(F("Free Heap before HTTP.END: "));
			DEBUG_PRINT(ESP.getFreeHeap());
			DEBUG_PRINT(F(" Fragmentation: "));
			DEBUG_PRINT(ESP.getHeapFragmentation());
			DEBUG_PRINT(F("% MaxBlock: "));
			maxFreeBlock = ESP.getMaxFreeBlockSize();
			DEBUG_PRINTLN(maxFreeBlock);
			http.end();  //Close connection
			DEBUG_PRINT(F("Free Heap before FREE POST REQ: "));
			DEBUG_PRINT(ESP.getFreeHeap());
			DEBUG_PRINT(F(" Fragmentation: "));
			DEBUG_PRINT(ESP.getHeapFragmentation());
			DEBUG_PRINT(F("% MaxBlock: "));
			maxFreeBlock = ESP.getMaxFreeBlockSize();
			DEBUG_PRINTLN(maxFreeBlock);
			free(postRequest);
			DEBUG_PRINT(F("Free Heap before RETURN: "));
			DEBUG_PRINT(ESP.getFreeHeap());
			DEBUG_PRINT(F(" Fragmentation: "));
			DEBUG_PRINT(ESP.getHeapFragmentation());
			DEBUG_PRINT(F("% MaxBlock: "));
			maxFreeBlock = ESP.getMaxFreeBlockSize();
			DEBUG_PRINTLN(maxFreeBlock);
			return JSON_POST_OK;
		}
		DEBUG_PRINT(F("Error to String: "));
		DEBUG_PRINTLN(http.errorToString(httpCode));
		DEBUG_PRINT(F("FAILED: Response Payload: "));
		DEBUG_PRINTLN(http.getString());
		http.end();  //Close connection
		free(postRequest);
		return SERVER_BAD_RESPONSE;
	} else {
		DEBUG_PRINTLN(F("WiFi not connected during temp report"));
		return NOT_CONNECTED_TO_WIFI;
	}
}

Sorry for the long post, most of it is debug output...

Next is the output:

Send sensors from 0 to 99
Free Heap before jsonBuffer: 5776 Fragmentation: 8% MaxBlock: 5344
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
json Length: 1750
Max Block: 2816
Free Heap before postRequest malloc: 3248 Fragmentation: 13% MaxBlock: 2816
postRequest: {"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
Free Heap before jsonBuffer.clear: 1488 Fragmentation: 26% MaxBlock: 1056
Free Heap before http.begin: 4016 Fragmentation: 32% MaxBlock: 2528
Free Heap before http.POST: 3800 Fragmentation: 28% MaxBlock: 2528
Free Heap after http.POST: 3368 Fragmentation: 38% MaxBlock: 1768
JSON Data Sent
Response: -3
Error to String: send payload failed
FAILED: Response Payload: 

Send sensors from 0 to 99
Free Heap before jsonBuffer: 5448 Fragmentation: 32% MaxBlock: 3272
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
json Length: 1750
Max Block: 1768
Free Heap before postRequest malloc: 2920 Fragmentation: 34% MaxBlock: 1768
postRequest: {"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
Free Heap before jsonBuffer.clear: 1160 Fragmentation: 31% MaxBlock: 744
Free Heap before http.begin: 3688 Fragmentation: 11% MaxBlock: 3272
Free Heap before http.POST: 3472 Fragmentation: 6% MaxBlock: 3272
Free Heap after http.POST: 2840 Fragmentation: 28% MaxBlock: 1880
JSON Data Sent
Response: 200
Response Payload: 100 OK
Free Heap before HTTP.END: 3544 Fragmentation: 25% MaxBlock: 2544
Free Heap before FREE POST REQ: 3600 Fragmentation: 6% MaxBlock: 3384
Free Heap before RETURN: 5360 Fragmentation: 29% MaxBlock: 3384

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5584 Fragmentation: 32% MaxBlock: 3384
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368162,1548368162,1548368162,1548368162,1548368162,1548368162]}
json Length: 1750
Max Block: 1768
Free Heap before postRequest malloc: 3056 Fragmentation: 35% MaxBlock: 1768
postRequest: {"cmd":"insert","device":2,"sens":[59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368162,1548368162,1548368162,1548368162,1548368162,1548368162]}
Free Heap before jsonBuffer.clear: 1296 Fragmentation: 30% MaxBlock: 856
Free Heap before http.begin: 3824 Fragmentation: 12% MaxBlock: 3384
Free Heap before http.POST: 3608 Fragmentation: 7% MaxBlock: 3384
Free Heap after http.POST: 3176 Fragmentation: 32% MaxBlock: 1768
JSON Data Sent
Response: -3
Error to String: send payload failed
FAILED: Response Payload: 

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512
WIFI turned OFF
Free Heap: 4992

As you can see from the output, the exact same size of data succeeds only on the second try and eventually aborts due to lack of minimum free heap required for the json parser buffer.
The response of my http.post seems to arbitrarily fragment my memory unnecessarily, I'm not sure if it requires a minimum of heap to operate "normally"...
Until now I had been solving this issue by reducing the packet size, but now it just seems weird for my esp not to being able to send more than 1.5kB of data on one post, splitting it into smaller pieces would take forever to send.

Anyone has this problem? Is this a bug in HTTPClient? Am I asking too much for my esp?
General recommendations are welcome :)

ligantx · 2020-03-15T20:26:29Z

Ok ... this is what I wondered. I actually created a workaround that I'm using successfully, just in case anyone else requires a similar fix. To do it I had to use WifiClient:
#include <ESP8266HTTPClient.h>

String jsonPayload = "{ my big giant 3.5k byte json payload ....}";
int contentLength = jsonPayload.length();

  // create the request and headers
  String request = "POST " + String(endpointURL) + " HTTP/1.1\r\n" +
    "Host: " + String(endpointHost) + "\r\n" + 
    "Accept: application/json" + "\r\n" + 
    "Content-Type: application/json\r\n" +
    "Content-Length: " + String(contentLength) + "\r\n\r\n";

  // Use WiFiClient class to create TCP connections
  WiFiClient client;

  connectResult = client.connect(endpointHost, atoi(endpointPort));

  // This will send the request and headers to the server
  client.print(request);

  // now we need to chunk the payload into 1000 byte chunks
  int cIndex;
  for (cIndex = 0; cIndex < contentLength; cIndex = cIndex + 1000) {
    client.print(jsonPayload.substring(cIndex, cIndex+1000));
    //Serial.print(jsonPayload.substring(cIndex, cIndex+1000));
  }
  client.print(jsonPayload.substring(cIndex));
  //Serial.println(jsonPayload.substring(cIndex));
I have a little more response checking and other stuff ... but you get the idea. Using this I am now sending pretty big payloads, and it seems to work fine. The 1000 bytes was just an arbitrary guess ... if I have time I might try to adjust this to fine tune the operation.

Obviously there are limitations in this approach as I can't do SSL easily (or can I?) ... but maybe something like this ought to be rolled into the handling of the payload on the call to http.POST in the HTTPClient?

After spending MANY hours trying to upload ESP32-CAM photos, i found out the problem was only with big photos (e.g. bigger than VGA) and big files (txt) in general, so i finally found the problem was with the POST body in general..
The quote solution works fine with Strings but esp32-cam produce a uint8_t * buf that has to be used differently..
So, the code below that i wrote worked fine with every FRAMESIZE (even UXGA - 1600 x 1200):

camera_fb_t * fb = esp_camera_fb_get();
int imgLenInt = fb->len;
String imgLenStr = (String)imgLenInt;
//headers already POSTed
int howMuch = 0;
int sumHowMuch = 0;
int i = 0;
int myChunkSize = 1300;  //try 1000-2000 values if you want
for (i = 0; i < imgLenInt - myChunkSize; i = i + myChunkSize) {
  howMuch = client.write(fb->buf + i, myChunkSize);
  sumHowMuch += howMuch;
}
int left = imgLenInt % myChunkSize;
howMuch = client.write(fb->buf + (i - myChunkSize), left);
sumHowMuch += howMuch;
Serial.print("you had to send:");
Serial.print(imgLenStr);
Serial.print(", and you send:");
Serial.println(sumHowMuch);`

AndrewIvashkin · 2021-05-28T12:56:31Z

@ligantx Hey i am trying to send base64 image in POST request 2
Right now i has the correct base64 string but when i am trying to send it i end up with error code -1

I tryed simmilar approach like you: I divide my string by 10 parts with code like this

`void SendChunks()
{ 
  HTTPClient http;
  http.setTimeout(6000);
  http.begin(endpoint,fingerprint);
  http.addHeader("Content-Type", "application/json"); 
  
  int ChunkCount = 10;
  int PicLenght = B64Pic.length();
  Serial.println(PicLenght);
  int ChunkLenght =  PicLenght / ChunkCount;
  Serial.println(ChunkLenght);
  
  for(int i = 0; i < ChunkCount; i++)
  {
    //Create json
    DynamicJsonDocument doc(40000); //size of the doc can 
    doc["index"] = i;
    
    doc["Base64Chunk"] = B64Pic.substring(i*ChunkLenght, (i+1)*ChunkLenght);
    //Serialize json
    String requestBody;
    serializeJson(doc, requestBody);
    Serial.println(requestBody);
    delay(500);

    //Do POST request
    int httpResponseCode = http.POST(requestBody);
    if(httpResponseCode>0)
    {
      Serial.println(httpResponseCode);   //Print return code
    }
    else
    { 
      Serial.print("Error on sending request: ");
      Serial.println(httpResponseCode);
    }
    
  }

  //send tail
  if(PicLenght -(ChunkLenght*ChunkCount)> 0)
  {
    DynamicJsonDocument doc(40000); //size of the doc can 
    doc["index"] = ChunkCount + 1;
    doc["Base64Chunk"] = B64Pic.substring(ChunkCount * ChunkLenght);
    //Serialize json
    String requestBody;
    serializeJson(doc, requestBody);
    Serial.println(requestBody);
    delay(500);
    //Do POST request
    int httpResponseCode = http.POST(requestBody);
    if(httpResponseCode>0)
    {
      Serial.println(httpResponseCode);   //Print return code
    }
    else
    { 
      Serial.print("Error on sending request: ");
      Serial.println(httpResponseCode);
    }
  }
  
  http.end();
}`

And finished with output like this(i made base64 chunks short on purpose) :
{"index":0,"Base64Chunk":"/9j/U"} 400 {"index":1,"Base64Chunk":"UCF"} Error on sending request: -11 {"index":2,"Base64Chunk":"ASi"} Error on sending request: -1
So only first request end up with 400 code, second with -11 all the other return -1.

devyte added the waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. label Oct 17, 2017

d-a-v self-assigned this Jan 30, 2018

d-a-v added a commit to d-a-v/Arduino that referenced this issue Jan 31, 2018

ClientContext:_write_some: release buffer once data really accepted f…

fc93c5c

…or sending may fix esp8266#1872

d-a-v mentioned this issue Jan 31, 2018

ClientContext:_write_some: release buffer once data really accepted for sending #4265

Merged

devyte added type: bug component: network component: web and removed waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. labels Jan 31, 2018

devyte added this to the 2.4.1 milestone Jan 31, 2018

d-a-v added a commit to d-a-v/Arduino that referenced this issue Feb 1, 2018

ClientContext:_write_some: release buffer once data really accepted f…

6369439

…or sending may fix esp8266#1872

d-a-v closed this as completed in #4265 Feb 1, 2018

d-a-v added a commit that referenced this issue Feb 1, 2018

ClientContext:_write_some: release buffer once data really accepted f…

199fe0f

…or sending (#4265) may fix #1872

d-a-v reopened this Feb 1, 2018

d-a-v added the waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. label Feb 1, 2018

d-a-v mentioned this issue Feb 7, 2018

HTTPClient POST larger files doesn't work anymore. #2437

Closed

igrr closed this as completed Mar 8, 2018

igrr removed the staged-for-release label Mar 8, 2018

TD-er mentioned this issue Apr 30, 2018

Issues saving rules - Rules truncated at submit letscontrolit/ESPEasy#1296

Closed

EdoMon93 mentioned this issue Jan 26, 2019

Response Code -3 in HTTPClient.POST #5674

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http.POST() limit of payload size? #1872

http.POST() limit of payload size? #1872

TheAustrian commented Apr 6, 2016 •

edited by igrr

Loading

Humancell commented May 5, 2016

chaeplin commented May 5, 2016

Humancell commented May 5, 2016 •

edited

Loading

chaeplin commented May 5, 2016 •

edited

Loading

Humancell commented May 5, 2016

espmut commented May 15, 2017

melvinpmathew commented May 31, 2017

devyte commented Oct 17, 2017

mcanyucel commented Jan 29, 2018

espmut commented Jan 29, 2018 via email

Silberlachs commented Jan 30, 2018

mcanyucel commented Jan 30, 2018

d-a-v commented Jan 30, 2018

d-a-v commented Jan 31, 2018

martinius96 commented Feb 1, 2018

mcanyucel commented Feb 1, 2018

martinius96 commented Feb 1, 2018

mcanyucel commented Feb 1, 2018

d-a-v commented Feb 1, 2018

mcanyucel commented Feb 1, 2018

devyte commented Feb 20, 2018

mcanyucel commented Feb 20, 2018

Humancell commented Sep 15, 2018

EdoMon93 commented Jan 25, 2019

ligantx commented Mar 15, 2020 •

edited

Loading

AndrewIvashkin commented May 28, 2021 •

edited

Loading

http.POST() limit of payload size? #1872

http.POST() limit of payload size? #1872

Comments

TheAustrian commented Apr 6, 2016 • edited by igrr Loading

Humancell commented May 5, 2016

chaeplin commented May 5, 2016

Humancell commented May 5, 2016 • edited Loading

chaeplin commented May 5, 2016 • edited Loading

Humancell commented May 5, 2016

espmut commented May 15, 2017

melvinpmathew commented May 31, 2017

devyte commented Oct 17, 2017

mcanyucel commented Jan 29, 2018

espmut commented Jan 29, 2018 via email

Silberlachs commented Jan 30, 2018

mcanyucel commented Jan 30, 2018

d-a-v commented Jan 30, 2018

d-a-v commented Jan 31, 2018

martinius96 commented Feb 1, 2018

mcanyucel commented Feb 1, 2018

martinius96 commented Feb 1, 2018

mcanyucel commented Feb 1, 2018

d-a-v commented Feb 1, 2018

mcanyucel commented Feb 1, 2018

devyte commented Feb 20, 2018

mcanyucel commented Feb 20, 2018

Humancell commented Sep 15, 2018

EdoMon93 commented Jan 25, 2019

ligantx commented Mar 15, 2020 • edited Loading

AndrewIvashkin commented May 28, 2021 • edited Loading

TheAustrian commented Apr 6, 2016 •

edited by igrr

Loading

Humancell commented May 5, 2016 •

edited

Loading

chaeplin commented May 5, 2016 •

edited

Loading

ligantx commented Mar 15, 2020 •

edited

Loading

AndrewIvashkin commented May 28, 2021 •

edited

Loading