Slow performance of FSBrowser on Chrome and Firefox

[rogerclarkmelbourne]

Hi,
I'm using the latest version of the repo from a few days ago, and I'm using the FSBrowser example as a base for a larger sketch.

I'm seeing very slow transfers to Chrome and Firefox on WIndows 7, but Internet explorer is fine.
I've also tested on Android (Firefox) and iOS Safari, and they do not have this problem.
Nor does Safari on OSX.

As an example. a 13k gzipped html file on IE 9 takes less than 100 milliseconds to load
However the same file on Chrome or Firefox takes a minimum of 1.8 secs to load and sometimes as long as 2.5 seconds.

The page is self contained and served from SPIFFS

So this looks like its an issue with the transfers to these specific browsers

Edit.
I tried reducing the HTTP_DOWNLOAD_UNIT_SIZE to 1024 but it didn't make any difference.

I guess I'm going to need to some packet analysis with WireShark :-(

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[me-no-dev]

seems like the browsers are waiting for more data. That is what usually causes delays around 2 seconds.
Any chance your content-length is wrong?

[rogerclarkmelbourne]

I'm just using the FSBrowser example without any modifications

The code that sends the file back to the browser is

bool handleFileRead(String path){
  unsigned long m=millis();
  DBG_OUTPUT_PORT.println("handleFileRead: " + path);
  if(path.endsWith("/")) path += "index.htm";
  String contentType = getContentType(path);
  String pathWithGz = path + ".gz";
  if(SPIFFS.exists(pathWithGz) || SPIFFS.exists(path)){
    if(SPIFFS.exists(pathWithGz))
      path += ".gz";
    File file = SPIFFS.open(path, "r");
    size_t sent = server.streamFile(file, contentType);

    DBG_OUTPUT_PORT.println(millis()-m);
    return true;
  }
  return false;
}

I just added the stuff to print the millis()

Looking at

server.streamFile(file, contentType);

which I think is the issue...

It does this

template<typename T> size_t streamFile(T &file, const String& contentType){
  setContentLength(file.size());
  if (String(file.name()).endsWith(".gz") &&
      contentType != "application/x-gzip" &&
      contentType != "application/octet-stream"){
    sendHeader("Content-Encoding", "gzip");
  }
  send(200, contentType, "");
  return _currentClient.write(file, HTTP_DOWNLOAD_UNIT_SIZE);
}

So its reading the file size from the SPIFFS

When I look at chrome's network display, it behaves as if the data is comming from a very slow source, i.e the progress bar slowly fills in.

What I don't really understand is quite how the data is being written in chunks back to the browser. There appears to be a setting

#define HTTP_DOWNLOAD_UNIT_SIZE 1460

which is passed to

_currentClient.write(file, HTTP_DOWNLOAD_UNIT_SIZE);

However I'm not sure where the call tree goes from this point.

Because _currentyClient is a WifiClient, but WifiClient doesnt have a write method that matches (File, int)

well it has

  template <typename T>
  size_t write(T& source, size_t unitSize);

Which I presume is what is being called

But quite where the actual code that gets executed is, I'm not sure.

[rogerclarkmelbourne]

Update.

I've looked at the transfers in Wireshark and it just looks like individual packets have delays between them.

I don't know if its the ESP8266 which is delayed in sending them, or whether its Chrome etc that is not requesting the next packet straight away.

I've compared it visually (in wireshark) and IE gets all the packets

Trying to compare whats happening on 2 different WireShark captures, there appears to be a 200ms delay between Chrome receiving one packet from the ESP8266 and it requesting the next packet.
Where as on IE there is only 0.06 milliseconds delay before IE requests the next packet.

The delay of 200ms on Chrome and Firefox does point to some internal time-out because the TCP code doesn't like what its been sent.

[drmpf]

I regularly measure 200mS on windows between packets, 10 to 40mS on Mac and Android
Not sure why IE is quicker.
Esp only handles one packet at a time (due to need to handle re-requests)
see #922

[igrr]

Could it be related to Windows delayed acks? https://support.microsoft.com/en-us/kb/823764

If you make a network trace with a network sniffer such as Microsoft Network Monitor, the TCP server sends a TCP ACK segment to the last TCP segment in a TCP data stream in the delayed acknowledgement timer (also known as the delayed ACK timer). By default, for Windows operating systems, the value for this timer is 200 milliseconds (ms).

[rogerclarkmelbourne]

Hi Ivan,

I found that as well when I looked for the 200ms delay ;-)

This is an absolute performance killer on Windows. As the packet size is about 1.5k, which then limits data rates to 5 packets per second or around 7.5k per second.

Its strange it only applies to Chrome and Firefox and not IE. I thought that they would all use the same Windows TCP services and not have their own TCP stack.

BTW. I have tested this on multiple machines, 2 different Windows 7 machines and also a Windows 8.1 machine. I have XP as well, but it takes some time to boot, so I've not bothered to try that as yet.

[drmpf]

My biggest issue is that the whole sketch is blocked for 10mS to 200mS, so I modified the code to make it non-blocking.

[rogerclarkmelbourne]

@drmpf

Thanks. I will try your code tomorrow ( as its 22:40 local here and too late to try it now)

It will be interesting to see if this is the same issue, because as far as I can see, it's the client end that seems to be inserting the delays, but I'm not an expert in TCP/IP or Wireshark, so the issue could be at the ESP8266 end, in my case

[me-no-dev]

you guys can you post here part of the wireshark results? I'm interested in the ACK packets timing compared to the data packets. More data on the packets will also help.
I do not think that the browsers use a different TCP stack then IE but maybe IE uses more options for the socket (TCP_NODELAY and such). If the content-length is set correctly, the browser should immediately close the connection regardless of options. In our WebServer implementation we set proper headers and generally do everything we can to get the browser to close as soon as the last byte is received.

[rogerclarkmelbourne]

OK. I will run Wireshark for both IE11 and Chrome on Windows 7 and upload the results somewhere they can be downloaded from.

BTW.
What format do you want the results to be saved in. Wireshark seems to save in loads of different formats.

[me-no-dev]

even a screenshot of the wireshark window will do. Information needed is visible in the grid

[rogerclarkmelbourne]

LOL
A screenshot was actually what I ended up using myself to compare the previous and last run's

But of course it doesn't let you dig down into more details.

BTW, there were also some differences when IE started to download the page, from what Chrome did.
but I didn't understand what the differences actually meant.

[rogerclarkmelbourne]

I tried to save the packet captures, but the files seem to be huge. (10Mb, perhaps this is because they save before the filters are applied)

Anyway, I've put the screen grabs in a google drive, as I can't seem to upload them to this github issue

With a bit of luck you should be able to access the shared folder here

https://drive.google.com/folderview?id=0B4AdvMiTQuacMzBBbjl3ZHdKUTQ&usp=sharing

[me-no-dev]

delayed ACK all the way... nothing you can do about it, neither can we...

[rogerclarkmelbourne]

Sorry I dont understand what you mean.

Delayed ACK from which end ?

I see delays in Chrome requesting more data.

How come I dont see the same thing when Chrome requests data from other web servers.?

[igrr]

Other web servers with more RAM have larger TCP windows, and support many segments in flight. With multiple segments in flight, ACK delays are not visible.

[me-no-dev]

I can clearly see in the screenshots that in the first one the ACKs from the browser are delayed. Not a delay from the ESP. Browser really delays the ACK for some reason.

[me-no-dev]

if I got it right .100 is the ESP and .120 is the PC

[rogerclarkmelbourne]

Yes.

PC is 120
Esp8266 is 192.168.1.00

Sorry I should have already described which was which.

Are there any pure SDK examples of running a web server. (actually I think I have an old linux VM that may have the old SDK running as a web server, I'll need to check)

[me-no-dev]

the problem is in the computer, not in the ESP. Regardless of what implementation you use the result should be exactly the same.

[rogerclarkmelbourne]

OK.
I'll run some tests using different servers.

Sorry. I've not had time to do this today, I've been struggling with another esp8266 issue.

[rogerclarkmelbourne]

I've not had time to test another sever yet, but I did try Safari on Windows (yes, I know its discontinued)
And it was just the same as Chrome and Firefox.

[JarKol]

There is a way how to force ACK from Win. If another packed arrived, previous packed is confirmed by ACK anyway (before timeout). So there is question if esp8266 can send another packed before is ACK received. I try to find that information, but I am not sure about that yet.

[drmpf]

My understanding is that ESP can only handle one outgoing packet at a time, only has one buffer.
So needs to keep that packet for resends until ACK received.
@igrr may have a more details.

[rogerclarkmelbourne]

In the screengrabs, its the PC (192.168.1.120) which is delaying sending the ACK

e.g. If you look at line 13 its the EPS sending 194 bytes to the PC
But the PC does not respond for almost 200ms (line 15)

I suspect that this is a timeout because the packet that was sent on line 13 was somehow incorrect, perhaps the amount of data sent didnt match what the browser expected, or perhaps there is something else wrong with the structure or data of the packet

[rogerclarkmelbourne]

OK.

I see what @iggr means.

The delay is caused by optimisation (concatenation) of the ACK's by Chrome and Firefox (where as IE does not concatenate)

See

https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment

I'm not sure if the non-blocking code fixes this, i.e whether we can control things at that level of the TCP stack, or whether there is a way of telling Chrome and Firefox that they should not do delayed ACK

[JarKol]

@rogerclarkmelbourne Yes you are right, there is many information about that delay called "TCP delayed acknowledgment":
https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
https://support.microsoft.com/en-us/kb/214397

So it is default setting of any Windows OS and probably on Linux is this delay too (maybe shorter period, but I dont know it at exactly). In that case it will be good think about solution on ESP side, bacause that problem is on great % of all end devices with TCP/IP communication.

[rogerclarkmelbourne]

I think we cross posted ;-)

PS. I forgot to say. I did a screengrab of Wireshark of a download from IIS on a Windows machine to Chrome on another and I've uploaded it to the same google drive folder as posted above.

Chrome only ACK's every 2 data packets.

[JarKol]

@rogerclarkmelbourne Yes, we really cross posted :) But I dont think, that it is problem mainly on browser side. It is about OS settings. Some versions of IE maybe can "bypass" this setting, but it is not case of my version of IE (Vista 32bit, IE 9.0.33).
There is manual how to disable this setting in Win registery, but it is not the best solution for end user of web app hosted by ESP:
http://www.icpdas.com/root/support/faq/card/software/FAQ_Disable_TCP_ACK_Delay_en.pdf

[rogerclarkmelbourne]

LOL We keep doing it !

[rogerclarkmelbourne]

@ drmpf seems to have a non-blocking lib , but I have not tried it yet

[rogerclarkmelbourne]

http://www.forward.com.au/pfod/pfodParserLibraries/pfodESP8266WiFi.zip

[rogerclarkmelbourne]

BTW.

Changing the windows setting is not going to work for a lot of people, i.e if you want to use a ESP8266 as a local mini server via wifi for anyone to use, then people will not want to change their settings

I think we need to investigate the non-blocking solution where the next packet is sent as long as less than 500ms has elapsed (as this is the time in the TCP spec according to wikipedia)

[JarKol]

@rogerclarkmelbourne Yes, I agree, that it is not a good solution as I write in previous post.

Yes, we need to have two packet in sending buffer at same time because after client recive second packed, first packed is ACKed at that time before delay ended...
But I am not sure that ESP can do that. @drmpf

[drmpf]

My non-blocking lib does nothing about the 200mS delay from Windows (only 10mS to 40mS from Android and Linux) although it does buffer WiFiClient.writes() to reduce the number of packets.

ESP stack needs to support more then one packet on the fly at the same time.

[JarKol]

@drmpf If I can understand it correctly non-blocking mean something like that:
https://en.wikipedia.org/wiki/Nagle's_algorithm
I am right ?

[drmpf]

Actually the 'buffering' part of my library works like that. WiFiClient.write writes to a buffer and then sends when buffer full OR nothing written for 10mS.
By 'non-blocking' I mean calls to WiFiClient.write return immediately without waiting for ESP to send the packet and get an ACK.
Ofcourse if you fill up the buffer AND the ESP is still waiting for an ACK on the previous packet then the sketch will block.
If the sketch is blocked for 5sec then it returns and closes the connection.

[JarKol]

@drmpf Thanks for explaining.

About that slow ACK I am now experimenting with variable TCP timeouts settings to force client send ACK earlier but no success yet. Is here someone expirienced in TCP/IP communication, who can tell me if there is some solution through this way please ?

[rogerclarkmelbourne]

There are various postings on stackoverflow but none of them seems to have a sever side solution to this :-(

[mkeyno]

hi @rogerclarkmelbourne , I have same problem when browser request for gzip Compressed content , webserver library only send Content-Type: application/x-gzip and sendHeader("Content-Encoding", "gzip"); not work properly

[va7ta]

Greetings,
I have run into this same issue where initial connection is very slow with both FireFox and Chrome but OK with MS Edge and IE. I also happen to have an ESP8266 application that I programmed with the NodeMCU LUA language alternative (I much prefer Arduino). I can say for certain that I have not encountered a similar delay with LUA using either FireFox or Chrome. This leads me to believe that this issue is not due to a limitation of the ESP8266 silicon. One would think that it should be possible to have the Arduino library perform in a manner similar to NodeMcu LUA. Possibly this issue should be raised to bug status.
tma

[va7ta]

Further to my last I decided to compare Windows 10 Firefox and MS Edge for the time it takes to load my rather tiny 1,351 byte index.htm webpage file. For the initial load Firefox takes about 90 seconds or roughly an average of 67 mS per byte and Edge roughly 9 seconds, about 10 times faster. I placed a debug loop counter in the send byte loop to confirm the send-to-PC byte count which matches the 1,351 byte file size thus there does not seem to be any repeat requests.

I agree that a modification to the Windows registry to correct the problem is not a practical solution for the masses trying to make use of an IOT device. Interesting that the problem does not occur with NodeMCU code.

I noticed there hasn't been a comment added regarding this issue in over 6 months. I wonder if this matter remains of interest to anyone? Any comments?

[duncan-a]

Well, it's of interest to me but, due to lack of competence, I've got nothing to contribute to the discussion - other than I hope somebody comes up with a viable solution. 90 seconds is way past 'back-out' time...

…

On 17 December 2016 at 03:45, va7ta ***@***.***> wrote: Further to my last I decided to compare Windows 10 Firefox and MS Edge for the time it takes to load my rather tiny 1,351 byte index.htm webpage file. For the initial load Firefox takes about 90 seconds or roughly an average of 67 mS per byte and Edge roughly 9 seconds, about 10 times faster. I placed a debug loop counter in the send byte loop to confirm the send-to-PC byte count which matches the 1,351 byte file size thus there does not seem to be any repeat requests. I agree that a modification to the Windows registry to correct the problem is not a practical solution for the masses trying to make use of an IOT device. Interesting that the problem does not occur with NodeMCU code. I noticed there hasn't been a comment added regarding this issue in over 6 months. I wonder if this matter remains of interest to anyone? Any comments? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1027 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALP8xRE9cjYCYbFOGmJy_zgWPdw4VW4Fks5rI0zLgaJpZM4GiukO> .

[andig]

I guess we'll have to wait for sdk 2.0 and see if the upgraded lwip can do anything for us?

[daOEKL]

Hi guys, did anybody find out, why one get's trouble if there are more than 3 request on an esp running as webserver ?
For me it is of interest too but I wasn't skilled enoug either to find the reason.

So biggest hint so far was that there aren't any problems with Lua..
Any news ?

[devyte]

The Lua firmware also has trouble with multiple requests. I could get 2, sometimes 3 requests, but never 4 working. The reason in that case was memory: each request ate just over 10k. I would expect this firmware to work better, considering that the mem requirements are far lower. What comes to mind is that the webserver is a globally declared singleton, and it probably has state. How does the object behave when there are multiple clients with different requests? Most likely the object state gets messed up. That's just conjecture, though. Did you try asyncwebserver? It's also used via a globally declared singleton, but maybe it handles multiple requests better. On Mar 17, 2017 8:31 PM, "daOEKL" <[email protected]> wrote: Hi guys, did anybody find out, why one get's trouble if there are more than 3 request on an esp running as webserver ? For me it is of interest too but I wasn't skilled enoug either to find the reason. So biggest hint so far was that there aren't any problems with Lua.. Any news ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1027 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQC6Bhyf9_2W0PqYUKCePNFAp2jR5J-gks5rmxfCgaJpZM4GiukO> .

[va7ta]

Greetings,

I believe there are multiple issues being discussed here.

There is a browser related delay with the Arduino library which is the subject of this thread. I found Firefox and Chrome to be extremely slow establishing initial connection with the Arduino library whilst IE and Edge are just a bit sluggish. I did not notice a BROWSER dependent problem with LUA.

I did however find that there is a different issue with LUA which I never tracked down. I found it would work for a few connects and then just stop connecting until reset. I wondered if possibly the previous connections were not being dropped and cleared from memory. Possibly the previous connections remained in memory instead of being cleared properly after the connection was dropped eventually filling all available memory. I have not noticed this issue with the Arduino library. I have never attempted multiple simultaneous connections with either Arduino or LUA.

I also noticed that with Arduino there is sometimes a data transfer delay after it has responded properly to several previous data transfers. Suddenly the data transfer will stop and then after a few attempts and further delay all the missing data transfers will come through, as if the data from previous failed attempts were held in a buffer waiting for a clear to send signal. This may in fact be another Arduino related issue.

LUA seemed to connect OK for me with Firefox and Chrome suggesting that the main issue here for this thread is uniquely Arduino library related.

Possibly a new thread should be created for the multiple connection issue with LUA.

[devyte]

@d-a-v I seem to remember from somewhere that #3362 could help the FSBrowser example slowness under Chrome? I can't find the specific discussion anymore, maybe I'm too dizzy from looking at so many issues...

[d-a-v]

I don't remember. I made some tests anyway. I uploaded the FSBrowser and some random files. With current master (lwip1.4) I had no problems with Chrome. Then I tried this script:

(while true; do curl http://esp8266fs.local/esp-dhcpserver.c 2>/dev/null; done) | pv > /dev/null

... and all went well, even while loading the same page through chrome at the same time.
So I put my finger on the antenna, and of course bandwidth stopped. But I also had a reset.
So I tried with lwip2 and the same finger. Bandwidth stopped, but started again with finger removed, and no hang. Repeatedly (this was too the kind of test I made while trying to debug lwip1.4).

So to say that you all should try again with the current master. I think, as per #2925, that tcp handling was repaired since you all last tried and it seems now OK (without #3362).

-- fingerpower :)

[devyte]

This issue is old, a lot has happened since. I just tested with chrome and firefox and I can't reproduce slowness.
Closing.
If you still encounter the problem, please open a new issue and follow the template instructions.