Performance degrades when using node-http-proxy to proxy http calls to EC2 instance

[sgerace]

My current setup is as follows: I have a Node.js server that is serving http requests (let's call that the main server) and another Node.js server that is proxying requests to the main server (running node-http-proxy v1.0.3). If the main server and proxy server are both running on my machine, everything works great and I don't see any drop in performance when going through the proxy. However, as soon as the main server is running on an EC2 instance (and the proxy still on my local machine), I see the average response time go from 50-100ms to about 800-3000ms per request when I make requests through the proxy vs. directly calling the main server. I'm kind of at a loss because it seems like as far as EC2 (or the network/internet) is concerned, the actual request leaving my machine and coming back to my machine should be the same in both scenarios, correct? If that is the case, then if feels like the issue is somewhere in node/node-http-proxy. However, then why isn't the performance impacted when the main server is running on my machine? Honestly I'm not even sure about the right way to go about debugging the issue and if anyone can provide some guidance I'd be more than happy to run some more tests to help point people in the right direction.

Also, I saw issues #334, #305, and #491, but I didn't feel like any of those could have the same root cause as the issues I'm seeing; feel free to correct me though if that assumption happens to be wrong.

[sgerace]

Alright, so now I'm really confused...I wanted to confirm that the problem was, in fact, something in node-http-proxy, so rather than proxying the request I used mikeal's request module to make the same request, and wouldn't you know it, it too was way slower. Okay, I thought, it is probably an issue in Node's http module, so I tried just using http.get, and to my surprise the requests consistently came back in 50-100ms, rather than the 800-3000ms it is taking everything else. SO, I have no idea whats going on now. Clearly there is something in common between node-http-proxy and mikeal's request that is causing this delay to be introduced. I'll continue to investigate, but if anyone else has any bright ideas I'm all ears.

Oh, the other thing is that I'm now connecting to my EC2 instance using an Elastic IP, so I wouldn't think there is any DNS resolution going on here (and even if there was, I would think it should affect the http module as well).

[jcrugzz]

@sgerace Could you provide a reproducible test case that is not reliant on EC2? It is also quite plausible it is your internet provider's latency to amazon and you only have a minimal sample size that possibly consists of spurious results. I would try and test this from an EC2 machine to another EC2 machine to see if you get the same behavior.

We run this in production on joyent with substantial traffic and see no latency issues.

[sgerace]

@jcrugzz The issue at the moment is that I'm only seeing it when going to EC2 (though the only other scenario I've tested against is a local server and I'm not sure that is a fair comparison to anything over the internet). I don't believe that it is my internet since the response performance is fine both from my browser as well as node when I'm using http.get calls. To illustrate, when I run code like this:

console.time('request');
http.get('http://<ec2_ip_address>/', function(res) {
    res.on('data', function (data) {
        // I verified that data contains the entire response
        console.timeEnd('request');
    })
});

The time reported is consistently between 50 and 100ms (same as if I make the request in my browser)

However, as soon as do something like this:

proxy.web(req, res, { target: 'http://<ec2_ip_address>/' });

OR

console.time('request');
request.get({
    url: 'http://<ec2_ip_address>/'
}, function (err, res, body) {
    console.timeEnd('request');
});

The response time is always > 1s.

After further searching, it sounds awfully like this Stack Overflow post.

[sgerace]

...and possibly nodejs/node-v0.x-archive#6481, though it doesn't exactly explain how to fix it and why it only effects higher-level interfaces (possibly because I wasn't piping the response in my http.get example?).

[sgerace]

Just tried adding

server.on('connection', function (socket) { 
    console.log('server.on.connection - setNoDelay'); 
    socket.setNoDelay(true); 
});

to both the main server and proxy server to no effect.

[sgerace]

@jcrugzz So here is where I'm at after messing with this over the weekend; I've placed the following server code up onto a public EC2 instance so you can test it yourself (to rule out the possibility of internet effects):

var http = require('http');

var server = http.createServer(function (req, res) {
    res.end('Hello World');
}).listen(3000);

// This was added to test what others have reported as fixing the issue
server.on('connection', function (socket) {
    'use strict';
    console.log('server.on.connection - setNoDelay');
    socket.setNoDelay(true);
});

The public ip address of the server is 54.209.167.233

When I request http://54.209.167.233:3000/ in my browser (I'm using the Advanced REST Client for chrome), I consistently get a response between 60-120ms.

If I run the following code, both cases respond for me in roughly the same amount of time (about 100ms each):

var http = require('http');

// Stream request
console.time('stream');
http.get("http://54.209.167.233:3000/", function(res) {
    var data = '';
    res.on('data', function (chunk) {
        data += chunk;
    });
    res.on('end', function (chunk) {
        console.log(data.toString());
        console.timeEnd('stream');
    });
});

// Pipe request
console.time('pipe');
http.get("http://54.209.167.233:3000/", function (res) {
    res.pipe(process.stdout);
    res.on('end', function (chunk) {
        console.log();
        console.timeEnd('pipe');
    });
});

However, if I run the following example, and then request the response from localhost, I get times in the 800-2000ms range:

var http = require('http');
var httpProxy = require('http-proxy');

var proxy = httpProxy.createProxyServer();
proxy.on('proxyRes', function (res) {
    res.pipe(process.stdout);
    res.on('end', function (chunk) {
        console.log();
        console.timeEnd('proxy');
    });
});

// Proxy requests
http.createServer(function (req, res) {
    console.time('proxy');
    proxy.web(req, res, { target: 'http://54.209.167.233:3000/' });
}).listen(3000);

For good measure, and it illustrate that I don't think the root cause is necessarily specific to http-proxy (though certainly something it is doing is causing the delay over the base http implementation), I provide the following additional example which uses mikeal/request to the same effect:

var request = require('request');

// Get using request module
console.time('request');
request.get({
    url: 'http://54.209.167.233:3000/'
}, function (err, res, body) {
    console.log(body);
    console.timeEnd('request');
});

This is as simple as I can make the cases and still illustrate the problem. Clearly it has something to do with EC2, but the fact that the http requests come back in a reasonable amount of time makes me think that it also has something to do with the way http-proxy (and mikeal/request for that matter) are handling requests at the Node.js level.

I'm going to go ahead and bring this up in the mikeal/request forum as well, to see if any others might have an idea what might be going on. I'll keep the server up and running as long as necessary to troubleshoot the issue.

Thanks!

[jcrugzz]

@sgerace This is very interesting. One test I think will be useful is to try it on 0.11.x just to see if it is associated with the http fixes that will exist in 0.12.x.

[vkadam]

While doing pipe, when I set keepAlive=true for agent, I got the response in near time as direct call, around 3-5ms extra.

[zoheb]

Is there a resolution to this? I am seeing the same issue.

[antouhou]

I've got the same issue, and for me setting up http.agent.maxSockets to 10000 solve the problem. Not sure why, since maxSocket defaults to infinity.

var httpsAgent = https.globalAgent;
httpsAgent.maxSockets = 10000;

var proxy = httpProxy.createProxyServer({
  agent: httpsAgent
});

[manueltimita]

Perhaps this will help those for whom the keepAlive / maxSockets settings don't make a difference: it may very well be down to DNS lookups. I noticed that even when proxying locally, to a virtual machine, if the target is a hostname rather than IP, the response is 100 to 150 ms slower

[alexw23]

👍 getting this when using a host on /etc/hosts, Node v5 if that helps

[Kulikowski]

Is there any other fix for this? I am seeing the same issue.

[indexzero]

The solution to this issue is to use a custom agent or to up the sockets on the default agent, which by default in many older vesions of node is set very low (5 sockets). If you are running on node@4, however, the default number of sockets is high.