performance

[manast]

I would like to open a thread regarding performance, because I think it requires a bit more of attention than it has actually received lately.

Although this project does not aim to be as performant as other leading proxies such as HAProxy or nginx, I think most of its users would certainly be happy if the proxy does not degrade the performance to a nodejs server by a factor of 10x or 15x.

This closed issue: #929 shows that it is very easy to actually verify the performance degradation produced by the proxy. The workaround is to send a http.Agent with keepAlive: true, and maybe other finetunings. The issue refers to a FAQ, but I cannot find any FAQ. Also it would be great to provide a couple of things regarding http.Agent: 1) what is the tradeoff of using it? (if none, why is it not enabled by default), 2) Which is the optimal set of options for it? just changing arbitrarily and re-testing does not seem like a good approach, and also a dangerous one I may add, since the user most probably does not know what he is doing. 3) Why is http.Agent required to begin with?

One thing that may make many wondering is how is it possible that if a dummy http server in node js is capable of delivering responses in the order of magnitud 10k, a simple proxy infront of it, that should, in its most basic conceptual form, just pipe the data from source to target, reduces the performance to order of magnitude 0.5k. One could accept 50% degration, but not this much. Thing is, this may not be related to http-proxy at all, for instance I wrote this super simple example:

var http = require('http');
var request = require('request');

http.createServer(function(req, res){
  request.get('http://127.0.0.1:3000').pipe(res);
}).listen(8080);

http.createServer(function(req, res){
  res.writeHead(200);
  res.write('HELLO WORLD');
  res.end();
}).listen(3000);

And got this results (in node 4.5):

$ wrk -c 64 -d 15s http://localhost:3000
Running 15s test @ http://localhost:3000
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.85ms    3.41ms 307.55ms   99.38%
    Req/Sec     6.73k   505.00     7.10k    93.38%
  202128 requests in 15.10s, 24.87MB read
Requests/sec:  13384.99
Transfer/sec:      1.65MB

$ wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    55.45ms   12.50ms 158.42ms   88.76%
    Req/Sec   569.33    128.19   760.00     81.94%
  8265 requests in 15.07s, 0.98MB read
Requests/sec:    548.31
Transfer/sec:     66.40KB

So can it really be node's streams are so amazingly slow? It's a bit worrisome I must admit. Anybody as any insights that he wouldn´t mind to share?

[manast]

I did two more experiments that I find somehow interesting.

Using minimum http-proxy(I already got the same figures using https://github.com/OptimalBits/redbird based on http-proxy):

var httpProxy = require('http-proxy');
httpProxy.createProxyServer({target:'http://localhost:3000'}).listen(8080);

Gives basically the same result as using request streaming:

$ wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    52.03ms   10.90ms 119.16ms   87.05%
    Req/Sec   607.21    104.95   747.00     79.41%
  8257 requests in 15.04s, 0.98MB read
Requests/sec:    548.97
Transfer/sec:     66.48KB

On the other hand, using req-fast (https://github.com/Tjatse/req-fast) I get consistently this:

var reqFast = require('req-fast');
var http = require('http');

http.createServer(function(req, res){
  reqFast('http://127.0.0.1:3000').pipe(res);
}).listen(8080);

$ wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    49.73ms    9.96ms 142.05ms   89.36%
    Req/Sec   648.36    136.26     0.97k    72.80%
  16320 requests in 15.03s, 2.01MB read
Requests/sec:   1086.16
Transfer/sec:    136.83KB

So about twice as fast as request and http-proxy. Meaning that it is possible to implement a streaming proxy that is at least that fast inside http-proxy.

[manast]

Using keepAlive: true on the http.Agent with http-proxy:

wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    27.49ms    4.07ms 118.95ms   94.56%
    Req/Sec     1.17k   126.61     1.43k    88.67%
  35017 requests in 15.03s, 4.31MB read
Requests/sec:   2330.04
Transfer/sec:    293.53KB

Using keepAlive with req-fast:

var keepAliveAgent = new http.Agent({ keepAlive: true });
http.globalAgent = keepAliveAgent;

$wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.62ms    7.44ms 102.17ms   85.51%
    Req/Sec   656.12     98.75     0.94k    79.84%
  16294 requests in 15.08s, 2.00MB read
Requests/sec:   1080.32
Transfer/sec:    136.09KB

So http-proxy gets a huge 4 times better performance with keepAlive, while req-fast stays the same.

[manast]

Previous experiments were aimed to just check what is the overhead for a minimal web server, now check this results when serving strings of different sizes from 32 bytes to 256Kb.

Test code:

var http = require('http');

var httpProxy = require('http-proxy');
var keepAliveAgent = new http.Agent({ keepAlive: true, maxSockets: 1000 });

var randomstring = require("randomstring");
var msg = randomstring.generate(2*1024);

httpProxy.createProxyServer({target:'http://localhost:3000', agent: keepAliveAgent}).listen(8080);

http.createServer(function(req, res){
  res.writeHead(200);
  res.write(msg);
  res.end();
}).listen(3000);

In this case I used needle for implementing a proxy since it gave best performance than req-fast and request:

var needle = require('needle');

http.createServer(function(req, res){
  needle.request('get', 'http://127.0.0.1:3000', null, {agent: keepAliveAgent, connection: 'keep-alive'}).pipe(res);
}).listen(8080);

And the results:

For me, the interesting here are basically 3 things:

• There is a lot of overhead to setup the streaming.
• For strings of size 32 and above, the proxy overhead is negligible.
• The streaming gives best performance with strings of size 32Kb, after that it starts degrading, which is strange, I was expecting less and less overhead so more and more raw throughput with large strings.

[manast]

I filled the gaps to get a more linear chart:

[dtjohnson]

I would definitely like to see this addressed as well. I've been using Nginx as a dynamic reverse proxy for some time now. It's worked well, but the Nginx configuration spec is massive and confusing. And the dynamic part requires scripting in Lua, which isn't the easiest. I'd like to implement some more sophisticated features to my proxy, but that's painful in Nginx/Lua. It would be much simpler to do in Node.js, but I'm running into the same performance issues as manast with this module.

I did my own benchmarking. I spun up a Docker container first with a simple Nginx server serving static text (because I knew it would be lightning fast). I spun up a second Nginx container set up as a reverse proxy for the first to use as a comparison. I then spun up a Node.js 6.3 container testing a proxying with this module and the built in Node http client as well as serving static content for comparison. I benchmarked with wrk using 10 and 100 connections. Then I repeated the whole process with a Node.js source (instead of the Nginx one) introducing a 100 ms delay before serving content.

Here is my proxy code:

const http = require('http');
const httpProxy = require('http-proxy');
const keepAliveAgent = new http.Agent({ keepAlive: true });

// Plain HTTP server w/ static content.
http.createServer((req, res) => {
    res.statusCode = 200;
    res.setHeader('Content-Type', 'text/plain');
    res.end('Hello World\n');
}).listen(8000);

// http-proxy w/ default global agent
const defaultAgentProxy = httpProxy.createProxy();
http.createServer((req, res) => {
    defaultAgentProxy.web(req, res, {
        target: "http://10.224.51.210:8080"
    });
}).listen(9000);

// http-proxy w/ keep-alive agent
const keepAliveAgentProxy = httpProxy.createProxy({ agent: keepAliveAgent });
http.createServer((req, res) => {
    keepAliveAgentProxy.web(req, res, {
        target: "http://10.224.51.210:8080"
    });
}).listen(9001);

// http-proxy w/ no agent
const noAgentProxy = httpProxy.createProxy({ agent: false });
http.createServer((req, res) => {
    noAgentProxy.web(req, res, {
        target: "http://10.224.51.210:8080"
    });
}).listen(9002);

// Node http client w/ default global agent
http.createServer((req, res) => {
    http.get({
        hostname: '10.224.51.210',
        port: 8080,
        path: '/'
    }, upstreamRes => {
        upstreamRes.pipe(res);
    });
}).listen(10000);

// Node http client w/ keep-alive agent
http.createServer((req, res) => {
    http.get({
        hostname: '10.224.51.210',
        port: 8080,
        path: '/',
        agent: keepAliveAgent
    }, upstreamRes => {
        upstreamRes.pipe(res);
    });
}).listen(10001);

// Node http client w/ no agent
http.createServer((req, res) => {
    http.get({
        hostname: '10.224.51.210',
        port: 8080,
        path: '/',
        agent: false
    }, upstreamRes => {
        upstreamRes.pipe(res);
    });
}).listen(10002);

And here were the results:

The Nginx and Node.js static results are just there as a theoretical floor. The proxy server can't possible be faster than just serving static content. Nginx was definitely faster, but not quite as fast as I thought. It does perform better with 100 connections, but that's probably just because Node is using a single thread and Nginx is using 2 (one per CPU on my server).

The Nginx proxy results are the real target. If we can get close to the performance of Nginx we're in great shape.

With the node-http-proxy module, we see a 2x latency and 1/2 the requests compared to Nginx. I didn't see any difference between the default global agent (no keep-alive and infinity sockets) and no agent at all. Unexpectedly, when I enabled keep-alive the latency quadruples for 10 connections. For the 100 connections it performed much better.

I also proxied using the built-in http client to make a request to the upstream source and piped the results into the response. I did this with and without keep-alive. The results with keep-alive were fantastic! Even with a single thread the response times were lightning fast--even faster than the Nginx proxy--in all scenarios.

So what am I missing? Why is there such a huge latency added by node-http-proxy?

[dtjohnson]

I just did a little more benchmarking. This time I put the source Node.JS container on one server, the proxy containers on a second server, and the wrk container on a third to get a more representative benchmarking and to avoid any competition of resources. I also added HAProxy to the mix and went up to 1000 connections. I also added a test with the Node.JS http client using cluster so there are 2 Node.JS workers (one per CPU). These are all with Node.JS 6.5. Here were the results:

Here "Direct" means direct access to the source server with no proxy. There's a lot of network fluctuation, but it looks like the direction connection, HAProxy, and the Node.JS HTTP client piping with cluster are all about equally performant. node-http-proxy, by comparison, performs horribly. :( I was also surprised to see how poorly Nginx performed.

Here's my nginx.conf

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    server {
        listen 8080;
        location / {
            proxy_pass http://source.mydomain.com:8080;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

Here's my haproxy.cfg

global
    maxconn 1024

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in
    bind *:8080
    default_backend default-server

backend default-server
    option http-keep-alive
    server s0 source.mydomain.com:8080

Here's my node-http-proxy JS code

const http = require('http');
const httpProxy = require('http-proxy');
const proxy = httpProxy.createProxy();
http.createServer((req, res) => {
    proxy.web(req, res, {
        target: "http://source.mydomain.com:8080"
    });
}).listen(8080);

Finally, here is my custom Node.JS HTTP client piping proxy with cluster (not sure if I got the error/abort handling done correctly though):

const cluster = require('cluster');
const http = require('http');

const numCPUs = require('os').cpus().length;
const keepAliveAgent = new http.Agent({ keepAlive: true });

if (cluster.isMaster) {
    for (var i = 0; i < numCPUs; i++) {
        cluster.fork();
    }
} else {
    http.createServer((req, res) => {
        const proxyReq = http.request({
            method: req.method,
            path: req.url,
            headers: req.headers,
            hostname: 'source.mydomain.com',
            port: 8080,
            agent: keepAliveAgent
        }, proxyRes => {
            res.writeHead(proxyRes.statusCode);
            proxyRes.pipe(res);
        });

        req.pipe(proxyReq);

        proxyReq.on("error", e => {
            res.write(e.message);
            res.end();
        });

        req.on("error", e => {
            proxyReq.abort();
            res.write(e.message);
            res.end();
        });

        req.on('aborted', function () {
            proxyReq.abort();
        });
    }).listen(8080);
}

[manast]

Quite interesting results. But how large is the data being proxied? as you can see in my results about it has a lot impact in the results. In any case, it will be interesting to know why http-proxy is performing so bad, I have to check the sources but I guess it is using http.request internally...

[dtjohnson]

Following your suggestion, I ran the benchmarks with a variety of message lengths (from 1B to 1MB) using the randomstring module like you did. I didn't see the same issue as you. I wonder if you are seeing some strange behavior by running your proxy server and your upstream source on the same single Node.js thread. The results were very interesting though. Here are the results with the same 3 server configuration but with an even faster upstream source.

Here is 10 connections:

And zoomed in on the <= 1kB messages:

Now 100 connections:

And zoomed in on the <= 1kB messages:

Now 1000 connections:

And zoomed in on the <= 1kB messages:

HAProxy performs very well, adding only a small overhead in all scenarios. Node.JS client piping makes a decent showing. Nginx performs surprisingly poorly, and node-http-proxy lags far behind. node-http-proxy failed completely in most of the 1000 connection runs.

[manast]

I guess more people is needed to verify this results, but if so, this means that is not unrealistic to say that we could have a nodejs based proxy that is competitive enough to be used for high traffic sites. Its still quite amazing that nginx performs so bad in these tests, I wonder if there is not something in the configuration or the setup that makes it perform so bad (disclaimer: I am a novice in nginx).

[manast]

Btw, it would be also highly relevant to test with HTTPS. We should agree that any serious site should work using HTTPS anyway, so that is the performance we should care about :)

[dtjohnson]

I came to the same conclusion. Despite whatever is going on with node-http-proxy and Nginx, it certainly seems a Node.js-based proxy is a realistic option.

I'm stumped as to what is wrong with Nginx. I have to believe it is some configuration issue. I tried disabling proxy buffering and caches--no help. I tried using IP address instead of DNS for the target--no difference. I disabled keep-alive (which I need to support server-sent-events) and that did improve performance but still well below the the performance of Node.js.

HTTPS is a good idea to test too. I tend not to think about that as in my setup my proxies are fronted by an AWS Elastic Load Balancer that does SSL termination. Would be interesting to see any differences there though. Gzip on the proxy would interesting too.

One other point of consideration is that Nginx and HAProxy in my tests aren't set up with scripting. They are just fixed proxying. In the Node.js proxy we have access to the entire JS language and libraries. To do a fair test with Nginx and HAProxy we'd have to test dipping into a Lua script execution on each request to do a fair comparison.

I'd love to script the entire benchmarking workflow so we can iterate on this more quickly and consistently. So dynamically spin up 3 servers, start the proxy servers, benchmark them, and then terminate the servers.

[manast]

As I mentioned above I am the author of https://github.com/OptimalBits/redbird, and I am using http-proxy for the actual proxy work, but maybe it is not so difficult to implement a http.request based proxy instead. The risk is that there are probably a lot of cases, common as well as edge, regarding handling of headers and other stuffs that are not so easy to get right without a long period of battle testing such as http-proxy already has have.

[dtjohnson]

Agreed. It's the variety of cases I worry about too, but I'm afraid it's a bridge I may have to cross. I don't think I'll be able to evolve my current dynamic Nginx proxy easily enough (especially seeing the Nginx performance issues from the above benchmarking). I really think I need to swtich to a Node.js based one.

Would love to see the authors of node-http-proxy chime in. I feel like I must be doing something majorly incorrect that would explain the performance issues...

[eezing]

@manast @dtjohnson Have you taken a look at #614? Providing node-http-proxy an https agent and setting maxScockets may help. There also appears to be potential slowdown from DNS when setting target other than ip address. May not make a difference in Node v4+

[dtjohnson]

@eezing, yeah, I tried a variety of options. See my first post on this thread. Keep-alive performance was definitely the worst option, but I need it for server-sent events (though I do have an alternative idea for that).

So I went ahead and automated the full benchmark process:
https://github.com/dtjohnson/proxy-benchmark
The code spins up 3 AWS EC2 servers on-demand (one for the upstream, one for the proxies, and one for wrk). It then runs through a suite of benchmarks. The results are viewable here:
https://dtjohnson.github.io/proxy-benchmark/

Here's an image, which is fairly consistent with the one above:

This tool should make it easy to iterate on the Node proxy and see relatively quickly the performance implications of various configurations.

A number of next steps to try:

• Proxying headers (the current Node piping just sends the status code, no headers). I expect this will hurt performance.
• SSL
• Gzip
• No keep-alive
• Piping the sockets. There is an intriguing looking piece of code showing piping the underlying sockets here: https://nodejs.org/api/http.html#http_event_connect. I'm curious how that will perform.

[dtjohnson]

Hmmm... Still no response?

I played some more with the Node proxy. In my examples before I wasn't sending the proxied response headers back. When I added it with:

res.writeHead(proxyRes.statusCode, proxyRes.headers);

The performance dropped dramatically--from 5ms to 40ms latency. I figured out it was because the upstream server sent a Content-Length header that was sent in the response. That prevented Node from using Transfer-Encoding: chunked. Deleting the header (and the Connection header) before sending the response restored the performance (code here).

I looked at the proxy with node-http-proxy and it wasn't using Transfer-Encoding: chunked. I wonder if that might be responsible for the performance hit we're seeing. Unfortunately, I couldn't figure out from the docs how to enable chunked transfer or how to modify the response headers.

[manast]

@dtjohnson strange that having content-length prevented node from using chunked transfer, this needs to be verified somewhere, does not makes complete sense to me :/.

[manast]

ok, you are right: https://en.wikipedia.org/wiki/Chunked_transfer_encoding
Buy still, the content should be streamed in chunks, there should not be any major performance differences.

[jcrugzz]

Hmm thats a good point, this requires a bit more investigation into how node core behaves. Its not a bad idea to have certain headers that get stripped but we may want to make that opt in or it would be a breaking change. Thoughts?

[jcrugzz]

In regards to the overall discussion here. Im +1 to have performance optimizations considered and implemented into this. My approach to start this was to extract some of the core logic out of http-proxy into http-proxy-stream.

In reality we shouldn't be able to beat the performance of nginx or haproxy but we should do as best we can while maintaining 100% correctness given the foundation of node that we build upon.

[dtjohnson]

Unless my nginx configuration is completely wrong (which it may be), my benchmarks show that a Node proxy could absolutely outperform nginx. I'm not sure what I'm missing but it seems node-http-proxy is way slower than it should be...

[manast]

@dtjohnson It does not seem that the http-proxy team has done any serious performance benchmarks, and that they have just assumed it is not possible to compete to other standardised solutions such as nginx or haproxy. I think benchmark should be a part of the development process of this module. It is paramount. Lets not give up in being faster than nginx until proved that it is not possible.

[dtjohnson]

Picking this up again after a while away. I did some more experimenting. I spent some more time being careful about upstream keep-alives in my proxy benchmarking suite. This time I ran the node-http-proxy with a keep-alive agent and the results were actually pretty good:

The results in light-green are comparable with the results of my simple Node.js proxy in dark green.

When running without keep-alives on any of the proxies, the results were not as good but still decent.

When enabling gzip the performance drops:

I'm guessing this is due to performance issues with the zlib module, but it's still not terrible.

The results are pretty encouraging. I wish the gzip performance was better, but I'm much more comfortable using node-http-proxy now.

[manast]

@dtjohnson thanks for the results. We could then conclude that node-http-proxy is as fast as what is currently possible with node. The dev team should use a test like this to always verify that the proxy has not been degraded in performance between releases, and that it always is kept at the same level as what plain nodejs can offer as maximum throughput.

[manast]

btw, what version of nodejs did you use? (this will be pretty relevant since improvements in the http module as well as on streams will have huge impact on the benchmarks)

[manast]

another remark, what about HTTPS support? As internet is moving, HTTPS support is almost the standard now, so any relevant benchmark should include it.

[dtjohnson]

Certainly seems to be about as fast as possible--with the keep-alive agent!

I used the latest node Docker image, which is v7.3. I suspect the Node version is going to be the biggest driver of performance too.

I didn't bother with HTTPS as I use an AWS ELB for SSL termination in my use case. I also didn't want to figure out how to configure SSL certs for Apache, Nginx, and HAProxy.

[manast]

Even if you use AWS for HTTPS, lets say that the performance drop is 10x, then the performance of nodejs is just 10% av the total, which makes it even more irrelevant compared to the other contenders... (I am somehow trying to reach to the conclusion that node proxy is just as good as any other proxy) :).

[acanimal]

Any comparison with Netlix Zuul proxy?

[dtjohnson]

Apologies for the late response. Things have been hectic.

@manast, fair point about SSL. I'll work on getting that benchmark in place. I just need to chase down all of the SSL configs for the various proxies.

@acanimal, nope, but I'm happy to add it if you want to give me a Docker container and config. Pull requests are welcome: https://github.com/dtjohnson/proxy-benchmark

[manast]

@dtjohnson a simple test would be to use AWS HTTPS for all the proxies, and compare results. If my theory is true there will almost no difference in performance between all of them...

[acanimal]

@dtjohnson There is an official docker image for zuul: https://hub.docker.com/r/netflixoss/zuul/

[dtjohnson]

@manast, is your theory that the latency introduced by the ELB would outweigh the proxy latencies? I'd have to test that but I would guess it would just add the same additional latency to each. ELBs are also a little tough to test because AWS will add/remove nodes as the load demands so it's a bit tough to control the test.

I went ahead and added SSL support to the various proxies so we can compare SSL directly. I'll kick off the benchmarking soon, but it will take some time to complete.

@acanimal, the docs on zuul are pretty light. Could you provide a sample config for the proxy that includes SSL and Gzip support? If zuul can read environmental vars for the upstream settings that would be great too.

[dtjohnson]

Sorry for the delay. I had a bug to work through. The results with SSL are here:
https://dtjohnson.github.io/proxy-benchmark/#?compression=false&keepAlive=true&ssl=true&field=transferBytesPerSec&connections=2

Across all of the proxies, including Node, there isn't much of in impact from SSL. So I think it's safe to say that Node is fine with HTTPS.

What I do find very concerning is gzip performance. Here is the proxy requests/sec WITHOUT gzip:

Node looks pretty good. (I'm also not sure what the drop at 10kB is all about, but it does seem real.)

Now, here is the requests/sec WITH gzip:

You can see the Node performance tanks. I've tried playing with the zlib setup without any luck. The dark green line is Node piping the request right into a zlib gzip stream transform, while the light green is node-http-proxy using the compression module. No real difference.

I'm afraid the poor gzip performance is a show stopper for me as it seems to consistently add 50 ms to each request. I'm going to look into options for offloading gzip. I really wish I knew what was going on with zlib though...

[manast]

I think that when using gzip, the messages should not be compressed unless of certain size, maybe at least 5Kb or 10Kb, that is because for small sizes the compression is not so good anyway. There seems to be also some setup cost that is independent of the size of the response as if zlib requires some expensive setup.

[manast]

you may also be interested in this module: https://www.npmjs.com/package/snappy-stream

[dtjohnson]

Correct. You wouldn't get any benefit from compression unless the content size is large. In fact, you would expect gzip to hurt performance for small stuff as you mention.

The compression module does, in fact, check the content length and won't gzip if it's too small. The problem is that the default transfer encoding in HTTP 1.1 is chunked. The total content length is not sent as a header. The upstream in my benchmarks is using chunked encoding so compression will always try. All of the other proxies are able to gzip without a big performance loss. There's something about zlib that has a huge penalty.

I'll look into snappy. Thanks for that.

[dtjohnson]

Being very unsatisfied with the Node gzip results, I tried to get to the bottom of why the latencies were so bad. I found that gzipped output from Node was consistently showing a +40ms latency with wrk. However, when I tried benchmarking this by using a Node script to call it over and over, I got a much lower latency. I also tried some other benchmarking tools; some showed the large latency, some didn't.

Clearly Node was doing something that some of these benchmark tools like wrk didn't like. So I used tcpdump to see the raw TCP packets being sent. When sending a small message, the upstream server was packing the HTTP headers and the body into a single TCP packet. When HAProxy, for example, received this, it gzipped the body and sent the response on to the client again in a single TCP packet. The Node proxy handled things a little differently. Once Node receives the response it pipes it into the gzip transform and then into the response stream. What seems to be happening is that once the stream is piped, Node sends the HTTP headers in a packet by themselves. Then a data event fires from the upstream request stream. Gzip compresses the packet and sends it along as a second packet. Then the request stream finishes, which causes the gzip stream to send whatever closing bytes it needs to. So while HAProxy sends a single packet, Node sends 3. I'm not sure it's possible to change this behavior because it's fundamental to how streams behave, but I'm also not sure it's a problem.

This told me that wrk must not be handling the multiple packets correctly. I think the latencies are getting inflated because of the order in which wrk processes the packets coming from the numerous outstanding requests. That led me to wrk2, which is a variation of wrk focused on accurate latency times. It also benchmarks in a different way. It focuses on what the response latency is under a fixed request load, which seems to me to be a better way to benchmark the proxies anyway.

I ran some initial tests with wrk2 and found that at lower request rates the performance of Node is on par with the other proxies (all around 4-7ms latency). The performance tends to degrade more quickly than the others as the request rates go up, which makes sense given the higher CPU usage. I just started the full benchmark suite of tests (SSL, gzip, upstream keep-alive, various connections, message lengths, and now request rates). The full suite takes more than 24 hours at this point so it'll be a bit before I have the results, but I'm very optimistic!

[manast]

@adjohnson916 really good job!, I wonder if somebody else has made such a comprehensive benchmark on node just yet? Maybe even the core developers of node do not really know where we are in terms of performance. It seems also that some people, authors of node-http-proxy included, have given up on competing with other servers because they believe V8 is not capable of deliver as efficiently as a C/Asm optimized solution, which is a pity.

[dtjohnson]

The results are in:
https://dtjohnson.github.io/proxy-benchmark/

First off, all proxies perform much better if keep alive connections are kept to the upstream. However, node-http-proxy seems to suffer especially poorly in this regard:

For the rest of the results below, upstream keep alives are turned on.

Without gzip and at low request rates and connections, the latencies of all of the proxies very close together (4-6 ms). HAProxy and Apache tend to lead the pack. Node and Nginx perform similarly. I also don't see any real difference between node-http-proxy and the Node proxy without dependencies.

SSL doesn't seem to make a difference for any of the proxies. At low rates you don't notice. At high rates it just ticks up slightly. (No point sharing a graph.)

If we increase the request rate to 1000 req/s, all of the proxies struggle to keep up, but the trend is the same:

The raw Node proxy edges out node-http-proxy and both beat Nginx.

The number of connections (from 2 to 100) doesn't seem to make a huge difference in the relative behavior of the proxies either. However, at high request rates and 100 connections, Nginx suffers:

Now gzip. There are some fluctuations as you look across the various connection and request rate options, but the results across the proxies seem pretty close. Here is 20 requests/sec with 10 connections:

At 100 connections and 1000 req/s:

So here's my summary:

• The latencies across all of the proxies are quite close together. They should all do a good job serving as a proxy without a noticeable difference between them.
• In general, HAProxy and Apache are the best performing. Node and Nginx performance is similar.
• It does seem that, in general, node-http-proxy does not perform as well as it could. However, the performance gap is small (except when not using upstream keep alives).

[manast]

Why is gzip compression not affecting performance so badly as in previous tests? or you mean that it was wrk fault, with wrk2 works as expected?

[dtjohnson]

As I explained earlier, I think that was just an artifact of wrk and the way Node sends out more than one packet when gzipping. I switched to wrk2, which gives better latency measurements.

I'm thinking it might be interesting to swap the x-axis of the plots such that I'm sampling a large number of request rates instead of message lengths. Then we could see better at which request rates each proxy starts suffering....

[mikestead]

@dtjohnson @manast Thanks for spending so much time looking into this, really useful thread.

[ctessier]

Hi,

Thank you a lot @manast for this thread. I have been reading it because I am having some performance issue with node-http-proxy.
Setting the agent with keep alive seems to increase performance a lot but I have a question regarding endurance.

On my project, the team noticed performance issues thanks to a Gatling scenario that triggers about 10req/sec for 30 minutes. It is very little, and the proxy handles it well, except after approximately 2min where we start seeing a lot of timeout requests.

• would you see any reason why the proxy would start struggling after 2 minutes only but do fine at first? or do I have to assume this particular behavior would be caused by the upstream service?
• what is the reason behind this performance issue? The proxy constantly has to open and close new requests, but is there something specific that this impacts? Are we limited to what we can do with the protocole (open and closing requests is time consuming) or could it also be solved with more resource or Node optimization?

Thank you a lot!

[manast]

@ctessier I would suggest you that you isolate the problem in a super simple server and see if you can reproduce the timeouts. If that is the case it may be an issue with http-proxy, if it works well then the issue is elsewhere.

[ctessier]

Thank you for your quick answer @manast I tried that, using Artillery, but I struggle to get consistent results as it seems that any use of my computer while the benchmark is running affects the results. Any advise maybe? I should find a way with remote machines maybe and try again with wrk.

I was also wondering if there is a way to mesure upstream response time with accuracy? I use proxyReq and proxyRes events to respectively start a timer and log the response time but since my app seems overwhelmed (with no agent and still a bit even with keep-alive agent), it seems that the times I get includes the time for my app to process all requests being queued or something.

Thank you for your help.

[manast]

@ctessier you should definitively use different machines, otherwise you do not really know what you are benchmarking, the the proxy? the server or benchmarker itself? :)

[ctessier]

Thank you very much @manast for your help. I was able to isolate the problem with a stub and I can now focus on other parts of the infrastructure, where the problem actually is.

About logging the upstream response time, do you have any advice maybe or shoud I open a new thread? I am afraid that starting the timer on proxyReq and stopping in on proxyRes can still include some delay caused by Node on big loads. Thank you very much.

[claytongulick]

Has anyone checked benchmarks against modern NodeJS versions? Streams have gotten a lot of love over the years, right? Have we seen the performance issues reduce?