HTTP connection pooling (using HTTP connection keep-alives)

[kostix]

Hi!

Here at my $dayjob, it appears that we've hit what appears to be a limitation of goreplay. I'm not sure it's really a limitation and not some misunderstanding on our part, so I'm reaching out for help.

The setup is as follows: we've recorded a dump of some 15 minutes of HTTP traffic using goreplay, which is saved in its native format. The traffic was created by multiple clients making requests against a single service. Hence, the dump was intended to be replayed against a single service as well.
Dry-running the dump using gor -input-file $filename -input-file-dry-run -output-null -output-http-stats shows sensbile number of requests and estimates the play time to be ca. 12m30s.
Dividing the number of requests by the estimated time to replay them shows that gor should perform the requests at roughly 2500 RPS.

Now we actually play the dump using gor -input-file $filename -output-http=$service_url -stats -output-http-stats and see that the actual RPS fluctuates between 650 and 950 — far below the estimated 2500; this can also be easily seen by observing that the replaying does not complete within the estimated time (and twice that time etc).

We've tried to debug the problem by replacing the service by a no-nonsense configured nginx instance told to return 200 OK with a small precanned response to any request, made sure it had sheer limits in terms of nofile rlimit and so on, but gor was still unable to go above 950 RPS or so.
Attempting to force it go faster by appending |1000% to the input file name made it reach ca. 1k RPS but not above.
There are no EMFILE errors (all processes have sheer limits set), and CPU/memory usage is low.

Using a build of the master's tip (to have this problem solved) and multiple -http-output= command-line parameters (targeting different IP addresses all listened on by the receiving nginx instance) does not change anything w.r.t. the sustained RPS.

So, we've recorded a TCP dump of the exchanges between gor and nginx and saw that goreplay handles every single request using the "connect(), send request, read response, close()" sequence, does not place Connection: keep-alive in its requests and ignores the same field sent by nginx in its responses.
In other words, it does not reuse TCP sessions it creates, and it looks like it's exactly the limiting factor.

I have read about the PRO version's capability of exactly replaying TCP sessions as they were captured but in our case, it's too advanced a feature to have; what we need is just the ability to use HTTP keep-alives to be able to maintain the RPS as recorded in the dump.

I fail to find any command-line option controlling this aspect of the goreplay behavior.

So, the question: does gorplay support connection pooling?
Or maybe we cannot see anything obvious to make it what we want some using some other approach?