Idea: TCP Request pipelining support

[TysonAndre]

Currently, gomemcache

1. acquires a connection
2. sends a request
3. receives a response
4. Releases a connection back to the connection pool

https://github.com/bradfitz/gomemcache/blob/f7880d5bcff45511fd1679c4c42e611c349e954a/memcache/memcache.go#L290-L304

It's possible in the memcache protocol to send requests without waiting for the response (https://en.wikipedia.org/wiki/Protocol_pipelining) and this is done in various clients/proxies. This allows for lower resource usage (one connection instead of multiple connection, fewer syscalls, etc), and higher throughput when there is higher latency (e.g. memcached in a different datacenter)

Is this something you'd be interested in a PR for?

This is something I'd implemented in https://github.com/TysonAndre/golemproxy (a hackathon project to implement something easier to customize than twemproxy, e.g. for custom business logic to replicate commands or compare results. The source was combined to make the development cycle easier since I was the only one working on that, but the client remains as an independent component with a similar api. golemproxy worked in unit/integration tests)

golemproxy is a local memcache proxy written in Golang. Like twemproxy, it supports pipelining and shards requests to multiple servers.

(Another difference is that that repo accepts bytes instead of str, but that was because it was a proxy for clients that send/receive non-utf8 data, such as serialized data)

(The gomemcache project appeared inactive when I'd worked on that support, so I hadn't created a ticket then)

[bradfitz]

Yes! Totally! This has been bugging me for 5+ years and is the reason why most PRs around connection management in this repo don't interest me too much; the core use of connections in this package just are wrong to begin with (not pipelined)

[TysonAndre]

Okay, I should have time later this week to work on this

[dormando]

Looks like @TysonAndre 's later that week never happened :)

Since this issue's near the top I'll add some notes: in the memcached proxy we pipeline all storage commands (either old text or new meta). This works so long as you track where the request came from and you don't try to pipeline certain things (watch, stats, etc).

Example:

get foo\r\n
set foo 0 0 2\r\n
hi\r\n
mg foo v\r\n

... can all be sent down the same pipe and responses can be read in order. Even mix and matching of text and meta protocols works.

Caveats:

• Incompatible with noreply - in the proxy I strip noreply/quiet flags.
• If CLIENT_ERROR is seen the connection must be closed
• If SERVER_ERROR is seen the request may be retried, but the only way to know if the response matches the request is via managing a request stack.

If you're using pure meta protocol and no text protocol, pipelining gets a bit better:

mg foo v O1 q\r\n
mg bar v O2 q\r\n
ms 2 baz O3 q\r\n
hi\r\n
mn\r\n

In this example we pipeline three commands with the quiet flag and an opaque flag.

If responses are found: mg hits or ms errors, the O is reflected.

mn caps the pipeline set, and when MN\r\n is seen you know all prior commands have been processed.

Caveats: There's no meta-specific error (yet...), so CLIENT_ERROR and SERVER_ERROR cannot be matched with their requests. To handle errors you need to:

• Close if CLIENT_ERROR seen
• Process until MN is seen, ignoring errors
• report a generic error state or retry for any keys not seen

In practice it's not the worst thing and might even get improved this year.

Finally, you can do the same as above without quiet flags and just pipeline and read as fast as you want. The memcached server will opportunistically batch responses into fewer syscalls. IE: if the server wakes and reads multiple requests off the socket at once, it will send as many responses as it can in the same syscall back to the client.

This means even if you do nothing but shove commands down the pipe as they come in, the client will save on recv syscalls.

[dormando]

That all said having two connections and routing new requests to the least-depth connection gives a useful average performance boost in my own testing. So keeping maxidle and defaulting it to 2 might be just fine.