Disable HTTP keep-alives for the OPA rest client

[jasonsouza]

What is the underlying problem you're trying to solve?

Frequently pulling bundles, in certain cases, keeps a large number of connections open in OPA instances.

Describe the ideal solution

Identify under what scenario(s) are preventing such connections from closing.

Describe a "Good Enough" solution

Disabling HTTP keep-alives for the OPA rest-client can reduce the number of idle connections staying open for an extended period of time.

[jasonsouza]

cc @ashutosh-narkar @tsandall

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[ashutosh-narkar]

This change would involve updating the http transport by setting DisableKeepAlives: true.

[srenatus]

Alternatively, we'd actually reuse the connection, I suppose.

[Parsifal-M]

Hey @srenatus, @ashutosh-narkar

I seem to be hitting a wall when trying to see the before and after changes on this, it seems when I make a bundle and server it and then make a few requests to get the bundles, the connections go to:

➜  netstat -ant | grep :8080 | awk '{print $6}' | sort | uniq -c | sort -n
      1 LISTEN
      1 TIME_WAIT
      4 ESTABLISHED

This doesn't seem to change whether i try and get 5 bundles or 20, only the TIME_WAIT increases (I am making an assumption that the connection just automatically closes once the request is done?)

So I am not seeing much difference before and after adding:

DisableKeepAlives: true

I am not sure if I am testing this incorrectly or I am missing something in how to reproduce the issue...

Let me know your thoughts?

Thanks!

[ashutosh-narkar]

If would expect to see more open connections netstat -ant | grep 8000 | grep -i established | wc -l before vs after the change where OPA keeps fetching bundles from the server.

[Parsifal-M]

I will run a few more tests tonight and see, then report back :+1:

Edit because I did some testing, if I curl the bundles to download, or just visit the endpoint and download bundles.

➜  ~ netstat -ant | grep :8080 | grep -i established | wc -l
0
➜  ~ netstat -ant | grep :8080 | grep -i time_wait | wc -l             
24

It seems as soon as you download a bundle the connection immediately goes into TIME_WAIT not sure if this is because I am doing it on localhost or not.

Let me know your thoughts on this?

Thanks!

[Parsifal-M]

Hey @jasonsouza,

Was wondering as you said in the initial post that "in certain cases" do you know in which case you saw this happening? Maybe I can try to replicate it this way instead :+1:

Let me know?

Thanks!

[ashutosh-narkar]

It seems as soon as you download a bundle the connection immediately goes into TIME_WAIT not sure if this is because I am doing it on localhost or not.

This would be the expected behavior after the fix as the connection is now closed, correct?

[Parsifal-M]

Hey @ashutosh-narkar,

Indeed but I also noticed the same behavior before the fix. So I am not sure if its actually needed, that was the issue I was facing.

I am not sure what results you get now if you test this? Let me know!

Thanks!

[boranx]

one possible way might be tweaking the os' TCP configs to enable the reuse of TIME_WAIT sockets by adding the following to /etc/sysctl.conf

net.ipv4.tcp_tw_reuse = 1

If I'm not mistaken in linux, the duration is not adjustable and is defined as one minute: https://github.com/torvalds/linux/blob/master/include/net/tcp.h#L123-L124

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[aarushisoni]

Hi, I want to contribute to this issue.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.