graval icon indicating copy to clipboard operation
graval copied to clipboard

Stops accepting connections after a time.

Open mclark4386 opened this issue 4 years ago • 2 comments

We have a VERY simple ftp server that is based on the MemDriver example. All the server is trying to do is accept PUTs from an authorized user and then putting the content of that file onto a RabbitMQ. It seems to do perfectly at first and then over time (and after a lot of people/bots trying to poke it) it will get to the point that it wont accept any connects and doesn't out put anything to the terminal until I hit a key and then all at once the terminal is filled with the log below

For context the server name is "customFTP to RabbitMQ":

<...>
2021/04/21 19:59:55 04ddd99e3e71e101dd15 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 eb4bea5f0671a6b9422e < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 a59ed5afe828bd1a7e35 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 c051f443cf41e1951c17 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 c97bbf855411d0e2519c < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 1ea95c9b919f3fbb4ea2 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 4d6ae8213285278b6d40 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 bdaf9bc67ebc4b2a2192 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 7b9bc71471b5473cded1 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 519f3e86acee326fdfd1 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 84880d8355473fd48b9d < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 c8d93c4d73b1c9dafefd < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 69d79e6b9f6afdc4dc4a < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 1b14fed69e3d235d29c2 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 97db1e07f44308a7774b   Connection Terminated
2021/04/21 19:59:55 7c764a2d542b9ca7718d   Connection Terminated
2021/04/21 19:59:55 f64d2a10de15d43124bf   Connection Terminated
2021/04/21 19:59:55 18bbe7ab30353b8eb572   Connection Terminated
2021/04/21 19:59:55 38d1c35e88c49e9e8f89   Connection Terminated
2021/04/21 19:59:55 6a609514292e5325fc8a   Connection Terminated
2021/04/21 19:59:55 7ea2e9b97f7534697307   Connection Terminated
2021/04/21 19:59:55 99a39e0cd4a95cdf9dc8   Connection Terminated
2021/04/21 19:59:55 410928f066cd0d6d8d47   Connection Terminated
2021/04/21 19:59:55 41188211fa8be2ce8777   Connection Terminated
2021/04/21 19:59:55 f6f3ae623ca6d77fd010   Connection Terminated
2021/04/21 19:59:55 3e3a75de4d35c19bff56   Connection Terminated
2021/04/21 19:59:55 25b7ed6e6b381fd4b315   Connection Terminated
2021/04/21 19:59:55 9146fdda835b3ef85e6f   Connection Terminated
<...>

Would love to hear any ideas! Happy to add more info on request, but I checked and we seem to be handling all the Close() calls for the various connections and the rest of the system seems stable, so I don't think it's a memory leak. For other context this is running on a windows server, not sure if that makes a difference! Thanks!

mclark4386 avatar Apr 21 '21 20:04 mclark4386

(made an empty issue by accident so filled it out, hopefully y'all still see it now!^^)

mclark4386 avatar Apr 21 '21 20:04 mclark4386

Bummer, sorry about that!

I'm actually not running graval in production any more myself, so my knowledge is a bit rusty.

It definitely sounds like a resource exhaustion issue, even if it's not memory. I wonder what other resources you could monitor to see if they're growing unreasonably.

What about running goroutines, or maybe open TCP sockets.

or other context this is running on a windows server, not sure if that makes a difference

Sadly I haven't really used windows for a long time, so I'm not sure if it makes a difference. It might, although everything in graval should be pretty OS agnostic :thinking:

yob avatar Apr 22 '21 14:04 yob