Exceptions probably shouldn't close the writer

[billoneil]

Just hit an issue where the http writer got a timeout exception from our metrics relay which may have been backed up or had a hiccup. This line closed the http writer which made it so metrics could not report for the rest of the JVM processes lifetime.

Is there a particular reason the writers are closed here? That makes it so they can't recover. I think the writers should be able to handle their own failures and it seems like they already can. The HTTP writer will just get a new http connection and try again on the next pass. Once its closed it just fails forever.

[kyeah]

hey @billoneil, sorry for the delayed response — I missed this until now. The TcpWriter should recreate the connection after closing, and the HTTP writer should do the same (but currently doesn't).

I think the writers should be able to handle their own failures and it seems like they already can. The HTTP writer will just get a new http connection and try again on the next pass. Once its closed it just fails forever.

I wasn't aware of this behavior in the writers, but if so, then you're right — we can avoid manually closing the connection here.

[billoneil]

@kyeah I'll try to make some pull requests when I get around to it for you. Have been a bit busy