vector
vector copied to clipboard
vectordotdev
enhancement(http sink): Retrying the HTTP sink in case of 404s and request timeouts
Adding the functionality to retry the log sending on getting a not found error(404), forbidden request (403) or a request timeout (408) from the HTTP sink. Since the defualt behavior is to drop the events during such events, this would affect the log propagation.
solves: #10870
Hi @noble-varghese, thank you for the contribution. I think retrying for these codes is reasonable, I will take a closer look tomorrow.
Note: Need to update the docs as well. See https://vector.dev/docs/reference/configuration/sinks/http/#retry-policy.
@pront Thanks for pointing it out ! I have updated the docs page and added unit tests on the http utils as well :)
To unblock this PR, perhaps we can add 408 to the defaults? Still a breaking change, but seems desirable and low risk. cc @jszwedko
I tried to push directly to the fork but I don't have write permissions.
I will push those changes. Thanks !
@pront Just curious to know, how do you run these tests on the local. It was taking me about 30-40mins for it to run. Is there a faster way ?
@pront Just curious to know, how do you run these tests on the local. It was taking me about 30-40mins for it to run. Is there a faster way ?
You could try cargo test --no-default-features --features sinks-http. This and other tips are mentioned here: https://github.com/vectordotdev/vector/blob/master/docs/DEVELOPING.md#tips-and-tricks
@pront Just curious to know, how do you run these tests on the local. It was taking me about 30-40mins for it to run. Is there a faster way ?
You could try
cargo test --no-default-features --features sinks-http. This and other tips are mentioned here: https://github.com/vectordotdev/vector/blob/master/docs/DEVELOPING.md#tips-and-tricks
Thanks Jesse, very neat.
Also, I always run the following before pushing to the remote branch:
cargo install -f --path vdev #cd to the repo root(only need this once)cargo vdev fmtcargo vdev check rust --clippy