Webhook provider support HTTP 304 not modified status code

[mannbiher]

Is your feature request related to a problem? Please describe. I am using Webhook provider to pull data from an internally developed config-management application. The application sends HTTP-304 status code when it verifies the ETag header is not changed. However, current webhook provider considers 304 status as error and continues retrying. The only workaround I have is to not send the ETag header and keep getting same data which is not changed. This puts unnecessary load on the our config-management application.

Describe the solution you'd like Change WebHook provider to respect 304 status code. When 304 status is received, provider should simply do nothing except updating the last sync metadata.

Describe alternatives you've considered The only alternative is to not send ETag data and receive full secret every-time.

Additional context NA

[Skarlso]

except updating the last sync metadata.

What would this be? If there was no need to update anything, nothing should have changed in the cluster objects, right?

[mannbiher]

What would this be? If there was no need to update anything, nothing should have changed in the cluster objects, right?

I have not extensively looked at the code, I thought external-secrets keeps a metadata so that it knows when to sync the secret again. If there is no such thing, its better.

[Skarlso]

I have not extensively looked at the code, I thought external-secrets keeps a metadata so that it knows when to sync the secret again. If there is no such thing, its better.

It's based on an interval that is defined in the object. So no such data is kept. :)

Okay, I'll implement the change with that in regard.

[Skarlso]

@mannbiher Which parts is the one putting strain on your management system? How often do you pull data from it? Wouldn't it be sufficient to decress the poll time?

[Skarlso]

Here's the thing. :)

The attached pull request will not work with other customer bases that use multiple data providers including the webhook since it cancels all operations.

It turns out it's really not easy to implement this pseudo caching logic for a single provider without re-writing all the logic for the rest of them since they are under a single interface umbrella.

This means we will probably not do this work here. It would be too much of a disruption for the codebase.

[mannbiher]

@mannbiher Which parts is the one putting strain on your management system? How often do you pull data from it? Wouldn't it be sufficient to decress the poll time?

Returning the whole data for every request. We currently poll for data every five minutes for every microservice. However as we are just migrating this functionality to external-secrets, I don't know how much additional load our config-management servers would have. We are good for now having to pull data every time.

Thank you for quickly taking a look and trying to fix it.

[Skarlso]

Yeah sorry. We tried brainstorming some ideas, but the disruption to the entire codebase would be just way too much right now. :/

[rickymulder]

Since you're using webhook and you need some custom functionality, would it be feasible to write a specific provider for this service?

[Skarlso]

I think I actually have a viable solution to this possibly.

[Skarlso]

@mannbiher Would this be still something that you would require?

[Skarlso]

This should do it honestly. I'll write some unit tests for it too, but this should generally be enough.