hydra icon indicating copy to clipboard operation
hydra copied to clipboard

Set `maxQueryExecutionTime` through an environment variable

Open thesan opened this issue 11 months ago • 4 comments

Address part of https://github.com/Joystream/joystream/issues/5088

IMO rate limiting (DDOS prevention...) should be taken cared of by specialized tools (e.g Fail2ban).

affects: @joystream/hydra-cli

Add optional timeout for gql request and db queries

thesan avatar Mar 04 '24 10:03 thesan

@kdembler I had made a mistake on the server config. Now I made a bunch of tests and even though according to the telemetry queries keep on going after the request timed out especially when the timeout is less than 300 ms, it looks like it's working. But the best would be to identify one of the slow queries thank to the telemetry and see what happens when the timeout is set.

thesan avatar Mar 05 '24 15:03 thesan

Also I set this.httpServer.setTimeout(requestTimeout); in @joystream/warthog directly and it had the exact same effect than using the middleware (i.e the flow is not interrupted immediately but the request does timeout).

thesan avatar Mar 05 '24 15:03 thesan

Hey @kdembler. You were right: I tested the express middleware with a partially processed mainnet, which made it clear that the flow wasn't stopping at all after the request timed out.

So I tried a similar approach but using a type-graphql global middleware and this time the flow does appear to be aborted. It did take a few 100s of ms after the timeout but with a query that would take ~800ms to execute (without timeout): the last event reported by the telemetry was at ~400ms (for a timeout of 200ms) so it seems to be working now.

thesan avatar Mar 08 '24 16:03 thesan

~~Actually it didn't work with the slow distributionBuckets query this one ran in 51s without timing out. So I'll look at it some more on Monday...~~

That was just due to a the last rename of GRAPHQL_SERVER_RESOLVER_TIMEOUT.

Still I'm seeing some strange results so I'll look some more on Monday.

thesan avatar Mar 08 '24 17:03 thesan