Marco Pracucci
Marco Pracucci
Another hypothesis is that the set operations are saturating the fetches. Just for testing, may you try with the following settings? ``` max_async_concurrency: 10 max_async_buffer_size: 1000000 ```
I've also asked about the memcached instance type. May you also confirm Thanos store is not running on a t2/t3 instance, please? I don't think so, but just a confirmation.
I've run some benchmarks in our cluster, using two tools: 1. [`memtier_benchmark`](https://github.com/RedisLabs/memtier_benchmark) by RedisLabs 2. `bench`: a quick and dirty benchmark tool based on the Thanos `MemcachedClient` ([code here](https://github.com/thanos-io/thanos/compare/master...pracucci:memcached-client-benchmark)) Both...
Thanks for all your help. I will continue spend a bit more of time on this by myself, and I will get back to you in case I find out...
I talked offline to @kuberkaul and shared some tips. Parallelly, I'm investigating a potential issue causing unfair prioritisation within the memcached client due to the usage of the gate. I...
> I'm still running into timeout or "failed to wait for turn: context canceled" issues This log means that a query request has been cancelled. For example, if you open...
Adding an endpoint to show overrides would be useful too: https://github.com/cortexproject/cortex/issues/1324
_Still valid_
_Still valid_
Today we had another incident where this feature would have been very useful.