Marco Pracucci

> There is a memcached bug It's not really a bug, but storing an entry to the cache could just fail if memcached memory is full and it can't reclaim...

> My best theory, for now, is that the transfer gets cut off abruptly, and minio-go eats that error up as io.EOF From a quick look at Minio internals (very...

_Still valid, help wanted!_

From a quick look through the code, I've the feeling that we also suffer hash collisions in results cache too (see `cache.HashKey` usage).

Thanks @kamilhristov for the report. I'm seeing in the config you're using AWS ElastiCache: which instance type are you using for ElastiCache?

Could you also show the following metrics graphed? - `rate(thanos_memcached_operations_total[1m])` - `histogram_quantile(1, rate(thanos_memcached_operation_duration_seconds_bucket[1m]))` - `histogram_quantile(0.9, rate(thanos_memcached_operation_duration_seconds_bucket[1m]))` - `sum by (item_type) (rate(thanos_store_index_cache_hits_total[1m]) / rate(thanos_store_index_cache_requests_total[1m]))` Random thoughts: - We run memcached inside...

So, from your graphs I can see the 90th percentile latency is good, while the 100th is not. In my experience with AWS, performances on t2/t3 are unpredictable, so as...

1. Thanks for trying the m5 2. May you show the same charts but with irate() instead of rate()? 3. May you try to increase the max idle connections setting...

> `level=error ts=2020-01-10T20:46:07.001594337Z caller=memcached.go:126 msg="failed to cache series in memcached" err="the async buffer is full"` This can be easily fixed increasing `max_async_buffer_size`

> `level=warn ts=2020-01-10T20:46:09.910966063Z caller=memcached_client.go:277 msg="failed to fetch items from memcached" err="read tcp 172.17.0.4:52078->10.42.8.159:11211: i/o timeout"` I still have no clue about this. Do memcached metrics show anything catching your attention?...