dvd

Marking this as a P2 for now, since "1m timeouts" are currently the most common autoretrieve error, and this is more of a data gathering task.

cc @kylehuntsman

cc @kylehuntsman

I did a quick check, and it looks like the median is about 2.5hrs with the lowest TTFB at 68m and the highest around 10hrs. So the numbers are consistently...

What is the actual "stress" being caused, too many queries, are the Count() queries taking too long, are the tables not indexed properly?

Assigning to @ischasny for now. @brendalee are all of these still relevant? Maybe we should have a specific issue on yugabyte instructions documentation?

@LexLuthr @dgreuel2002, any updates here, has this issue been resolved?

@hannahhoward can we add a task around metrics cleanup for http retrievals attempts for SPs to distinguish from other HTTP providers, like web3.storage?

Triaging this to the Boost team ( @jacobheun ) to prioritize in their backlog, since this may be causing the "1m timeout" errors being seen in the autoretrieve dashboard: https://protocollabs.grafana.net/d/lDh_Fko4k/autoretrieve-estuary?orgId=1&refresh=5s&from=1675121579495&to=1675294379495&viewPanel=39....

Marking this as a tech debt item in the Tornado backlog.