Ryan Barrett

I don't understand why the DID `Object` lookups aren't using memcache. The ndb contexts in all three services seem to have it configured right. Here are the top DIDs by...

Progress here! Managed to cut datastore lookups and queries both way down, 5-10x each.

Now [looking at optimizing log storage](https://console.cloud.google.com/monitoring/metrics-explorer;duration=P7D?pageState=%7B%22xyChart%22%3A%7B%22constantLines%22%3A%5B%5D%2C%22dataSets%22%3A%5B%7B%22plotType%22%3A%22LINE%22%2C%22targetAxis%22%3A%22Y1%22%2C%22timeSeriesFilter%22%3A%7B%22aggregations%22%3A%5B%7B%22crossSeriesReducer%22%3A%22REDUCE_NONE%22%2C%22groupByFields%22%3A%5B%5D%2C%22perSeriesAligner%22%3A%22ALIGN_MEAN%22%7D%5D%2C%22apiSource%22%3A%22DEFAULT_CLOUD%22%2C%22crossSeriesReducer%22%3A%22REDUCE_NONE%22%2C%22filter%22%3A%22metric.type%3D%5C%22logging.googleapis.com%2Fbilling%2Flog_bucket_bytes_ingested%5C%22+resource.type%3D%5C%22global%5C%22+metric.label.%5C%22log_bucket_id%5C%22%3D%5C%22_Default%5C%22+metric.label.%5C%22log_bucket_location%5C%22%3D%5C%22global%5C%22%22%2C%22groupByFields%22%3A%5B%5D%2C%22minAlignmentPeriod%22%3A%2260s%22%2C%22perSeriesAligner%22%3A%22ALIGN_MEAN%22%7D%7D%5D%2C%22options%22%3A%7B%22mode%22%3A%22COLOR%22%7D%2C%22y1Axis%22%3A%7B%22label%22%3A%22%22%2C%22scale%22%3A%22LINEAR%22%7D%7D%7D&project=bridgy-federated&invt=AbjE3Q&inv=1). We were doing 250G/mo a bit ago, we're now down to ~150G/mo or so. The main cost here is initial ingest. We get 50G/mo...

The other next cost to look at is CPU. router is currently on four cores, atproto-hub on one. We should be able to get them both down. Here's one place...

Next step for datastore reads: https://github.com/snarfed/arroba/issues/30

Results from the datastore reads optimization here have been disappointing. I cut them by ~3x, but spend on datastore read API calls only went down maybe 1/4-1/3, from $10-12/d to...

Another opportunity here could be reducing router and atproto-hub allocated CPU. Router CPU is down to mostly under two cores, atproto-hub to maybe half a core. Could either drop router...

^ The difficulty with merging router and atproto-hub is that we currently run four WSGI workers for router, and we want to run the atproto-hub threads on just one of...

Another idea here: stop storing and fetching transient activities (create, update, delete, undo, accept/reject, etc) in the datastore entirely. Just use memcache for them.

...but looking at our incoming activity types, that might not make a big difference. The bulk of incoming activities are likes, follows, and reposts, in that order. Measure first, then...