sync_gateway Sync optimizations

Sync optimizations

Open snej opened this issue 1 year ago • 1 comments

CBG-2952

Improvements to the sync code in 'db' and to 'go-blip', to optimize replication by reducing the amount of memory and goroutines used.

Significant changes to concurrency in go-blip; see https://github.com/couchbase/go-blip/pull/62
Added blipRevSender which manages the fetching of doc revisions and sending of 'rev' messages. It throttles this to a limited rate so that the number being written to the socket and their total byte size don't exceed a limit.
Fixed some code that sent BLIP messages and blocked waiting for a reply; instead it uses the new Message.OnReply method to schedule a callback on the thread-pool.
Increased the maximum number of concurrent outgoing 'changes' messages from 2 to 4.
Made outgoing 'changes' messages have Urgent delivery so they arrive sooner.

[ ] Removed debug logging (fmt.Print, log.Print, ...)
[ ] Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
[ ] Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

[ ] GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/000/

May 10 '23 17:05 snej

Still awaiting performance testing by perf team: CBPS-1153. Review is blocked on this.

Jun 05 '23 16:06 snej