gossamer icon indicating copy to clipboard operation
gossamer copied to clipboard

feat(dot/sync): improve worker pool

Open haikoschol opened this issue 4 months ago • 3 comments

The main difference in the worker pool API is that SubmitBatch() does not block until the whole batch has been processed. Instead, it returns an ID which can be used to retrieve the current state of the batch. In addition, Results() returns a channel over which task results are sent as they become available.

The main improvement this brings is increased concurrency, since results can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface; getting a new batch of tasks with NextActions() and processing the results with Process().

Changes

  • replaced the code in dot/sync/worker_pool.go
  • adapted SyncService to the API changes of the new worker pool
  • adapted some expectations in tests regarding how often some mocks are called (hopefully without changing the logic being tested)

Tests

go test github.com/ChainSafe/gossamer/dot/sync

Issues

Closes #4232

haikoschol avatar Oct 15 '24 12:10 haikoschol