myvariant.info icon indicating copy to clipboard operation
myvariant.info copied to clipboard

Accelerate `SnpeffPostUpdateUploader`

Open erikyao opened this issue 3 years ago • 0 comments

The ideal implementation would involve a message queue:

  1. Any uploader extending SnpeffPostUpdateUploader will simply put the new IDs into the message queue in the post_update_data step, then return.
  2. SnpeffBaseUploader then takes over the responsibility to upload the SnpEff documents by looking for the new IDs in the message queue.

If the above implementation can not be done in time, at least use concurrency and/or parallelism in SnpeffPostUpdateUploader to upload the SnpEff documents. The slowness of a full release may be attributed to this SnpeffPostUpdateUploader. E.g. the wellderly uploading only takes 15min, while its post_update_data step takes 160min.

erikyao avatar Apr 29 '21 01:04 erikyao