myvariant.info
myvariant.info copied to clipboard
Accelerate `SnpeffPostUpdateUploader`
The ideal implementation would involve a message queue:
- Any uploader extending
SnpeffPostUpdateUploader
will simply put the new IDs into the message queue in thepost_update_data
step, then return. -
SnpeffBaseUploader
then takes over the responsibility to upload the SnpEff documents by looking for the new IDs in the message queue.
If the above implementation can not be done in time, at least use concurrency and/or parallelism in SnpeffPostUpdateUploader
to upload the SnpEff documents. The slowness of a full release may be attributed to this SnpeffPostUpdateUploader
. E.g. the wellderly uploading only takes 15min, while its post_update_data
step takes 160min.