pyalign
pyalign copied to clipboard
efficient parallel list-list solver
In rapidfuzz there's a cdist function that computes a matrix of alignment scores between each pair of two collections in parallel.
Is there something similar in pyalign, too?
There's a so-called batch mode, though I need to remind myself how it works. It makes use of SIMD calls to speed up computing multiple alignments at the same time. This is the code at the end of the example notebook - it does not seem to work right now. I need to check.
Ok, so in order to utilise multiple cores, I therefore have to wrap the solver (batched or not) in some Pythonic multiprocessing paradigm, right? Since builtin multithreading
suffers from GIL and multiprocessing
restricts sharing data to pickle-able objects, is there any one you can recommend in particular? (Like builtin concurrent.futures, or Joblib, Ray, MPIRE?)