D-SCRIPT icon indicating copy to clipboard operation
D-SCRIPT copied to clipboard

Add multi-GPU support for prediction (and embedding?)

Open samsledje opened this issue 4 years ago • 0 comments

A single process splits candidate interactions to the number of allocated GPUs, and all are able to access the same loaded embeddings in memory

  • currently can't run more than a certain number of different jobs in parallel, because each load all the embeddings into their memory, which fills up system memory quickly
  • for large numbers of predictions, running in serial on a single GPU is a waste of time and system resources
  • fix: cmd line allocates which GPUs are able to be used, and sub processes run on each one, after embeddings are loaded into a shared memory

samsledje avatar Feb 23 '21 20:02 samsledje