LibRecommender
LibRecommender copied to clipboard
Limit RAM Usage of GPU/s
By default, tensorflow 1.14 alloctates whole RAM in GPU/s even though they are not needed in run-time. For ranker task, in this library, if indexing lib that uses GPU (for instance through faiss-gpu) is to be deployed, memory problem like below is faced.
RuntimeError: Error in void faiss::gpu::allocMemorySpaceV(faiss::gpu::MemorySpace, void**, size_t) at gpu/utils/MemorySpace.cpp:26: Error: 'err == cudaSuccess' failed: failed to cudaMalloc 1610612736 bytes (error 2 out of memory)
I have added config.gpu_options.per_process_gpu_memory_fraction = 0.7
configuration for session creation at algorithm/base.py
. By doing this, RAM usage of GPU/s is explicitely limited during training and 0.3 of them are reserved for indexing. Of course, there are draw-backs, is there another path to go you think?
def _sess_config(self, tf_sess_config=None):
if not tf_sess_config:
# Session config based on:
# https://software.intel.com/content/www/us/en/develop/articles/tips-to-improve-performance-for-popular-deep-learning-frameworks-on-multi-core-cpus.html
tf_sess_config = {
"intra_op_parallelism_threads": 0,
"inter_op_parallelism_threads": 0,
"allow_soft_placement": True,
"device_count": {"CPU": self.cpu_num}
}
# os.environ["OMP_NUM_THREADS"] = f"{self.cpu_num}"
config = tf.ConfigProto(**tf_sess_config)
# limit GPU Usage
config.gpu_options.per_process_gpu_memory_fraction = 0.7
return tf.Session(config=config)
I think the best way is making tensorflow allocate GPU memory dynamically, but I don't know how to achieve this...