DIGITS nvmlDeviceGetHandleByPciBusId() failed with error #2

abc@abc:~/digits$ ./digits-devserver

| _ / | | / | | |) | | ( || | | | _
|/_|| || |__/ 6.1.1

Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/abc/digits/digits/main.py", line 70, in main() File "/home/abc/digits/digits/main.py", line 55, in main import digits.webapp File "digits/webapp.py", line 73, in import digits.model.images.classification.views # noqa File "/usr/local/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in import result = _import(*args, **kwargs) File "digits/model/images/classification/views.py", line 12, in from .forms import ImageClassificationModelForm File "/usr/local/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in import result = _import(*args, **kwargs) File "digits/model/images/classification/forms.py", line 4, in from ..forms import ImageModelForm File "/usr/local/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in import result = _import(*args, **kwargs) File "digits/model/images/forms.py", line 6, in from ..forms import ModelForm File "/usr/local/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in import result = _import(*args, **kwargs) File "digits/model/forms.py", line 18, in class ModelForm(Form): File "digits/model/forms.py", line 334, in ModelForm ) for index in config_value('gpu_list').split(',') if index], File "digits/device_query.py", line 259, in get_nvml_info raise RuntimeError('nvmlDeviceGetHandleByPciBusId() failed with error #%s' % rc) RuntimeError: nvmlDeviceGetHandleByPciBusId() failed with error #2

abc@abc:~/digits$ ./digits/device_query.py Device #0:

CUDA attributes: name GeForce GTX 1080 Ti totalGlobalMem 11706630144 clockRate 1582000 major 6 minor 1 NVML attributes: Total memory 11164 MB Used memory 564 MB Memory utilization 1% GPU utilization 0% Temperature 30 C

Device #1:

CUDA attributes: name GeForce GTX 1080 Ti totalGlobalMem 11715084288 clockRate 1582000 major 6 minor 1 NVML attributes: Total memory 11172 MB Used memory 11 MB Memory utilization 0% GPU utilization 0% Temperature 31 C

Device #2:

CUDA attributes: name GeForce GTX 1080 Ti totalGlobalMem 11715084288 clockRate 1582000 major 6 minor 1 NVML attributes: Total memory 11172 MB Used memory 11 MB Memory utilization 0% GPU utilization 0% Temperature 31 C