vak icon indicating copy to clipboard operation
vak copied to clipboard

BUG: Getting a warning about Nvidia Tensor Cores

Open yardencsGitHub opened this issue 2 years ago • 1 comments

Describe the bug When running with a GPU that has Tensor Cores I am getting the following warning:

You are using a CUDA device ('NVIDIA GeForce RTX 4090') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision

To Reproduce Simply run 'vak train' or 'vak predict' from the command line using the above GPU.

Desktop (please complete the following information):

  • Operating System: Windows
  • Version 1.0.0a2

yardencsGitHub avatar Sep 29 '23 14:09 yardencsGitHub

Thank you @yardencsGitHub.

We should consider adding an option for this to the config file and the ability to set it globally. I think maybe we'd want a separate table for torch options like this.

[vak.torch]
float_32_matmul_precision = 'medium'

NickleDave avatar Sep 30 '23 12:09 NickleDave