How to use GPU in a cloud computing environment?

[anderscui]

• What is your OS and architecture? Windows is not supported and Mac OS arm64 is not yet supported. For non-supported OS, you can use Docker (https://grobid.readthedocs.io/en/latest/Grobid-docker/)

I am using Docker (grobid/grobid:0.7.3). When I test on a local dedicated server, I can use --gpus all for docker run command, and it utilize the GPUs correctly.

But in a cloud computing env (some service like Amazon Lambda), it seems that I cannot easily set --gpus all, then even the machine has GPU, the container doesn't utilize it.

What can I do in this case?

• What is your Java version (java --version)?

• In case of build or run errors, please submit the error while running gradlew with --stacktrace and --info for better log traces (e.g. ./gradlew run --stacktrace --info) or attach the log file logs/grobid-service.log.

[kermitt2]

Hello @anderscui !

Grobid Docker image will normally work fine in a standard cloud computing environment with GPU set on the instance. Just be sure to have a proper NVIDIA GPU driver on the Linux host instance. I was able to run Grobid at scale in a few cloud environments with GPU without issue.

Do you have specific examples where it does not work, apart a Lambda env.?

For Amazon Lambda, I never tried it, but are you sure you can setup AWS Lambda with GPU? Maybe I misunderstand, but the idea with Lambda is that you just have a runtime environment and cannot control the infrastructure level?

[anderscui]

Hello @anderscui !

Grobid Docker image will normally work fine in a standard cloud computing environment with GPU set on the instance. Just be sure to have a proper NVIDIA GPU driver on the Linux host instance. I was able to run Grobid at scale in a few cloud environments with GPU without issue.

Do you have specific examples where it does not work, apart a Lambda env.?

For Amazon Lambda, I never tried it, but are you sure you can setup AWS Lambda with GPU? Maybe I misunderstand, but the idea with Lambda is that you just have a runtime environment and cannot control the infrastructure level?

My environment is not Lambda, it's hosted in a China cloud provider.

Maybe my environment is similar to this: when I deploy grobid using docker-compose, and don't add any configs specific to GPU, then grobid will only use CPU, although the host machine has an available GPU.

So in this case, how can I 'tell' grobid to use the GPUs?

PS: when I deploy a service (using PyTorch) in the save env, it will detect the GPUs automatically and use it. But for grobid, I think there should be some extra configs?