[MISC] Modify docker builds to make configurable base image

[blythed]

Needed for GPU/ CUDA compiled binaries.

• [ ] Release mechanisms also uploads 2 docker-images to open docker hub
• [ ] Use CUDA base image as base for superduperdb/cuda

[fnikolai]

Summary of tasks:

1. rename superduperdb/superduperdb to superduperdb/base
2. introduce superduperdb/cuda_base
3. Add an example of how users can use FROM superduperdb/base to build their custom application images.

[fnikolai]

For cuda, the main issue that cuda drivers and nvidia-container-runtime must be installed on the host.

https://developer.nvidia.com/cuda-downloads?target_os=Linux

To check if they are installed:

import GPUtil
import numpy as np

def main():
    # Get the list of available GPUs
    gpus = GPUtil.getAvailable(order='first', limit=1, maxLoad=0.5, maxMemory=0.5, includeNan=False, excludeID=[], excludeUUID=[])

    # Check if any GPUs are available
    if gpus:
        print("GPU is available")
        # Pick the first GPU
        gpu_id = gpus[0]
        # Set CUDA_VISIBLE_DEVICES environment variable
        import os
        os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu_id)

        # Example computation on GPU
        print("Running computation on GPU...")
        a = np.array([1.0, 2.0, 3.0])
        b = np.array([4.0, 5.0, 6.0])
        c = a + b
        print("Result of computation on GPU:")
        print(c)
    else:
        print("No GPU available, running computation on CPU instead")
        # Example computation on CPU
        a = np.array([1.0, 2.0, 3.0])
        b = np.array([4.0, 5.0, 6.0])
        c = a + b
        print("Result of computation on CPU:")
        print(c)

if __name__ == "__main__":
    main()

https://medium.com/@albertqueralto/enabling-cuda-capabilities-in-docker-containers-51a3566ad014

[fnikolai]

The trick is on --gpus=all. If this is enabled, even the basic nightly version works with gpu.

docker run --gpus=all -p 8888:8888 -ti --entrypoint /bin/sh superduperdb/nightly:dec04a88 returns

$ python test_gpu.py	
GPU is available
Running computation on GPU...
Result of computation on GPU:
[5. 7. 9.]

docker run -p 8888:8888 -ti --entrypoint /bin/sh superduperdb/nightly:dec04a88

returns

superduper@8d1c84475b8f:~/superduperdb$ python test_gpu.py 
No GPU available, running computation on CPU instead
Result of computation on CPU:
[5. 7. 9.]

[fnikolai]

According to this image https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/11.3.1/ubuntu2004/base/Dockerfile the only difference is a few environment variables that are set.

• Given that the enabling GPU supports relies solely on the host, that raises the question of whether we need to separate the CPU from GPU images *