Whipers' medium model: Cannot allocate memory

[brunocaballero]

Hi,

I was experimenting with this project and tried to use the whisper's "medium" model.

It fails allocating memory on one of the rknn_create_mem after alllocating around 2GB of memory.

Did I hit some limit?

Do you think is there a way to support bigger models?

This is the output of strace: rknn_create_mem issues some DRM_IOCTL and the perform a mmap.

ioctl(5, _IOC(_IOC_READ|_IOC_WRITE, 0x64, 0x45, 0x20), 0x7fc9b97a90) = 0
ioctl(5, DRM_IOCTL_QXL_ALLOC, 0x7fc9b99e00) = 0
ioctl(5, DRM_IOCTL_VIRTGPU_EXECBUFFER, 0x7fc9b97e90) = 0
ioctl(5, DRM_IOCTL_GEM_FLINK, 0x7fc9b99180) = 0
ioctl(5, DRM_IOCTL_PRIME_HANDLE_TO_FD, 0x7fc9b997c0) = 0
ioctl(5, DRM_IOCTL_ETNAVIV_GEM_INFO or DRM_IOCTL_OMAP_GEM_NEW or DRM_IOCTL_PANFROST_MMAP_BO or DRM_IOCTL_V3D_MMAP_BO or DRM_IOCTL_VC4_CREATE_BO or DRM_IOCTL_VIRTGPU_GETPARAM, 0x7fc9b99ae0) = 0
mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0x1adabc000) = 0x7eeaa0f000
ioctl(5, _IOC(_IOC_READ|_IOC_WRITE, 0x64, 0x45, 0x20), 0x7fc9b97a90) = 0
ioctl(5, DRM_IOCTL_QXL_ALLOC, 0x7fc9b99e00) = 0
ioctl(5, DRM_IOCTL_VIRTGPU_EXECBUFFER, 0x7fc9b97e90) = -1 ENOMEM (Cannot allocate memory)

[happyme531]

You can try to use the latest RKNPU2 library and kernel driver, which supports switching IOMMU domain:

https://github.com/airockchip/rknn-toolkit2/blob/77b71094e08391c543d9c65fea5f7cf98cc16eee/rknpu2/runtime/Linux/librknn_api/include/rknn_matmul_api.h#L153

to bypass the memory limit.