UCT/CUDA_IPC: use rcache instead of pgtable to cache mappings

[Akshay-Venkatesh]

Why ?

Use native capabilities in rcache to limit the size of mappings that can be cached by cuda_ipc transport. For example, UCX_CUDA_IPC_RCACHE_MAX_REGIONS=10 UCX_CUDA_IPC_RCACHE_MAX_SIZE=1mb limits the maximum number of cachings to 10 and max size of mappings cached to 1MB.

TODO:

• PR still untested with applications
• PR untested with corner cases like max_regions = 0 or max_size = 0
• Add capability to set max ratio of device mem that can be used to cache remote mappings

[Akshay-Venkatesh]

cc @yosefe

[Akshay-Venkatesh]

@yosefe I removed NO_PFN_CHECK flag temporarily as it allows for LRU region eviction. Can you explain when PFN_CHECK needs to be added? Also, would it be possible to create an issue to address rcache eviction logic bug that we discussed offline?

[Akshay-Venkatesh]

Replaced by https://github.com/openucx/ucx/pull/9039