CMA initialisation fails with 64k page-size

[srcshelton]

Describe the bug

When booting the 6.1.73 kernel (from stable_20240124) on an 8GB CM4 with 4k page-size, the early boot messages read:

Reserved memory: created CMA memory pool at 0x000000002ae00000, size 64 MiB
OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool

… however, with a 64k page-size, the output is instead:

Reserved memory: incorrect alignment of CMA region
OF: reserved mem: node linux,cma compatible matching fail
...
cma: Failed to reserve 512 MiB

Does the default devicetree data mandate a particular CMA size/alignment which no longer holds with non-4k page sizes?

Steps to reproduce the behaviour

1. Rebuild kernel with CONFIG_ARM64_64K_PAGES=y;
2. Boot new kernel;
3. Observe boot messages regarding CMA.

Device (s)

Raspberry Pi CM4

System

$ sudo vcgencmd version
Mar 17 2023 10:50:56 
Copyright (c) 2012 Broadcom
version 82f3750a65fadae9a38077e3c2e217ad158c8d54 (clean) (release) (start_cd)

$ uname -a
Linux rpi-cm4-8-16-18 6.1.73_p20240124 #3 SMP Mon Mar 18 09:53:30 GMT 2024 aarch64 GNU/Linux

Logs

Booting Linux on physical CPU 0x0000000000 [0x410fd083]
Linux version 6.1.73_p20240124 (root@turnpike) (clang version 14.0.6, LLD 14.0.6) #3 SMP Mon Mar 18 09:53:30 GMT 2024
random: crng init done
Machine model: Raspberry Pi Compute Module 4 Rev 1.0
Reserved memory: incorrect alignment of CMA region
OF: reserved mem: node linux,cma compatible matching fail
  DMA      [mem 0x0000000000000000-0x000000003fffffff]
  DMA32    [mem 0x0000000040000000-0x00000000ffffffff]
  Normal   [mem 0x0000000100000000-0x00000001ffffffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x000000000000000-0x000000003e5fffff]
  node   0: [mem 0x0000000040000000-0x00000000fbffffff]
  node   0: [mem 0x0000000100000000-0x00000001ffffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
On node 0, zone DMA32: 416 pages in unavailable anges
On node 0, zone Normal: 1024 pages in unavailable ranges
cma: Failed to reserve 512 MiB
percpu: Embedded 2 pages/cpu s3768 r8192 d89112 u131072
pcpu-alloc: s33768 r8192 d89112 u131072 alloc=2*65536
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
Detected IPT I-cache on CPU0
CPU features: detected: Spectre-v2
CPU features: detected: Spectre-v4
CPU features: detected: Spectre-BHB
CPU features: detected: ARM errata 1165522, 1319367, or 1530923
alternatives: applying boot alternatives
Built 1 zonelists, obility grouping on.  Total pages: 129504
Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_headphones=0 bcm2708_fb.fbwidth=0 bcm2708_fb.fbheight=0 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 smsc95xx.macaddr=E4:5F:01:A8:3B:18 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000  console=tty0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 root=PARTUUID=95649c8f-4e71-4123-855d-e1c075e221c3 rootfstype=ext4 fsck.repair=yes rootwait cpufreq.default_governor=performance cgroup_enable=memory cgroup_memory=1 zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=20 zswap.zpool=z3fold
cgroup: Enabling memory control group subsstem
Unknown kernel command line parameters "kgdboc=ttyAMA0,115200 cgroup_memory=1", will be passed to user space.
Dentry cach hash table entries: 1048576 (order: 7, 8388608 bytes, linear)
Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes,linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
software IO TLB: area num 4.
software IO TLB: mapped [mem 0x00000003a600000-0x000000003e600000] (64MB)
Memory: 8194688K/8296448K available (8000K kernel code, 1376K rwdata, 2048K rodata, 188K init, 590K bss, 101760K reserved, 0K cma-reserved)

Additional context

No response

[pelwell]

The kernel defines the minimum CMA alignment as being the area addressable by the next level up page table (a huge page, if I've got the terminology correct). Because this is a page full of descriptors of pages, the size grows with the square of the page size. With a 4kB page this is 2MB, 32MB for a 16kB page, and 512MB for a 64kB page.

The kernel is correctly complaining that it can't place a 512MB block with 512MB alignment in a place which is reachable by the DMA channels.

[srcshelton]

Thanks - that makes sense. Is the limit on DMA addressable memory architectural/hardware-limited, or something configurable in the kernel setup which could be increased to allow for 512MB allocation? Can CMA ever work on RPi hardware with a 64k page-size?

If not, then is anything significant lost if CMA is simply disabled? My layman's understanding is that CMA is used to alter the memory allocation between VideoCore and ARM (although I'm also using the cut-down 16MB firmware, so this may be inactive in any case)?

[popcornmix]

If not, then is anything significant lost if CMA is simply disabled? My layman's understanding is that CMA is used to alter the memory allocation between VideoCore and ARM (although I'm also using the cut-down 16MB firmware, so this may be inactive in any case)?

You can't use hardware that requires contiguous buffers. On a Pi4 that would include h264/hevc decoders, isp, camera and any planes you want displayed.