zfs icon indicating copy to clipboard operation
zfs copied to clipboard

Published 20 hours ago • verified publisher icon openzfs

unusual blocksizes reported by newfstatat() confuse GNU coreutils cp

Open akorn opened this issue 3 years ago • 4 comments
trafficstars

Linux 6.0.8
Distribution Name Debian
Distribution Version sid
Kernel Version 6.0.8
Architecture amd64
OpenZFS Version 2.1.6-3

Describe the problem you're observing

cp(1) runs out of memory trying to copy relatively small files around.

Describe how to reproduce the problem

Set a largish recordsize on the filesystem (mine is 4M). Place files on filesystem. Try to use cp -a to copy them around (e.g. install zfs-dkms, or try to update-initramfs).

The code at https://github.com/coreutils/coreutils/blob/master/src/copy.c#L1343 will try to compute the least common multiple of the input and output block sizes, then allocate a buffer of that size using mmap(). With unusual block sizes (when the file fits into a single zfs record), the least common multiple can be very large, causing the mmap() to return ENOMEM and cp to abort with memory exhausted.

Include any warning/errors/backtraces from the system logs

Example strace:

execve("/bin/cp", ["cp", "-a", "usr/src/zfs-2.1.6/configure", "tmp"], ["STY=2728", "TERM=screen.xterm-256color", "TERMCAP=<...>"..., "WINDOW=1", "SHELL=/bin/zsh", "LANG=en_US.UTF-8", "LC_ADDRESS=en_US.UTF-8", "LC_COLLATE=en_US.UTF-8", "LC_CTYPE=hu_HU.UTF-8", "LC_IDENTIFICATION=en_US.UTF-8", "LC_MEASUREMENT=hu_HU.UTF-8", "LC_MONETARY=hu_HU.UTF-8", "LC_NAME=en_US.UTF-8", "LC_NUMERIC=en_US.UTF-8", "LC_PAPER=en_US.UTF-8", "LC_TELEPHONE=en_US.UTF-8", "LC_TIME=C.UTF-8", "USER=root", "LOGNAME=root", "HOME=/root", "PATH=/root/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin", "XDG_SESSION_ID=1", "XDG_RUNTIME_DIR=/run/user/0", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus", "XDG_SESSION_TYPE=tty", "XDG_SESSION_CLASS=user", "MOTD_SHOWN=pam", "SHLVL=2", "PWD=/t", "OLDPWD=/", "HOSTNAME=grml-zfs", "LESSOPEN=|lesspipe %s", "READNULLCMD=less", "DISTCC_HOSTS=+zeroconf", "WRITE_ON_UDEV=yes", "EDITOR=vim", "PAGER=less", "MAIL=/var/mail/root", "LS_COLORS=<...>"..., "LESS_TERMCAP_mb=\33[01;31m", "LESS_TERMCAP_md=\33[01;31m", "LESS_TERMCAP_me=\33[0m", "LESS_TERMCAP_se=\33[0m", "LESS_TERMCAP_so=\33[01;44;33m", "LESS_TERMCAP_ue=\33[0m", "LESS_TERMCAP_us=\33[01;32m", "ALREADY_DID_CD_HOME=/root", "COLORTERM=yes", "_=/bin/strace", "LC_MESSAGES=en_US.UTF-8"]) = 0
brk(NULL)                               = 0x55a2ed88a000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff7220aa000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3</etc/ld.so.cache>
newfstatat(3</etc/ld.so.cache>, "", {st_dev=makedev(0, 0x1b), st_ino=4436, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=88, st_size=42518, st_atime=1668786993 /* 2022-11-18T16:56:33.321671387+0100 */, st_atime_nsec=321671387, st_mtime=1668786993 /* 2022-11-18T16:56:33.321671387+0100 */, st_mtime_nsec=321671387, st_ctime=1668786993 /* 2022-11-18T16:56:33.321671387+0100 */, st_ctime_nsec=321671387}, AT_EMPTY_PATH) = 0
mmap(NULL, 42518, PROT_READ, MAP_PRIVATE, 3</etc/ld.so.cache>, 0) = 0x7ff72209f000
close(3</etc/ld.so.cache>)              = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3</usr/lib/x86_64-linux-gnu/libselinux.so.1>
read(3</usr/lib/x86_64-linux-gnu/libselinux.so.1>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\250\241\2\0\0\0\0\0\0\0\0\0@\08\0\n\0@\0\35\0\34\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0f\0\0\0\0\0\0\0f\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0p\0\0\0\0\0\0\0p\0\0\0\0\0\0\0p\0\0\0\0\0\0\205\247\1\0\0\0\0\0\205\247\1\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0 \2\0\0\0\0\0\0 \2\0\0\0\0\0"..., 832) = 832
newfstatat(3</usr/lib/x86_64-linux-gnu/libselinux.so.1>, "", {st_dev=makedev(0, 0x1c), st_ino=26245, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=341, st_size=174312, st_atime=1668235227 /* 2022-11-12T07:40:27+0100 */, st_atime_nsec=0, st_mtime=1668235227 /* 2022-11-12T07:40:27+0100 */, st_mtime_nsec=0, st_ctime=1668235227 /* 2022-11-12T07:40:27+0100 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
mmap(NULL, 186064, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libselinux.so.1>, 0) = 0x7ff722071000
mmap(0x7ff722078000, 110592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libselinux.so.1>, 0x7000) = 0x7ff722078000
mmap(0x7ff722093000, 32768, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libselinux.so.1>, 0x22000) = 0x7ff722093000
mmap(0x7ff72209b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libselinux.so.1>, 0x29000) = 0x7ff72209b000
mmap(0x7ff72209d000, 5840, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff72209d000
close(3</usr/lib/x86_64-linux-gnu/libselinux.so.1>) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>
read(3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220#\0\0\0\0\0\0@\0\0\0\0\0\0\0000\221\0\0\0\0\0\0\0\0\0\0@\08\0\t\0@\0\32\0\31\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\260\27\0\0\0\0\0\0\260\27\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\325D\0\0\0\0\0\0\325D\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0p\0\0\0\0\0\0\0p\0\0\0\0\0\0"..., 832) = 832
newfstatat(3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>, "", {st_dev=makedev(0, 0x1c), st_ino=25385, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=76, st_size=38832, st_atime=1630247187 /* 2021-08-29T16:26:27+0200 */, st_atime_nsec=0, st_mtime=1630247187 /* 2021-08-29T16:26:27+0200 */, st_mtime_nsec=0, st_ctime=1630247187 /* 2021-08-29T16:26:27+0200 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
mmap(NULL, 41008, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>, 0) = 0x7ff722066000
mmap(0x7ff722068000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>, 0x2000) = 0x7ff722068000
mmap(0x7ff72206d000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>, 0x7000) = 0x7ff72206d000
mmap(0x7ff72206f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>, 0x8000) = 0x7ff72206f000
close(3</usr/lib/x86_64-linux-gnu/libacl.so.1.1.2301>) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>
read(3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\"\0\0\0\0\0\0@\0\0\0\0\0\0\0000a\0\0\0\0\0\0\0\0\0\0@\08\0\t\0@\0\32\0\31\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\320\21\0\0\0\0\0\0\320\21\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\245\"\0\0\0\0\0\0\245\"\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0P\0\0\0\0\0\0\0P\0\0\0\0\0\0"..., 832) = 832
newfstatat(3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>, "", {st_dev=makedev(0, 0x1c), st_ino=25425, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=52, st_size=26544, st_atime=1630235045 /* 2021-08-29T13:04:05+0200 */, st_atime_nsec=0, st_mtime=1630235045 /* 2021-08-29T13:04:05+0200 */, st_mtime_nsec=0, st_ctime=1630235045 /* 2021-08-29T13:04:05+0200 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
mmap(NULL, 28696, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>, 0) = 0x7ff72205e000
mmap(0x7ff722060000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>, 0x2000) = 0x7ff722060000
mmap(0x7ff722063000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>, 0x5000) = 0x7ff722063000
mmap(0x7ff722064000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>, 0x5000) = 0x7ff722064000
close(3</usr/lib/x86_64-linux-gnu/libattr.so.1.1.2501>) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3</usr/lib/x86_64-linux-gnu/libc.so.6>
read(3</usr/lib/x86_64-linux-gnu/libc.so.6>, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0Ps\2\0\0\0\0\0@\0\0\0\0\0\0\0XD\35\0\0\0\0\0\0\0\0\0@\08\0\16\0@\0@\0?\0\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\20\3\0\0\0\0\0\0\20\3\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\220\n\32\0\0\0\0\0\220\n\32\0\0\0\0\0\220\n\32\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
pread64(3</usr/lib/x86_64-linux-gnu/libc.so.6>, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\20\3\0\0\0\0\0\0\20\3\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\220\n\32\0\0\0\0\0\220\n\32\0\0\0\0\0\220\n\32\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\08S\2\0\0\0\0\08S\2\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0`\2\0\0\0\0\0\0`\2\0\0\0\0\0\0`\2\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3</usr/lib/x86_64-linux-gnu/libc.so.6>, "", {st_dev=makedev(0, 0x1c), st_ino=25466, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=3755, st_size=1922136, st_atime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_atime_nsec=0, st_mtime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_mtime_nsec=0, st_ctime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
pread64(3</usr/lib/x86_64-linux-gnu/libc.so.6>, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\20\3\0\0\0\0\0\0\20\3\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\220\n\32\0\0\0\0\0\220\n\32\0\0\0\0\0\220\n\32\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\08S\2\0\0\0\0\08S\2\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0`\2\0\0\0\0\0\0`\2\0\0\0\0\0\0`\2\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1970000, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libc.so.6>, 0) = 0x7ff721e7d000
mmap(0x7ff721ea3000, 1396736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libc.so.6>, 0x26000) = 0x7ff721ea3000
mmap(0x7ff721ff8000, 339968, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libc.so.6>, 0x17b000) = 0x7ff721ff8000
mmap(0x7ff72204b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libc.so.6>, 0x1ce000) = 0x7ff72204b000
mmap(0x7ff722051000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff722051000
close(3</usr/lib/x86_64-linux-gnu/libc.so.6>) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>
read(3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\310\203\t\0\0\0\0\0\0\0\0\0@\08\0\t\0@\0\33\0\32\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0H\37\0\0\0\0\0\0H\37\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0u\257\6\0\0\0\0\0u\257\6\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\320\6\0\0\0\0\0\0\320\6\0\0\0\0\0"..., 832) = 832
newfstatat(3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>, "", {st_dev=makedev(0, 0x1c), st_ino=26146, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=1222, st_size=625288, st_atime=1666537638 /* 2022-10-23T17:07:18+0200 */, st_atime_nsec=0, st_mtime=1666537638 /* 2022-10-23T17:07:18+0200 */, st_mtime_nsec=0, st_ctime=1666537638 /* 2022-10-23T17:07:18+0200 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
mmap(NULL, 627592, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>, 0) = 0x7ff721de3000
mmap(0x7ff721de5000, 438272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>, 0x2000) = 0x7ff721de5000
mmap(0x7ff721e50000, 176128, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>, 0x6d000) = 0x7ff721e50000
mmap(0x7ff721e7b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>, 0x97000) = 0x7ff721e7b000
close(3</usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.0>) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff721de1000
arch_prctl(ARCH_SET_FS, 0x7ff721de2500) = 0
set_tid_address(0x7ff721de27d0)         = 139564
set_robust_list(0x7ff721de27e0, 24)     = 0
rseq(0x7ff721de2e20, 0x20, 0, 0x53053053) = 0
mprotect(0x7ff72204b000, 16384, PROT_READ) = 0
mprotect(0x7ff721e7b000, 4096, PROT_READ) = 0
mprotect(0x7ff722064000, 4096, PROT_READ) = 0
mprotect(0x7ff72206f000, 4096, PROT_READ) = 0
mprotect(0x7ff72209b000, 4096, PROT_READ) = 0
mprotect(0x55a2ec972000, 4096, PROT_READ) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff721ddf000
mprotect(0x7ff7220dc000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7ff72209f000, 42518)           = 0
statfs("/sys/fs/selinux", 0x7ffca928e590) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffca928e590)      = -1 ENOENT (No such file or directory)
getrandom("\xba\x20\x95\x16\x5f\x0e\x61\x47", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x55a2ed88a000
brk(0x55a2ed8ab000)                     = 0x55a2ed8ab000
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3</proc/filesystems>
newfstatat(3</proc/filesystems>, "", {st_dev=makedev(0, 0x14), st_ino=4026532069, st_mode=S_IFREG|0444, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1668600979 /* 2022-11-16T13:16:19.004057045+0100 */, st_atime_nsec=4057045, st_mtime=1668600979 /* 2022-11-16T13:16:19.004057045+0100 */, st_mtime_nsec=4057045, st_ctime=1668600979 /* 2022-11-16T13:16:19.004057045+0100 */, st_ctime_nsec=4057045}, AT_EMPTY_PATH) = 0
read(3</proc/filesystems>, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbdev\nnodev\tproc\nnodev\tcgroup\nnodev\tcgroup2\nnodev\tcpuset\nnodev\tdevtmpfs\nnodev\tdebugfs\nnodev\ttracefs\nnodev\tsecurityfs\nnodev\tsockfs\nnodev\tbpf\nnodev\tpipefs\nnodev\tramfs\nnodev\t"..., 1024) = 410
read(3</proc/filesystems>, "", 1024)    = 0
close(3</proc/filesystems>)             = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3</usr/lib/locale/locale-archive>
newfstatat(3</usr/lib/locale/locale-archive>, "", {st_dev=makedev(0, 0x1c), st_ino=9151, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=11737, st_size=6008912, st_atime=1668594547 /* 2022-11-16T11:29:07+0100 */, st_atime_nsec=0, st_mtime=1668594547 /* 2022-11-16T11:29:07+0100 */, st_mtime_nsec=0, st_ctime=1668594547 /* 2022-11-16T11:29:07+0100 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
mmap(NULL, 6008912, PROT_READ, MAP_PRIVATE, 3</usr/lib/locale/locale-archive>, 0) = 0x7ff721823000
close(3</usr/lib/locale/locale-archive>) = 0
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3</etc/locale.alias>
newfstatat(3</etc/locale.alias>, "", {st_dev=makedev(0, 0x1c), st_ino=1737, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=6, st_size=2996, st_atime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_atime_nsec=0, st_mtime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_mtime_nsec=0, st_ctime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
read(3</etc/locale.alias>, "# Locale name alias data base.\n# Copyright (C) 1996-2022 Free Software Foundation, Inc.\n#\n# This program is free software; you can redistribute it and/or modify\n# it under the terms of the GNU General"..., 1024) = 1024
read(3</etc/locale.alias>, "und for the time being for\n# backward compatibility.  Nobody should rely on the names defined here.\n# Locales should always be specified by their full name.\n\n# Note: This file used to contain the foll"..., 1024) = 1024
read(3</etc/locale.alias>, "-8859-1\ngalego\t\tgl_ES.ISO-8859-1\ngalician\tgl_ES.ISO-8859-1\ngerman\t\tde_DE.ISO-8859-1\ngreek           el_GR.ISO-8859-7\nhebrew          he_IL.ISO-8859-8\nhrvatski\thr_HR.ISO-8859-2\nhungarian       hu_HU.IS"..., 1024) = 948
read(3</etc/locale.alias>, "", 1024)    = 0
close(3</etc/locale.alias>)             = 0
openat(AT_FDCWD, "/usr/lib/locale/C.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/C.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = 3</usr/lib/locale/C.utf8/LC_TIME>
newfstatat(3</usr/lib/locale/C.utf8/LC_TIME>, "", {st_dev=makedev(0, 0x1c), st_ino=9150, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=7, st_size=3360, st_atime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_atime_nsec=0, st_mtime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_mtime_nsec=0, st_ctime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
mmap(NULL, 3360, PROT_READ, MAP_PRIVATE, 3</usr/lib/locale/C.utf8/LC_TIME>, 0) = 0x7ff7220a9000
close(3</usr/lib/locale/C.utf8/LC_TIME>) = 0
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3</usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache>
newfstatat(3</usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache>, "", {st_dev=makedev(0, 0x1c), st_ino=25070, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=53, st_size=27028, st_atime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_atime_nsec=0, st_mtime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_mtime_nsec=0, st_ctime=1668259385 /* 2022-11-12T14:23:05+0100 */, st_ctime_nsec=0}, AT_EMPTY_PATH) = 0
mmap(NULL, 27028, PROT_READ, MAP_SHARED, 3</usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache>, 0) = 0x7ff7220a2000
close(3</usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache>) = 0
futex(0x7ff722050a4c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
geteuid()                               = 0
openat(AT_FDCWD, "tmp", O_RDONLY|O_PATH|O_DIRECTORY) = 3</t/tmp>
newfstatat(AT_FDCWD, "usr/src/zfs-2.1.6/configure", {st_dev=makedev(0, 0x30), st_ino=29267, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=2647552, st_blocks=5177, st_size=2647046, st_atime=1668786119 /* 2022-11-18T16:41:59.760703544+0100 */, st_atime_nsec=760703544, st_mtime=1667705386 /* 2022-11-06T04:29:46+0100 */, st_mtime_nsec=0, st_ctime=1668785062 /* 2022-11-18T16:24:22.866737598+0100 */, st_ctime_nsec=866737598}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(3</t/tmp>, "configure", {st_dev=makedev(0, 0x32), st_ino=3838, st_mode=S_IFREG|0700, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4194304, st_blocks=1, st_size=0, st_atime=1668788205 /* 2022-11-18T17:16:45.416328293+0100 */, st_atime_nsec=416328293, st_mtime=1668788255 /* 2022-11-18T17:17:35.809758585+0100 */, st_mtime_nsec=809758585, st_ctime=1668788255 /* 2022-11-18T17:17:35.809758585+0100 */, st_ctime_nsec=809758585}, 0) = 0
openat(AT_FDCWD, "usr/src/zfs-2.1.6/configure", O_RDONLY|O_NOFOLLOW) = 4</t/usr/src/zfs-2.1.6/configure>
newfstatat(4</t/usr/src/zfs-2.1.6/configure>, "", {st_dev=makedev(0, 0x30), st_ino=29267, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=2647552, st_blocks=5177, st_size=2647046, st_atime=1668786119 /* 2022-11-18T16:41:59.760703544+0100 */, st_atime_nsec=760703544, st_mtime=1667705386 /* 2022-11-06T04:29:46+0100 */, st_mtime_nsec=0, st_ctime=1668785062 /* 2022-11-18T16:24:22.866737598+0100 */, st_ctime_nsec=866737598}, AT_EMPTY_PATH) = 0
openat(3</t/tmp>, "configure", O_WRONLY|O_TRUNC) = 5</t/tmp/configure>
ioctl(5</t/tmp/configure>, BTRFS_IOC_CLONE or FICLONE, 4) = -1 EXDEV (Invalid cross-device link)
newfstatat(5</t/tmp/configure>, "", {st_dev=makedev(0, 0x32), st_ino=3838, st_mode=S_IFREG|0700, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4194304, st_blocks=1, st_size=0, st_atime=1668788205 /* 2022-11-18T17:16:45.416328293+0100 */, st_atime_nsec=416328293, st_mtime=1668788266 /* 2022-11-18T17:17:46.326056957+0100 */, st_mtime_nsec=326056957, st_ctime=1668788266 /* 2022-11-18T17:17:46.326056957+0100 */, st_ctime_nsec=326056957}, AT_EMPTY_PATH) = 0
fadvise64(4</t/usr/src/zfs-2.1.6/configure>, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
copy_file_range(4</t/usr/src/zfs-2.1.6/configure>, NULL, 5</t/tmp/configure>, NULL, 9223372035781033984, 0) = -1 EXDEV (Invalid cross-device link)
mmap(NULL, 21688754176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0x55a7fa4b0000)                     = 0x55a2ed8ab000
mmap(NULL, 21689794560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2</dev/pts/3<char 136:3>>, "cp: ", 4) = 4
write(2</dev/pts/3<char 136:3>>, "memory exhausted", 16) = 16
write(2</dev/pts/3<char 136:3>>, "\n", 1) = 1
lseek(0</dev/pts/3<char 136:3>>, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0</dev/pts/3<char 136:3>>)        = 0
close(1</dev/pts/3<char 136:3>>)        = 0
close(2</dev/pts/3<char 136:3>>)        = 0
exit_group(1)                           = ?
+++ exited with 1 +++

stat(3) says of st_blksize "This field gives the "preferred" block size for efficient filesystem I/O." Perhaps this should be rounded up to the nearest power of two?

(I'll also file a bug against coreutils, because it really needs to be able to deal with weird block sizes.)

akorn avatar Nov 18 '22 18:11 akorn

Internally, ZFS has a variable recordsize on small files. This is being reported externally when perhaps it should not be. I believe this occurs because zfs_getattr_fast() is calling sa_object_size(), which is allowed to return these weird record sizes. What we should do is come up with a clean way to check dn->dn_maxblkid == 0 and when that is the case, we can return zfsvfs->z_max_blksz, which should fix this problem.

sa_object_size() is a wrapper for dmu_object_size_from_db(). Blindly changing that for newfstatat() would break things that expect the current behavior, such as space_map_open_impl(). I have a few thoughts on how to fix this, but I want to think about it some more.

ryao avatar Nov 19 '22 19:11 ryao

FWIW, a patch for coreutils that drops the LCM logic completely and limits buffer size to 32M has already been submitted, but it's probably still best to also do something about this in zfs, for the following reasons:

  1. all coreutils versions of the last few years have the LCM behaviour, so it's likely to be present on many systems;

  2. other tools may also be confused by unusual st_blksize values.

I'm not qualified to comment on what the best approach is; thanks for thinking about it!

akorn avatar Nov 19 '22 20:11 akorn

@akorn Would you provide a link to that patch?

That said, I am in favor of changing this. This report comes at a time when I am extremely busy working on some high priority bugs. However, I am mixing easier low priority tasks into my schedule to protect myself from burn out. I will probably tackle this as part of that in the next week or two.

ryao avatar Nov 19 '22 22:11 ryao

@ryao yes, sorry: it's in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59382, the specific patch is https://debbugs.gnu.org/cgi/bugreport.cgi?att=1;filename=copy-buffer-limit.patch;bug=59382;msg=8.

Update: meanwhile another, alternative fix was proposed for cp in the same thread.

akorn avatar Nov 19 '22 22:11 akorn

FWIW, coreutils applied their proposed in early January 2023, so that recent versions aren't affected by this problem anymore; but it may still make sense to do something about it within zfs as well.

akorn avatar Nov 12 '23 09:11 akorn

See also my prior report #12323

telsasoft avatar Nov 24 '23 19:11 telsasoft