nixos-generators raw-efi fails with 25.05

Changing nothing else but the version from 25.05 <-> 24.11 I successfully build every time with 24.11 and fail every time with 25.05.

nixos-disk-image>   Disk image size: 11534336000 bytes
nixos-disk-image> Model:  (file)
nixos-disk-image> Disk /build/nixos.raw: 11.5GB
nixos-disk-image> Sector size (logical/physical): 512B/512B
nixos-disk-image> Partition Table: gpt
nixos-disk-image> Disk Flags:
nixos-disk-image> 
nixos-disk-image> Number  Start   End     Size    File system  Name     Flags
nixos-disk-image>  1      8389kB  1075MB  1066MB  fat32        ESP      boot, esp
nixos-disk-image>  2      1075MB  11.5GB  10.5GB  ext4         primary
nixos-disk-image> 
nixos-disk-image> Warning: The kernel is still using the old partition table.
nixos-disk-image> The new table will be used at the next reboot or after you
nixos-disk-image> run partprobe(8) or kpartx(8)
nixos-disk-image> The operation has completed successfully.
nixos-disk-image> mke2fs 1.47.2 (1-Jan-2025)
nixos-disk-image> Discarding device blocks: done           
nixos-disk-image> Creating filesystem with 2553344 4k blocks and 638976 inodes
nixos-disk-image> Filesystem UUID: b959b86d-3470-4c22-a378-1210cd0b36d7
nixos-disk-image> Superblock backups stored on blocks:
nixos-disk-image>       32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
nixos-disk-image> 
nixos-disk-image> Allocating group tables: done 
nixos-disk-image> Writing inode tables: done 
nixos-disk-image> Creating journal (16384 blocks): done
nixos-disk-image> Writing superblocks and filesystem accounting information: done 
nixos-disk-image> 
nixos-disk-image> copying staging root to image...
nixos-disk-image> [    0.000000] Linux version 5.15.0 (nixbld@localhost) (gcc (GCC) 14.2.1 20250322, GNU ld (GNU Binutils) 2.44) #1 Tue May 13 20:27:01 UTC 2025
nixos-disk-image> [    0.000000] memblock address range: 0x7fffe9c00000 - 0x7ffff0000000
nixos-disk-image> [    0.000000] Zone ranges:
nixos-disk-image> [    0.000000]   Normal   [mem 0x00007fffe9c00000-0x00007fffefffffff]
nixos-disk-image> [    0.000000] Movable zone start for each node
nixos-disk-image> [    0.000000] Early memory node ranges
nixos-disk-image> [    0.000000]   node   0: [mem 0x00007fffe9c00000-0x00007fffefffffff]
nixos-disk-image> [    0.000000] Initmem setup node 0 [mem 0x00007fffe9c00000-0x00007fffefffffff]
nixos-disk-image> [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 25250
nixos-disk-image> [    0.000000] Kernel command line: mem=100M virtio_mmio.device=316@0x1000000:1
nixos-disk-image> [    0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
nixos-disk-image> [    0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
nixos-disk-image> [    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
nixos-disk-image> [    0.000000] Memory: 100768K/102400K available (5761K kernel code, 1103K rwdata, 1302K rodata, 18014398509476318K init, 267K bss, 1632K reserved, 0K cma-reserved)
nixos-disk-image> [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
nixos-disk-image> [    0.000000] NR_IRQS: 4096
nixos-disk-image> [    0.000000] lkl: irqs initialized
nixos-disk-image> [    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
nixos-disk-image> [    0.000000] lkl: time and timers initialized (irq2)
nixos-disk-image> [    0.000004] pid_max: default: 4096 minimum: 301
nixos-disk-image> [    0.000019] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
nixos-disk-image> [    0.000024] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
nixos-disk-image> [    0.003926] random: get_random_bytes called from _etext+0xebf2/0x180dd with crng_init=0
nixos-disk-image> [    0.004363] printk: console [lkl_console0] enabled
nixos-disk-image> [    0.004379] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
nixos-disk-image> [    0.004800] NET: Registered PF_NETLINK/PF_ROUTE protocol family
nixos-disk-image> [    0.004842] lkl_pci: probe of lkl_pci failed with error -1
nixos-disk-image> [    0.171763] raid6: int64x8  gen() 22232 MB/s
nixos-disk-image> [    0.343698] raid6: int64x8  xor() 12253 MB/s
nixos-disk-image> [    0.516315] raid6: int64x4  gen() 16242 MB/s
nixos-disk-image> [    0.688841] raid6: int64x4  xor()  9487 MB/s
nixos-disk-image> [    0.861288] raid6: int64x2  gen() 10448 MB/s
nixos-disk-image> [    1.033323] raid6: int64x2  xor()  5949 MB/s
nixos-disk-image> [    1.205648] raid6: int64x1  gen() 10234 MB/s
nixos-disk-image> [    1.377561] raid6: int64x1  xor()  5357 MB/s
nixos-disk-image> [    1.377569] raid6: using algorithm int64x8 gen() 22232 MB/s
nixos-disk-image> [    1.377571] raid6: .... xor() 12253 MB/s, rmw enabled
nixos-disk-image> [    1.377574] raid6: using intx1 recovery algorithm
nixos-disk-image> [    1.377629] vgaarb: loaded
nixos-disk-image> [    1.379665] clocksource: Switched to clocksource lkl
nixos-disk-image> [    1.379823] NET: Registered PF_INET protocol family
nixos-disk-image> [    1.380127] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
nixos-disk-image> [    1.380404] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
nixos-disk-image> [    1.380423] TCP established hash table entries: 1024 (order: 1, 8192 bytes, linear)
nixos-disk-image> [    1.380434] TCP bind hash table entries: 1024 (order: 1, 8192 bytes, linear)
nixos-disk-image> [    1.380445] TCP: Hash tables configured (established 1024 bind 1024)
nixos-disk-image> [    1.380517] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
nixos-disk-image> [    1.380529] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
nixos-disk-image> [    1.380608] PCI: CLS 0 bytes, default 32
nixos-disk-image> [    1.380656] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100013b, IRQ 1.
nixos-disk-image> [    1.381061] workingset: timestamp_bits=62 max_order=15 bucket_order=0
nixos-disk-image> [    1.381987] SGI XFS with ACLs, security attributes, no debug enabled
nixos-disk-image> [    1.382716] xor: automatically using best checksumming function   8regs
nixos-disk-image> [    1.382728] io scheduler mq-deadline registered
nixos-disk-image> [    1.382730] io scheduler kyber registered
nixos-disk-image> [    1.384577] virtio_blk virtio0: [vda] 22528000 512-byte logical blocks (11.5 GB/10.7 GiB)
nixos-disk-image> [    1.385059]  vda: vda1 vda2
nixos-disk-image> [    1.385447] NET: Registered PF_INET6 protocol family
nixos-disk-image> [    1.385968] Segment Routing with IPv6
nixos-disk-image> [    1.385975] In-situ OAM (IOAM) with IPv6
nixos-disk-image> [    1.386012] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
nixos-disk-image> [    1.386690] Btrfs loaded, crc32c=crc32c-generic, zoned=no, fsverity=no
nixos-disk-image> [    1.386819] Warning: unable to open an initial console.
nixos-disk-image> [    1.386823] This architecture does not have kernel memory protection.
nixos-disk-image> [    1.386827] Run /init as init process
nixos-disk-image> [    1.389607] random: fast init done
nixos-disk-image> [    1.395610] EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: . Quota mode: disabled.
nixos-disk-image> [    1.967777] random: crng init done
nixos-disk-image> [   19.809021] blk_update_request: I/O error, dev vda, sector 16983712 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809170] Buffer I/O error on dev vda2, logical block 1860564, lost async page write
nixos-disk-image> [   19.809180] blk_update_request: I/O error, dev vda, sector 16983720 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809186] Buffer I/O error on dev vda2, logical block 1860565, lost async page write
nixos-disk-image> [   19.809216] blk_update_request: I/O error, dev vda, sector 16983728 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809222] Buffer I/O error on dev vda2, logical block 1860566, lost async page write
nixos-disk-image> [   19.809232] blk_update_request: I/O error, dev vda, sector 16983736 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809235] Buffer I/O error on dev vda2, logical block 1860567, lost async page write
nixos-disk-image> [   19.809244] blk_update_request: I/O error, dev vda, sector 16983744 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809247] Buffer I/O error on dev vda2, logical block 1860568, lost async page write
nixos-disk-image> [   19.809256] blk_update_request: I/O error, dev vda, sector 16983752 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809258] Buffer I/O error on dev vda2, logical block 1860569, lost async page write
nixos-disk-image> [   19.809268] blk_update_request: I/O error, dev vda, sector 16983760 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809271] Buffer I/O error on dev vda2, logical block 1860570, lost async page write
nixos-disk-image> [   19.809297] blk_update_request: I/O error, dev vda, sector 16983768 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809304] Buffer I/O error on dev vda2, logical block 1860571, lost async page write
nixos-disk-image> [   19.809364] blk_update_request: I/O error, dev vda, sector 16983776 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809367] Buffer I/O error on dev vda2, logical block 1860572, lost async page write
nixos-disk-image> [   19.809370] blk_update_request: I/O error, dev vda, sector 16983784 op 0x1:(WRITE) flags 0x107000 phys_seg 1 prio class 0
nixos-disk-image> [   19.809373] Buffer I/O error on dev vda2, logical block 1860573, lost async page write
nixos-disk-image> [   19.815814] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm host1: Error while async write back metadata
nixos-disk-image> [   19.847726] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm kworker/u2:1: Error while async write back metadata
nixos-disk-image> [   19.847837] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322450 starting block 2129868)
nixos-disk-image> [   19.847859] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm kworker/u2:0: Error while async write back metadata
nixos-disk-image> [   19.847867] Buffer I/O error on device vda2, logical block 1867467
nixos-disk-image> [   19.848024] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322452 starting block 2129869)
nixos-disk-image> [   19.848072] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322454 starting block 2129870)
nixos-disk-image> [   19.848078] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322456 starting block 2129871)
nixos-disk-image> [   19.848082] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322459 starting block 2129872)
nixos-disk-image> [   19.848087] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322460 starting block 2129873)
nixos-disk-image> [   19.848091] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322461 starting block 2129874)
nixos-disk-image> [   19.848094] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322462 starting block 2129875)
nixos-disk-image> [   19.848098] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322463 starting block 2129876)
nixos-disk-image> [   19.848271] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 322464 starting block 2129877)
nixos-disk-image> [   19.848330] Buffer I/O error on device vda2, logical block 1867468
nixos-disk-image> [   19.848334] Buffer I/O error on device vda2, logical block 1867469
nixos-disk-image> [   19.848338] Buffer I/O error on device vda2, logical block 1867470
nixos-disk-image> [   19.848341] Buffer I/O error on device vda2, logical block 1867471
nixos-disk-image> [   19.848344] Buffer I/O error on device vda2, logical block 1867472
nixos-disk-image> [   19.848351] Buffer I/O error on device vda2, logical block 1867473
nixos-disk-image> [   19.848355] Buffer I/O error on device vda2, logical block 1867474
nixos-disk-image> [   19.848357] Buffer I/O error on device vda2, logical block 1867475
nixos-disk-image> [   19.848360] Buffer I/O error on device vda2, logical block 1867476
nixos-disk-image> [   20.080002] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm kworker/u2:0: Error while async write back metadata
nixos-disk-image> [   20.084220] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm kworker/u2:1: Error while async write back metadata
nixos-disk-image> [   20.084325] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm kworker/u2:0: Error while async write back metadata
nixos-disk-image> [   20.094174] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm host1: Error while async write back metadata
nixos-disk-image> [   20.124951] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm kworker/u2:1: Error while async write back metadata
nixos-disk-image> [   20.125063] EXT4-fs error (device vda2): ext4_check_bdev_write_error:215: comm kworker/u2:0: Error while async write back metadata
nixos-disk-image> [   20.332635] Aborting journal on device vda2-8.
nixos-disk-image> [   20.332671] EXT4-fs error (device vda2) in __ext4_new_inode:1085: Journal has aborted
nixos-disk-image> unable to open file /mnt/0000fe02///nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06/nixos/pkgs/by-name/mq/mqttmultimeter/package.nix for writing: Read-only file system
nixos-disk-image> error processing entry /build/root/nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06/nixos/pkgs/by-name/mq/mqttmultimeter/package.nix, aborting
nixos-disk-image> error processing entry /build/root/nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06/nixos/pkgs/by-name/mq/mqttmultimeter, aborting
nixos-disk-image> error processing entry /build/root/nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06/nixos/pkgs/by-name/mq, aborting
nixos-disk-image> error processing entry /build/root/nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06/nixos/pkgs/by-name, aborting
nixos-disk-image> error processing entry /build/root/nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06/nixos/pkgs, aborting
nixos-disk-image> error processing entry /build/root/nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06/nixos, aborting
nixos-disk-image> error processing entry /build/root/nix/store/4pmlpks4b2psmadlwrc8dq8bwlw0kw00-nixos-25.05.802746.7282cb574e06, aborting
nixos-disk-image> error processing entry /build/root/nix/store, aborting
nixos-disk-image> error processing entry /build/root/nix, aborting
nixos-disk-image> [   20.337178] EXT4-fs (vda2): Remounting filesystem read-only
nixos-disk-image> [   20.337187] EXT4-fs (vda2): ext4_writepages: jbd2_start: 9223372036854775807 pages, ino 40774; err -30
nixos-disk-image> [   20.357997] reboot: Restarting system
nixos-disk-image> ERROR: cptofs failed. diskSize might be too small for closure.

I have bumped the diskSize which didn't help so it appears to be something related to the ext4/fs package?

May 30 '25 19:05 travisghansen

For me it fails with 24.11 too. The raw-efi where building well until I started adding more apps and suddenly started failing to build because of this same space error.

Jul 09 '25 16:07 lazaroofarrill

I actually spent some time on this yesterday.

I also started getting errors with 24.11 so it could be related to number of apps. My image size should be around 11-12GB currently.

If I crank the disk size way up I get slightly different errors (oom-like errors). I moved my build to a laptop with 64GB of RAM (0 swap) and all issues went away. The odd thing is, on my primary machine with 32GB of physical memory I have an additional 32GB of swap and it's failing.

Jul 09 '25 16:07 travisghansen

#279

Jul 09 '25 16:07 travisghansen

Setting diskSize to "auto" and values from 10GiB to 50GiB builds the image successfully. But I am also getting OOM once diskSize is set to values >50GiB.

/nix/store/2l0dxg824mzqdga4k9njy7xxwldhvh45-vm-run: line 142: │ 1281 Aborted │ (core dumped) cptofs -p -P 2 -t ext4 -i $diskImage $root/* / │ > ERROR: cptofs failed. diskSize might be too small for closure.

Jul 11 '25 06:07 pipelight

The error origins from make-disk-image.nix on line 603.

echo "copying staging root to image..."
cptofs -p ${lib.optionalString (partitionTableType != "none") "-P ${rootPartition}"} \
       -t ${fsType} \
       -i $diskImage \
       $root${lib.optionalString onlyNixStore builtins.storeDir}/* / ||
  (echo >&2 "ERROR: cptofs failed. diskSize might be too small for closure."; exit 1)

Jul 11 '25 06:07 pipelight

For me the culprit was resizing the /tmp directory. I even had to mount it to an external disk when building an amazon ami and couldn't fit /tmp in ram anymore.

Jul 11 '25 08:07 lazaroofarrill

For me the culprit was resizing the /tmp directory. I even had to mount it to an external disk when building an amazon ami and couldn't fit /tmp in ram anymore.

This sounds promising/probable. How large does it need to be? The size of the disk (if not set to auto) or the size of the actual data?

This could also explain why the added ‘ram’ via swap has no impact on outcomes for me because I think /tmp for me is a certain percentage of physical memory on the machine. I will investigate.

Jul 11 '25 12:07 travisghansen

@lazaroofarrill indeed! I have moved away from tmpfs and just used a traditional folder for /tmp and the problem went away immediately. I think there are 2 obvious changes that could be made based on this discovery:

update the error message to indicate it could be hitting limits on /tmp
have some argument that can be passed to use an alternative path instead of /tmp so users can keep tmpfs for /tmp and avoid this issue

Jul 11 '25 19:07 travisghansen

@lazaroofarrill indeed! I have moved away from tmpfs and just used a traditional folder for /tmp and the problem went away immediately. I think there are 2 obvious changes that could be made based on this discovery:

update the error message to indicate it could be hitting limits on /tmp

have some argument that can be passed to use an alternative path instead of /tmp so users can keep tmpfs for /tmp and avoid this issue

Actually they did put it in the FAQ section at the end of the README.

Jul 11 '25 20:07 lazaroofarrill

Ah, well who reads faqs ;) too bad I missed that. The error message should be updated as well then as that’s the most likely exposure to the problem.

Thanks for the pointers! I will attempt setting TMPDIR etc and see how it goes.

Jul 11 '25 21:07 travisghansen