amazon/ec2 diskSize might be too small for closure

[hlolli]

I'm experiencing strange issue with too small diskSize, I'm as far as I see, not using tmpfs under /run, it seems to be using /tmp from the logs. The memory is huge on this machine and plenty of disk space left. I'll add that this build started to break after nixos upgrade I did recently (I'm posting nix-info below).

      packages.x86_64-linux = {
        import-blocks = nixos-generators.nixosGenerate {
          inherit pkgs;
          modules = [
            ./base.nix
            ./import-blocks/module.nix
          ];
          format = "amazon";
        };
      };

nix -L build .#import-blocks

nixos-amazon-image> installation finished!
nixos-amazon-image> Model:  (file)
nixos-amazon-image> Disk /tmp/nix-build-nixos-amazon-image-21.11.20220422.6927292-x86_64-linux.drv-0/nixos.raw: 2147MB
nixos-amazon-image> Sector size (logical/physical): 512B/512B
nixos-amazon-image> Partition Table: gpt
nixos-amazon-image> Disk Flags:
nixos-amazon-image> Number  Start   End     Size    File system  Name     Flags
nixos-amazon-image>  1      1049kB  2097kB  1049kB               no-fs    bios_grub
nixos-amazon-image>  2      2097kB  2146MB  2144MB  ext4         primary
nixos-amazon-image> mke2fs 1.46.5 (30-Dec-2021)
nixos-amazon-image> Discarding device blocks: done
nixos-amazon-image> Creating filesystem with 523520 4k blocks and 131072 inodes
nixos-amazon-image> Filesystem UUID: e9fb8f2e-d4ac-4175-8d52-d8b1628d79d2
nixos-amazon-image> Superblock backups stored on blocks:
nixos-amazon-image>     32768, 98304, 163840, 229376, 294912
nixos-amazon-image> Allocating group tables: done
nixos-amazon-image> Writing inode tables: done
nixos-amazon-image> Creating journal (8192 blocks): done
nixos-amazon-image> Writing superblocks and filesystem accounting information: done
nixos-amazon-image> copying staging root to image...
nixos-amazon-image> [    0.000000] Linux version 5.3.0 (nixbld@) (gcc version 10.3.0 (GCC)) #1 Wed Apr 20 11:28:57 UTC 2022
nixos-amazon-image> [    0.000000] memblock address range: 0x7fffe9c00000 - 0x7fffeffff000
nixos-amazon-image> [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 25249
nixos-amazon-image> [    0.000000] Kernel command line: mem=100M virtio_mmio.device=316@0x1000000:1
nixos-amazon-image> [    0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
nixos-amazon-image> [    0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
nixos-amazon-image> [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
nixos-amazon-image> [    0.000000] Memory available: 100764k/102396k RAM
nixos-amazon-image> [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
nixos-amazon-image> [    0.000000] NR_IRQS: 4096
nixos-amazon-image> [    0.000000] lkl: irqs initialized
nixos-amazon-image> [    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
nixos-amazon-image> [    0.000000] lkl: time and timers initialized (irq2)
nixos-amazon-image> [    0.000003] pid_max: default: 4096 minimum: 301
nixos-amazon-image> [    0.000018] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
nixos-amazon-image> [    0.000022] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
nixos-amazon-image> [    0.003810] random: get_random_bytes called from _etext+0xb6d0/0x14187 with crng_init=0
nixos-amazon-image> [    0.003841] printk: console [lkl_console0] enabled
nixos-amazon-image> [    0.003855] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
nixos-amazon-image> [    0.003857] xor: automatically using best checksumming function   8regs
nixos-amazon-image> [    0.003946] NET: Registered protocol family 16
nixos-amazon-image> [    0.171525] raid6: int64x8  gen()  5419 MB/s
nixos-amazon-image> [    0.343227] raid6: int64x8  xor()  3369 MB/s
nixos-amazon-image> [    0.514822] raid6: int64x4  gen()  5950 MB/s
nixos-amazon-image> [    0.686473] raid6: int64x4  xor()  3818 MB/s
nixos-amazon-image> [    0.858218] raid6: int64x2  gen()  6907 MB/s
nixos-amazon-image> [    1.029994] raid6: int64x2  xor()  4220 MB/s
nixos-amazon-image> [    1.201782] raid6: int64x1  gen()  5534 MB/s
nixos-amazon-image> [    1.373549] raid6: int64x1  xor()  3474 MB/s
nixos-amazon-image> [    1.373551] raid6: using algorithm int64x2 gen() 6907 MB/s
nixos-amazon-image> [    1.373552] raid6: .... xor() 4220 MB/s, rmw enabled
nixos-amazon-image> [    1.373553] raid6: using intx1 recovery algorithm
nixos-amazon-image> [    1.373723] clocksource: Switched to clocksource lkl
nixos-amazon-image> [    1.373886] NET: Registered protocol family 2
nixos-amazon-image> [    1.374025] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
nixos-amazon-image> [    1.374043] TCP established hash table entries: 1024 (order: 1, 8192 bytes, linear)
nixos-amazon-image> [    1.374049] TCP bind hash table entries: 1024 (order: 1, 8192 bytes, linear)
nixos-amazon-image> [    1.374073] TCP: Hash tables configured (established 1024 bind 1024)
nixos-amazon-image> [    1.374215] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
nixos-amazon-image> [    1.374221] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
nixos-amazon-image> [    1.374330] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100013b, IRQ 1.
nixos-amazon-image> [    1.374669] workingset: timestamp_bits=62 max_order=16 bucket_order=0
nixos-amazon-image> [    1.375666] SGI XFS with ACLs, security attributes, no debug enabled
nixos-amazon-image> [    1.376310] io scheduler mq-deadline registered
nixos-amazon-image> [    1.376315] io scheduler kyber registered
nixos-amazon-image> [    1.376328] virtio-mmio virtio-mmio.0: Failed to enable 64-bit or 32-bit DMA.  Trying to continue, but this might not work.
nixos-amazon-image> [    1.379142] virtio_blk virtio0: [vda] 4194304 512-byte logical blocks (2.15 GB/2.00 GiB)
nixos-amazon-image> [    1.380068]  vda: vda1 vda2
nixos-amazon-image> [    1.380383] NET: Registered protocol family 10
nixos-amazon-image> [    1.380847] Segment Routing with IPv6
nixos-amazon-image> [    1.380858] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
nixos-amazon-image> [    1.381228] Btrfs loaded, crc32c=crc32c-generic
nixos-amazon-image> [    1.381341] Warning: unable to open an initial console.
nixos-amazon-image> [    1.381380] This architecture does not have kernel memory protection.
nixos-amazon-image> [    1.381382] Run /init as init process
nixos-amazon-image> [    1.382527] EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts:
nixos-amazon-image> [    1.405532] random: fast init done
nixos-amazon-image> [    2.342083] random: crng init done
nixos-amazon-image> error writing file: No space left on device
nixos-amazon-image> error processing entry /tmp/nix-build-nixos-amazon-image-21.11.20220422.6927292-x86_64-linux.drv-0/root/nix/store/.links/1h9lv5z1i56jvbbljs37kvyai8gn5r4iiyj8lav631r97m17yyzz, aborting
nixos-amazon-image> error processing entry /tmp/nix-build-nixos-amazon-image-21.11.20220422.6927292-x86_64-linux.drv-0/root/nix/store/.links, aborting
nixos-amazon-image> error processing entry /tmp/nix-build-nixos-amazon-image-21.11.20220422.6927292-x86_64-linux.drv-0/root/nix/store, aborting
nixos-amazon-image> error processing entry /tmp/nix-build-nixos-amazon-image-21.11.20220422.6927292-x86_64-linux.drv-0/root/nix, aborting
nixos-amazon-image> [   10.737677] reboot: Restarting system
nixos-amazon-image> ERROR: cptofs failed. diskSize might be too small for closure.
error: builder for '/nix/store/9n009xy2qvr34gmzpy3mm6gjqcwrfisd-nixos-amazon-image-21.11.20220422.6927292-x86_64-linux.drv' failed with exit code 1

# nix-shell -p nix-info --run "nix-info -m"
this path will be fetched (0.06 MiB download, 0.30 MiB unpacked):
  /nix/store/vkv057i6d4h2sw3ym8m76j193d54vi5d-bash-interactive-5.1-p16-dev
copying path '/nix/store/vkv057i6d4h2sw3ym8m76j193d54vi5d-bash-interactive-5.1-p16-dev' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.34, NixOS, 22.05 (Quokka), 22.05.20220419.97315af`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.5.0pre20211206_d1aaa7e`
 - channels(root): `"nixos-22.05pre369988.1ffba9f2f68, nixpkgs-22.05pre369988.1ffba9f2f68"`
 - nixpkgs: `/root/.nix-defexpr/channels/nixpkgs`

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.2G  4.0K  3.2G   1% /dev
tmpfs            32G     0   32G   0% /dev/shm
tmpfs            16G   11M   16G   1% /run
tmpfs            32G  432K   32G   1% /run/wrappers
/dev/dm-0       836G  114G  680G  15% /
tmpfs           6.3G   12K  6.3G   1% /run/user/0

[paulvictor]

I faced the same issue, solved it by adding a line

      modules =
        existingModules ++ [({...}: { amazonImage.sizeMB = 16 * 1024; })];

[askandera]

This was an issue for me as well, and the above worked. Any reasons why this isn't the default?

[yusefnapora]

I hit this & just wanted to mention that if you're using the nixos module, you can set the amazonImage.sizeMB attribute inside of formatConfigs.amazon, e.g.:

{ config, inputs, ... }:
{
  imports = [
    inputs.nixos-generators.nixosModules.all-formats
  ];

  formatConfigs.amazon = {config, ...}: {
    amazonImage.sizeMB = 16 * 1024;
  };
}

[mraiser]

I hit this & just wanted to mention that if you're using the nixos module, you can set the amazonImage.sizeMB attribute inside of formatConfigs.amazon, e.g.:

{ config, inputs, ... }:
{
  imports = [
    inputs.nixos-generators.nixosModules.all-formats
  ];

  formatConfigs.amazon = {config, ...}: {
    amazonImage.sizeMB = 16 * 1024;
  };
}

Which file did you add that to? None of this makes any sense.

I check out this repo, cd into it, execute "nix-shell" then enter: ./nixos-generate -f amazon -c /home/mraiser/Documents/aws/build_ec2_ami/ec2config.nix -I nixpkgs=channel:nixos-23.11

I've set TMPDIR environment variable to a drive with 3TB free. I've increased /run/user/1000 to 24GB. I've tried passing "--option" flag.

Always the same, "ERROR: cptofs failed. diskSize might be too small for closure."

Very. Frustrating.

[mraiser]

In case it is helpful to anyone in the future, here's a step-by-step on building an amazon ami, if you're getting the "ERROR: cptofs failed. diskSize might be too small for closure."

1. git clone https://github.com/nix-community/nixos-generators.git
2. edit file "formats/amazon.nix" to add line " amazonImage.sizeMB = 4096;" right before final "}"
3. cd nixos-generators
4. nix-shell
5. ./nixos-generate -f amazon -c /home/mraiser/Documents/aws/build_ec2_ami/ec2config.nix -I nixpkgs=channel:nixos-23.11

CHANGE: 4096 to number of MB you need to contain your image files CHANGE: /home/mraiser/Documents/aws/build_ec2_ami/ec2config.nix to weherever your config for the new image lives.

I hope it helps someone! Thanks to everyone in this thread for pointing me in the right direction!

[TimekillerTK]

In case someone is still having trouble figuring out how to use the workaround for flakes, taking the example from the README.md:

{
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    nixos-generators = {
      url = "github:nix-community/nixos-generators";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = { self, nixpkgs, nixos-generators, ... }: {
    packages.x86_64-linux = {
      amazon = nixos-generators.nixosGenerate {
        system = "x86_64-linux";
        modules = [
          # ./configuration.nix
          ({...}: { amazonImage.sizeMB = 16 * 1024; }) # <--------------------------- here
        ];
        format = "amazon";
      };
    };
  };
}