501 for data too large

[lknite]

Overview of the Issue

==> proxmox-iso.ubuntu-2404: https://releases.ubuntu.com/noble/ubuntu-24.04.1-live-server-amd64.iso?checksum=sha256%3Ae240e4b801f7bb68c20d1356b60968ad0c33a41d00d828e74ceb3364a0317be9 => downloaded_iso_path/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso
2024/10/06 03:19:43 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/06 03:19:43 Leaving retrieve loop for ISO
2024/10/06 03:19:43 [INFO] (telemetry) ending ubuntu-2404
==> Wait completed after 1 minute 35 seconds
2024/10/06 03:19:43 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/10/06 03:19:43 machine readable: proxmox-iso.ubuntu-2404,error []string{"501 for data too large"}
==> Builds finished but no artifacts were created.
==> proxmox-iso.ubuntu-2404: 501 for data too large
Build 'proxmox-iso.ubuntu-2404' errored after 1 minute 35 seconds: 501 for data too large
2024/10/06 03:19:43 [INFO] (telemetry) Finalizing.
==> Wait completed after 1 minute 35 seconds
==> Some builds didn't complete successfully and had errors:
--> proxmox-iso.ubuntu-2404: 501 for data too large
==> Builds finished but no artifacts were created.
2024/10/06 03:19:43 waiting for all plugin processes to complete...
2024/10/06 03:19:43 /home/imagebuilder/.packer.d/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/06 03:19:43 /home/imagebuilder/.packer.d/plugins/github.com/hashicorp/proxmox/packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64: plugin process exited
2024/10/06 03:19:43 /home/imagebuilder/.local/bin/packer: plugin process exited
2024/10/06 03:19:43 /home/imagebuilder/.packer.d/plugins/github.com/yaleuniversity/goss/packer-plugin-goss_v3.2.12_x5.0_linux_amd64: plugin process exited
2024/10/06 03:19:43 /home/imagebuilder/.packer.d/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/06 03:19:43 /home/imagebuilder/.local/bin/packer: plugin process exited
2024/10/06 03:19:43 /home/imagebuilder/.local/bin/packer: plugin process exited
make: *** [Makefile:593: build-proxmox-ubuntu-2404] Error 1
Cleaning up project directory and file based variables
00:03
ERROR: Job failed: command terminated with exit code 1

Reproduction Steps

1. git clone https://github.com/kubernetes-sigs/image-builder
2. start up 'cluster-node-image-builder-amd64:v0.1.36' and mount git repo from previous step
3. cd ../image-builder/images/capi; make build-proxmox-ubuntu-2204

Versions

Plugin and Packer version: 1.9.5 Proxmox: 8.2.7

Ideas

I'm running all this in a gitlab pipeline. Am I just running out of space? If you could help me to understand why the error is occurring & what needs to be done to get things working.

Troubleshooting

I also attempted the same from a linux vm and got the same error, so not something special to the pipeline.

Permissions

I've tried as an Administrator as well as creating a role ImageBuilder with the following:

Exec

$ cat go.sh 
#!/bin/bash

# configure via env vars
export PACKER_LOG=1
export PROXMOX_BRIDGE=vmbr0
export PROXMOX_ISO_POOL=tower
export PROXMOX_NODE=pve-c
export PROXMOX_STORAGE_POOL=cephfs
export PROXMOX_TOKEN=<redacted>
export PROXMOX_URL=https://10.0.0.21:8006/api2/json/
export PROXMOX_USERNAME=capmox@pve!capi

# update dependencies
cd images/capi
make deps-proxmox
cd ../..

# build images
cd images/capi
#make help
make build-proxmox-ubuntu-2204
cd ../..

[lknite]

If I add '/api2/json' then I get a different error related 'use of closed network connection':

2024/10/06 17:10:39 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/06 17:10:39 Leaving retrieve loop for ISO
==> proxmox-iso.ubuntu-2204: Post "https://10.0.0.21:8006/api2/json/nodes/pve-c/storage/tower/upload": write tcp 10.0.0.99:52362->10.0.0.21:8006: use of closed network connection
2024/10/06 17:10:39 [INFO] (telemetry) ending ubuntu-2204
Build 'proxmox-iso.ubuntu-2204' errored after 20 seconds 139 milliseconds: Post "https://10.0.0.21:8006/api2/json/nodes/pve-c/storage/tower/upload": write tcp 10.0.0.99:52362->10.0.0.21:8006: use of closed network connection

==> Wait completed after 20 seconds 139 milliseconds
==> Wait completed after 20 seconds 139 milliseconds

2024/10/06 17:10:39 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
==> Some builds didn't complete successfully and had errors:
2024/10/06 17:10:39 machine readable: proxmox-iso.ubuntu-2204,error []string{"Post \"https://10.0.0.21:8006/api2/json/nodes/pve-c/storage/tower/upload\": write tcp 10.0.0.99:52362->10.0.0.21:8006: use of closed network connection"}
==> Builds finished but no artifacts were created

[mpywell]

Hi @lknite the reproduction steps in the issue description should include steps we can use to reproduce or analyse the problem - there isn't a link included for the referenced 'image-builder' repo (perhaps this is a private repository we're not meant to have access to?)

Without being able to see a copy of the Packer configuration in use, it sounds like the configured Proxmox API token might be missing permissions to upload ISOs to the configured location, use of closed network connection is mentioned in #81

[lknite]

I'm currently away from my desk.

The above steps are working out of an image builder container, but I'm getting a similar result when running directly on a Linux vm. I'll add the non-container steps in morning. In the meantime I've updated the reproduction steps with the image-builder url.

The test is running with Administrator privileges, as mentioned in the original post, which I set after reading #81 before . I just re-read 81 and saw a log file I hadn't noticed before var/log/pveproxy/access.log, I'll check that in the morning and see if it has anything relevant.

[lknite]

I thought I would have something extra to add when testing via linux vm instead of the image-builder container image, but essentially everything is the same except I had to install packer and such. The script to test is the same as in the original posting.

I want to give you everything you need to test. However, since really there is just cloning the git repo and setting some environment variables I'm not sure what else to share. I'm not customizing anything, just using what comes with image-builder by default.

I think the files you are looking for are here: https://github.com/kubernetes-sigs/image-builder/tree/main/images/capi/packer/proxmox

As noted a few posts ago, the original 501 error was without '/api2/json/', after adding that now I see the 'use of closed network connection'.

Am available to run whatever troubleshooting tests you need. Getting this working is necessary in order to get clusterapi setup, which is needed.

[lknite]

Read this link again and checked this log file: /var/log/pveproxy/access.log .

And, there is a 403 in there: ::ffff:10.0.0.99 - - [09/10/2024:15:41:08 -0600] "POST /api2/json/nodes/pve-a/storage/tower/upload HTTP/1.1" 403 -

I'm trying to understand why... I created the user and api token ... and I've granted the user Administrator permission on the storage. Should I be granting the access in a different way?

[lknite]

Ok, figured it out, as noted in the previous image I was able to add the selected API Token and the Role 'Administrator' and the upload would fail. This is because though I'm allowed to grant this role here on the Storage, the user also needs to already have the role. This gives the experience of granting the permissions but still getting a 403.

The answer was to grant the user the needed permission:

pveum aclmod / -user capmox@pve!capi -role Administrator

I actually created a more limited role and granted that to the user and the API Token on the storage. But for those who end up here, just granting Administrator would probably be the fastest test.

Once I understood the granting access to an api token didn't work unless the original user also had the permission, learned this when looking into Privilege Separation, I found the answer here: https://github.com/hashicorp/packer-plugin-proxmox/issues/81

[lknite]

Perhaps there is an opportunity to make the process easier for folks in the future in the documentation, and perhaps in the error messages. Instead of 501 maybe "501, did you remember /api2/json?" Instead of "use of closed network connection" maybe "use of closed network connection (403)" or "use of closed network connection, see /var/log/pveproxy/access.log for more detail'.