vagga
vagga copied to clipboard
tailhook
execvp: No such file or directory
I've managed to fully run NodeOS on Docker, but vagga is giving me a strange execvp: No such file or directory error. Since NodeOS is musl-based instead of using glibc as system library, could it be that the source of the problem, by not being able to find the dynamic loader?
More lines of log would be helpful, especially if you run RUST_LOG=debug vagga
Vagga almost never uses execvp (I'm not sure, if there are any cases at all). Or you say that you can't run vagga itself?
It may happen if you build vagga with cargo directly instead of building vagga with vagga, so vagga is linked against libc (rather than being statically compiled).
The problem is not with vagga itself, in fact it starts and exec flawlessly the /init process, but the problem happens later. The image is mostly the same as the one used with Docker, so since it's doing a chroot() as first step I suspect maybe vagga is not doing it? It's the only thing I can think so far... If you have any suggestion about what I could test against for I would thank you it so much.
Anyway, the output of RUST_LOG=debug vagga run is:
INFO:vagga::process_util: Running "id" "-u" "-n"
INFO:vagga::process_util: Running "vagga_wrapper" "_build" "bootfs"
DEBUG:vagga::container::mount: Procfs mount "/home/piranna/Proyectos/NodeOS/.vagga/.mnt/proc"
DEBUG:vagga::container::mount: Pseusofs mount "/home/piranna/Proyectos/NodeOS/.vagga/.mnt/dev/pts" devpts newinstance
INFO:vagga::process_util: Running "vagga_version" "bootfs" "--settings" "{\"version_check\":true,\"proxy_env_vars\":true,\"ubuntu_mirror\":null,\"alpine_mirror\":null,\"uid_map\":null,\"push_image_script\":null,\"build_lock_wait\":false,\"auto_apply_sysctl\":false,\"environ\":{}}"
DEBUG:vagga::version::version: Versioning items: 3
DEBUG:vagga::version::version: Versioning setup: Step(Tar { url: "./out/latest/barebones.tar.gz", sha256: None, path: "/", subdir: "" })
DEBUG:vagga::version::version: Versioning setup: Step(Tar { url: "./out/latest/initramfs.tar.gz", sha256: None, path: "/", subdir: "" })
DEBUG:vagga::version::version: Versioning setup: Step(Tar { url: "./out/latest/usersfs.tar.gz", sha256: None, path: "/tmp", subdir: "" })
DEBUG:vagga::version: Got hash "e30b505619387cdc0a927dff68898c4d1dc069b13bbd86fb07d9a7d940f3cdde9b58ad457f6b1611a457e1885b4507653b64c0e1db8e99c6b40b94f28ed09b7e"
DEBUG:vagga::wrapper::build: Container path: "/vagga/base/.roots/bootfs.e30b5056" (force: false) true
DEBUG:vagga::wrapper::build: Path "/vagga/base/.roots/bootfs.e30b5056" is already built
INFO:vagga::process_util: Running "id" "-u" "-n"
INFO:vagga::process_util: Running "vagga_wrapper" "--root" "bootfs.e30b5056" "run"
DEBUG:vagga::container::mount: Procfs mount "/home/piranna/Proyectos/NodeOS/.vagga/.mnt/proc"
DEBUG:vagga::container::mount: Pseusofs mount "/home/piranna/Proyectos/NodeOS/.vagga/.mnt/dev/pts" devpts newinstance
DEBUG:vagga::container::mount: Pseusofs mount "/vagga/root/dev/pts" devpts newinstance
DEBUG:vagga::container::mount: Procfs mount "/vagga/root/proc"
INFO:vagga::process_util: Running "init"
mount procfs: Resource busy
mount devtmpfs: Operation not permitted
execvp: No such file or directory
/work: Resource busy
/sys/fs/fuse/connections: Invalid argument
/sys/kernel/debug: Invalid argument
/sys/firmware/efi/efivars: Invalid argument
/sys/fs/pstore: Invalid argument
/sys/fs/cgroup/pids: Invalid argument
/sys/fs/cgroup/freezer: Invalid argument
/sys/fs/cgroup/devices: Invalid argument
/sys/fs/cgroup/blkio: Invalid argument
/sys/fs/cgroup/cpu,cpuacct: Invalid argument
/sys/fs/cgroup/perf_event: Invalid argument
/sys/fs/cgroup/net_cls,net_prio: Invalid argument
/sys/fs/cgroup/cpuset: Invalid argument
/sys/fs/cgroup/memory: Invalid argument
/sys/fs/cgroup/hugetlb: Invalid argument
/sys/fs/cgroup/systemd: Invalid argument
/sys/fs/cgroup: Invalid argument
/sys/kernel/security: Invalid argument
/sys: Resource busy
/dev/mqueue: Invalid argument
/dev/hugepages: Invalid argument
/dev/shm: Invalid argument
/dev/pts: Invalid argument
/dev: Resource busy
/: Invalid argument
INFO:vagga::process_util: Running "id" "-u" "-n"
DEBUG:vagga::wrapper::clean: Removing "/home/piranna/Proyectos/NodeOS/.vagga/.transient/bootfs.e30b5056.28123"
The mount and unmount lines are normal and expected, they are mostly safe warnings, don't worry about them.
INFO:vagga::process_util: Running "init"
mount procfs: Resource busy
mount devtmpfs: Operation not permitted
execvp: No such file or directory
Looks strange, so it can't mount few filesystems but tries to start process anyway?
Do these mounts fail in docker too?
Actually, I suspect that the issue is barely that environment is clean. In docker all ENV directives during build are kept when your're running commands. But in vagga build-time environment (!Env) and run-time environment (environ: {..}) are separate. Try to set PATH (or maybe all the variables) as in docker.
Looks strange, so it can't mount few filesystems but tries to start process anyway?
Yes, because in Docker and vagga the filesystems would be already available. In other cases, the boot process would fail soon anyway too.
Do these mounts fail in docker too?
procfs fails because it's already available on its place, but devtmpfs success:
mount procfs: Resource busy
Hello! I'm a user init script :-)
Welcome to NodeOS!: username: nodeos
· : password:
~ >
Actually, I suspect that the issue is barely that environment is clean. In docker all ENV directives during build are kept when your're running commands. But in vagga build-time environment (!Env) and run-time environment (environ: {..}) are separate. Try to set PATH (or maybe all the variables) as in docker.
I'm not paying too much attention during boot process to the environment variables, but I'll take a look on that. If it's of interest, the line dispatching the error is at https://github.com/piranna/nodeos-init/blob/ce02e341368d2bf603f8f73601d49925574fe969/init.c#L192
Sorry for the delay, I have been busy and also I was not able to correctly get the environment variables (my build system has several steps and also found a bug on one of them...). Finally I was able to get them:
root=container
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
They are somewhat expected, theroot=container is the way I have say "you are in container so there's no need to mount it", and the other ones are fairly common, so no problem here. What I've seen is that when using printf() once the execv error doesn't show and just only start the umount messages, but after being able to print the environment variables (several calls to printf) the /sbin/init command (already written in Node.js, and executed by /init with execvp) could run, so seems there's a race condition somewhere (probably mine, I'll investigate).
On the other hand, /sbin/init shows me that the environment variables are empty (process.env = {}), while they are being set on vagga.yaml and the /init process get them. That's the first time this happens to me (both in Docker, QEmu and real hardware environment variables are preserved and inherited by the child processes, as usual), so maybe Vagga is doing something here?
Just run everything under strace. You'll see execve commands with all correct paths and environment variables. Do you know how to run vagga under strace?
Ok, don't pay attention to the last paragraph, seems that since I was moving the environ pointer to read it I got up to an empty list when it was time to exec the /sbin/init process.
Don't know why, but now I don't have the execvp problem and I'm able to normally exec /sbin/init, and by using a container I'm able to access the users folders. Now the problem I have is that this container is mounted as read-only (/etc/mtab says /dev/sda3 /tmp ext4 ro,relatime,data=ordered 0 0). My current vagga.yaml file is:
containers:
bootfs:
auto-clean: true
hosts-file-path: null
resolv-conf-path: null
setup:
- !Tar
url: ./out/latest/barebones.tar.gz
- !Tar
url: ./out/latest/initramfs.tar.gz
# - !Tar
# url: ./out/latest/usersfs.tar.gz
# path: /tmp
# volumes:
# /tmp: !Persistent usersfs
usersfs:
data-dirs: [/nodeos, /root]
setup:
- !Tar
url: ./out/latest/usersfs.tar.gz
commands:
run: !Command
description: Start NodeOS
container: bootfs
volumes:
/tmp: !Container usersfs
# volumes:
# /tmp: !Persistent
# name: usersfs
environ:
root: 'container'
# PWD: /
# root: ''
# pid1mode: exec
run: [/init]
write-mode: transient-hard-link-copy
Ah, yes !Container volumes are always read-only, regardless of transient-hard-link-copy
You're probably want !Persistent volume seeded from a container, if you want writable system there.
Or a !Snapshot volume if you need temporary writable system (but be careful, it's tmpfs)
Ok, I have tried so far with !Container and !Persistent in several ways without luck. I could use !Snapshot, but then the data is not persistent. I have tried to initialize the volume using a minimal container like I was proposed at https://github.com/tailhook/vagga/issues/367#issuecomment-263088528 and worked for me, but feels like using a cannon to kill a fly because I just only need to use a tarfile to fill the volume, that's why I find more simple my proposal of allowing to use tarfiles on the !Persistent.seed attribute.
Anyway, I've been able to use it and both fuse and overlayfs works, so the port is almost finished :-D But I've found another problem that's not happening on Docker: the /init child processes works without problem, but the grandchilds (the child of the child processes) don't seem to be executed. They are being executed as the root process of a new processes group, but this happens also when changing it to be on the same one. Could there be some security limitation imposed by vagga because they are executed inside a chroot? You have the code that exec them available at https://github.com/piranna/jocker/blob/master/lib/chrootSpawn, putting a trace there gets executed, but the process executed by spawn() don't print anything on the console also when it's the only thing they do (in Docker it works flawlessly) and also don't write to the filesystem, that's why I think they don't get executed at all, but the point is that there's no error at all... :-/
Could there be some security limitation imposed by vagga because they are executed inside a chroot?
There are some limitations of chroot'ed containers imposed by kernel. Usually you need to use pivot_root instead of chroot to get rid of them. But I don't remember them well enough. You can't create user namespaces inside chroot, but I don't think you're doing that.
In cany case, if node.js doesn't show you exact error, just use strace to see what happens after execve or why the the system call returns error.
I noticed that the CPU was very high, and investigating a bit found that the ExclFS filesystem was receiving a lot of requests for /dev/srandom and /dev/egd-pool, although they don't exists on the host system. Seems Node.js checks on loading for several entropy generators on /dev to find what of them are available, what I don't understand is that on Docker or real hardware it works correctly but on vagga it start an endless loop checking for them over and over... :-/ That's the reason why they don't print anything, because they can't be able to continue execution :-( It's strange, isn't it?
Sure, it's strange. Firstly, /dev is pretty functional in container. Maybe you don't mount --bind it in chroot? Secondly, it shouldn't start endless loop for that (it's bug in node.js in this case).
Maybe you don't mount --bind it in chroot?
I have it mounted, in fact it's bind'ed to an instance of ExclFS, that's showing the content of the real /dev and where I have seen it's getting the huge loop of requests.
Secondly, it shouldn't start endless loop for that (it's bug in node.js in this case).
I think it too, just wanted to know if you could have an idea what could be making it to fail, since it only happens on vagga :-/
BTW, there were some issues with translating uids, gids and pids in fuse, when using user namespaces. Here is the one of them https://github.com/gittup/tup/issues/184 Maybe there are more.
What does ExclFS do?
It's a FUSE filesystem I've created. It records the files on an underlying directory (/dev in this case) that have been requested for access and show only the ones that are being used by the user or by nobody else, giving in fact access to each file to a single user each time. Doing this on /dev allow to the users to have direct access to the devices without needing explicit permissions and at the same time nobody can steal data or change the devices that are being used by others.
BTW, there were some issues with translating uids, gids and pids in fuse, when using user namespaces. Here is the one of them gittup/tup#184 Maybe there are more.
Interesting... I say that ExclFS was receiving the correct PID and uid, but didn't suspect there could be problems doing the translation in the other way, supossed it was fully supported... A good candidate for this problem, how could I be able to check it?