ventoy
About the BLOBs in Ventoy
In #2795 there are some discuss about the BLOBs in Ventoy.
For a long time, I devoted my limited spare time to adding new features and fixing bugs and didn't get around to considering this.
It should be noted firstly that Ventoy is 100% open souce and everything is transparent. Some binary files are directly get from other open source projects and Ventoy directly use them and don't need any change by default.
Now that someone cares about these BLOBs, so I think we can discuss about this and find a better way and use it step by step in Ventoy. This will definitely take a long time and need some help but I think it's good for Ventoy.
One approach I'm thinking of right now is that, for these BLOBs, I create some sub-repos in github and build them by github CI just once, and use the binary file in Ventoy. If we find some problem in these tools, we can update the source code and build again and use the new binary files in new Ventoy version.
What do you think about this?
For this, I want to confirm that
- Can github provide build environment for FreeBSD and Windows ?
- Can github provide build environment for Linux with glibc/musl ?
Update: 2025/05/14 Anyway, I think first step we need to get a full BLOBs list in current Ventoy source code. Refer to https://github.com/ventoy/Ventoy/issues/2795#issuecomment-2233938098 I create a list here. And also I create a file in the source tree https://github.com/ventoy/Ventoy/blob/master/BLOB_List.md to maintain the list. If you find some BLOBs which are not included in the list or some mistake, please let me known, thanks!
| BLOB | File Source | Desc |
|---|---|---|
| ./BUSYBOX/chmod/vtchmod32 | build | Build Instructions: ./BUSYBOX/chmod/build.sh |
| ./BUSYBOX/chmod/vtchmod64 | ||
| ./BUSYBOX/chmod/vtchmod64_musl | ||
| ./BUSYBOX/chmod/vtchmodaa64 | ||
| ./BUSYBOX/chmod/vtchmodm64e | ||
| ./cryptsetup/veritysetup32 | build | Build Instructions: ./cryptsetup/cryptsetup-build.txt |
| ./cryptsetup/veritysetup64 | ||
| ./DMSETUP/dmsetup32 | build | Build Instructions: ./DMSETUP/build.txt |
| ./DMSETUP/dmsetup64 | ||
| ./DMSETUP/dmsetupaa64 | ||
| ./DMSETUP/dmsetupm64e | ||
| ./FUSEISO/vtoy_fuse_iso_32 | build | Build Instructions: ./FUSEISO/build.sh ./FUSEISO/build_aarch64.sh ./FUSEISO/build_libfuse.sh ./FUSEISO/build_libfuse_aarch64.sh |
| ./FUSEISO/vtoy_fuse_iso_64 | ||
| ./FUSEISO/vtoy_fuse_iso_aa64 | ||
| ./IMG/cpio_arm64/ventoy/busybox/a64 | build | Build Instructions: ./BUSYBOX/build.txt ash |
| ./IMG/cpio_arm64/ventoy/busybox/vtchmodaa64 | build | Same with ./BUSYBOX/chmod/vtchmodaa64 Check the file hash to confirm |
| ./IMG/cpio_arm64/ventoy/busybox/xzminidecaa64 | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.17 |
| ./IMG/cpio_arm64/ventoy/tool/lz4cataa64 | build | Same with ./LZIP/lz4cataa64 Check the file hash to confirm |
| ./IMG/cpio_arm64/ventoy/tool/zstdcataa64 | build | Same with ./ZSTD/zstdcataa64 Check the file hash to confirm |
| ./IMG/cpio_mips64/ventoy/busybox/m64 | build | Build Instructions: ./BUSYBOX/build.txt ash |
| ./IMG/cpio_mips64/ventoy/busybox/vtchmodm64e | build | Same with ./BUSYBOX/chmod/vtchmodm64e Check the file hash to confirm |
| ./IMG/cpio_mips64/ventoy/busybox/xzminidecm64e | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.18 |
| ./IMG/cpio_mips64/ventoy/tool/lz4catm64e | build | Same with ./LZIP/lz4catm64e Check the file hash to confirm |
| ./IMG/cpio_x86/ventoy/busybox/64h | build | Build Instructions: ./BUSYBOX/build.txt ash |
| ./IMG/cpio_x86/ventoy/busybox/ash | upstream | Download from BusyBox website. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.4 |
| ./IMG/cpio_x86/ventoy/busybox/vtchmod32 | build | Same with ./BUSYBOX/chmod/vtchmod32 Check the file hash to confirm |
| ./IMG/cpio_x86/ventoy/busybox/vtchmod64 | build | Same with ./BUSYBOX/chmod/vtchmod64 Check the file hash to confirm |
| ./IMG/cpio_x86/ventoy/busybox/vtchmod64_musl | build | Same with ./BUSYBOX/chmod/vtchmod64_musl Check the file hash to confirm |
| ./IMG/cpio_x86/ventoy/busybox/xzminidec32 | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.15 |
| ./IMG/cpio_x86/ventoy/busybox/xzminidec64 | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.16 |
| ./IMG/cpio_x86/ventoy/busybox/xzminidec64_musl | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.16 |
| ./IMG/cpio_x86/ventoy/tool/ar | upstream | Download from BusyBox website. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.2 |
| ./IMG/cpio_x86/ventoy/tool/inotifyd | upstream | Download from BusyBox website. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.3 |
| ./IMG/cpio_x86/ventoy/tool/lz4cat | upstream | URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.1 |
| ./IMG/cpio_x86/ventoy/tool/lz4cat64 | build | Build Instructions: ./LZIP/buildlz4.txt |
| ./IMG/cpio_x86/ventoy/tool/zstdcat | build | Same with ./ZSTD/zstdcat Check the file hash to confirm |
| ./IMG/cpio_x86/ventoy/tool/zstdcat64 | build | Same with ./ZSTD/zstdcat64 Check the file hash to confirm |
| ./INSTALL/EFI/BOOT/BOOTAA64.EFI | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.1-Build grub2 |
| ./INSTALL/EFI/BOOT/BOOTMIPS.EFI | ||
| ./INSTALL/EFI/BOOT/grubia32_real.efi | ||
| ./INSTALL/EFI/BOOT/grubx64_real.efi | ||
| ./INSTALL/EFI/BOOT/grub.efi | upstream | https://github.com/ValdikSS/Super-UEFIinSecureBoot-Disk |
| ./INSTALL/EFI/BOOT/BOOTIA32.EFI | ||
| ./INSTALL/EFI/BOOT/BOOTX64.EFI | ||
| ./INSTALL/EFI/BOOT/grubia32.efi | ||
| ./INSTALL/EFI/BOOT/mmia32.efi | ||
| ./INSTALL/EFI/BOOT/MokManager.efi | ||
| ./INSTALL/tool/aarch64/ash | build | Build Instructions: ./DOC/BUSYBOX/build.txt |
| ./INSTALL/tool/aarch64/hexdump | ||
| ./INSTALL/tool/aarch64/xzcat | ||
| ./INSTALL/tool/i386/ash | ||
| ./INSTALL/tool/i386/hexdump | ||
| ./INSTALL/tool/i386/xzcat | ||
| ./INSTALL/tool/mips64el/ash | ||
| ./INSTALL/tool/mips64el/hexdump | ||
| ./INSTALL/tool/mips64el/xzcat | ||
| ./INSTALL/tool/x86_64/ash | ||
| ./INSTALL/tool/x86_64/hexdump | ||
| ./INSTALL/tool/x86_64/xzcat | ||
| ./INSTALL/tool/aarch64/Ventoy2Disk.gtk3 | build | Build Instructions: ./LinuxGUI/build_gtk.sh |
| ./INSTALL/tool/i386/Ventoy2Disk.gtk3 | ||
| ./INSTALL/tool/i386/Ventoy2Disk.gtk2 | ||
| ./INSTALL/tool/mips64el/Ventoy2Disk.gtk3 | ||
| ./INSTALL/tool/x86_64/Ventoy2Disk.gtk3 | ||
| ./INSTALL/tool/x86_64/Ventoy2Disk.gtk2 | ||
| ./INSTALL/tool/aarch64/Ventoy2Disk.qt5 | build | Build Instructions: ./LinuxGUI/build_qt.sh |
| ./INSTALL/tool/i386/Ventoy2Disk.qt5 | ||
| ./INSTALL/tool/mips64el/Ventoy2Disk.qt5 | ||
| ./INSTALL/tool/x86_64/Ventoy2Disk.qt5 | ||
| ./INSTALL/tool/aarch64/Plugson | build | Build Instructions: ./Plugson/build.sh |
| ./INSTALL/tool/i386/Plugson | ||
| ./INSTALL/tool/mips64el/Plugson | ||
| ./INSTALL/tool/x86_64/Plugson | ||
| ./INSTALL/tool/aarch64/V2DServer | build | Build Instructions: ./LinuxGUI/build.sh |
| ./INSTALL/tool/i386/V2DServer | ||
| ./INSTALL/tool/mips64el/V2DServer | ||
| ./INSTALL/tool/x86_64/V2DServer | ||
| ./INSTALL/tool/aarch64/mkexfatfs | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.9 ./ExFAT/buidexfat.sh ./ExFAT/buidexfat_aarch64.sh ./ExFAT/buidlibfuse.sh ./ExFAT/buidlibfuse_aarch64.sh |
| ./INSTALL/tool/aarch64/mount.exfat-fuse | ||
| ./INSTALL/tool/i386/mkexfatfs | ||
| ./INSTALL/tool/i386/mount.exfat-fuse | ||
| ./INSTALL/tool/mips64el/mkexfatfs | ||
| ./INSTALL/tool/mips64el/mount.exfat-fuse | ||
| ./INSTALL/tool/x86_64/mkexfatfs | ||
| ./INSTALL/tool/x86_64/mount.exfat-fuse | ||
| ./INSTALL/tool/aarch64/vlnk | build | Build Instructions: ./Vlnk/build.sh |
| ./INSTALL/tool/i386/vlnk | ||
| ./INSTALL/tool/mips64el/vlnk | ||
| ./INSTALL/tool/x86_64/vlnk | ||
| ./INSTALL/tool/aarch64/vtoycli | build | Build Instructions: ./vtoycli/build.sh |
| ./INSTALL/tool/i386/vtoycli | ||
| ./INSTALL/tool/mips64el/vtoycli | ||
| ./INSTALL/tool/x86_64/vtoycli | ||
| ./INSTALL/ventoy/imdisk/32/imdisk.cpl | upstream | Download from imdisk project. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.8 |
| ./INSTALL/ventoy/imdisk/32/imdisk.exe | ||
| ./INSTALL/ventoy/imdisk/32/imdisk.sys | ||
| ./INSTALL/ventoy/imdisk/64/imdisk.cpl | ||
| ./INSTALL/ventoy/imdisk/64/imdisk.exe | ||
| ./INSTALL/ventoy/imdisk/64/imdisk.sys | ||
| ./INSTALL/ventoy/iso9660_aa64.efi | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.17 |
| ./INSTALL/ventoy/udf_aa64.efi | ||
| ./INSTALL/ventoy/iso9660_ia32.efi | ||
| ./INSTALL/ventoy/udf_ia32.efi | ||
| ./INSTALL/ventoy/iso9660_x64.efi | ||
| ./INSTALL/ventoy/udf_x64.efi | ||
| ./INSTALL/VentoyGUI.aarch64 | build | Build Instructions: ./LinuxGUI/build_gtk.sh |
| ./INSTALL/VentoyGUI.i386 | ||
| ./INSTALL/VentoyGUI.mips64el | ||
| ./INSTALL/VentoyGUI.x86_64 | ||
| ./INSTALL/Ventoy2Disk.exe | build | Build Instructions: ./Ventoy2Disk/Ventoy2Disk.sln |
| ./INSTALL/Ventoy2Disk_ARM.exe | ||
| ./INSTALL/Ventoy2Disk_ARM64.exe | ||
| ./INSTALL/Ventoy2Disk_X64.exe | ||
| ./INSTALL/ventoy/vtoyjump32.exe | build | Build Instructions: ./vtoyjump/vtoyjump.sln |
| ./INSTALL/ventoy/vtoyjump64.exe | ||
| ./INSTALL/ventoy/ventoy_aa64.efi | build | Build Instructions: ./EDK2/buildedk.sh |
| ./INSTALL/ventoy/ventoy_ia32.efi | ||
| ./INSTALL/ventoy/ventoy_x64.efi | ||
| ./INSTALL/ventoy/vtoyutil_aa64.efi | ||
| ./INSTALL/ventoy/vtoyutil_ia32.efi | ||
| ./INSTALL/ventoy/vtoyutil_x64.efi | ||
| ./INSTALL/ventoy/ipxe.krn | build | Build Instructions: ./IPXE/buildipxe.sh |
| ./INSTALL/ventoy/memdisk | upstream | Download from syslinux project. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.9 |
| ./LiveCD/ISO/EFI/boot/vmlinuz64 | upstream | Download from TinyLinux website. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.14 |
| ./LiveCDGUI/EXT/busybox-x86_64 | build | Same with ./IMG/cpio_x86/ventoy/busybox/busybox64 Check the file hash to confirm |
| ./LiveCDGUI/GRUB/bootx64.efi | build | ./DOC/BuildVentoyFromSource.txt 4.1-Build grub2 |
| ./LiveCD/GRUB/bootx64.efi | ||
| ./LZIP/lunzip32 | build | Build Instructions: ./DOC/BuildVentoyFromSource.txt 4.19 |
| ./LZIP/lunzip64 | ||
| ./LZIP/lunzipaa64 | ||
| ./LZIP/lz4cat64 | build | Build Instructions: ./LZIP/buildlz4.txt |
| ./LZIP/lz4cataa64 | ||
| ./LZIP/lz4catm64e | ||
| ./Plugson/vs/VentoyPlugson/Release/VentoyPlugson.exe | build | Build Instructions: ./Plugson/vs/VentoyPlugson/VentoyPlugson.sln |
| ./Plugson/vs/VentoyPlugson/x64/Release/VentoyPlugson_X64.exe | ||
| ./SQUASHFS/unsquashfs_32 | build | Build Instructions: ./SQUASHFS/build.sh |
| ./SQUASHFS/unsquashfs_64 | ||
| ./SQUASHFS/unsquashfs_aa64 | ||
| ./Unix/ventoy_unix/DragonFly/sbin/dmsetup | upstream | Get from DragonFly ISO. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.13 |
| ./Unix/ventoy_unix/DragonFly/sbin/init | build | Build Instructions: ./Unix/ventoy_unix_src/DragonFly/build.sh |
| ./VBLADE/vblade-master/vblade_32 | build | Build Instructions: ./VBLADE/vblade-master/build.sh |
| ./VBLADE/vblade-master/vblade_64 | ||
| ./VBLADE/vblade-master/vblade_aa64 | ||
| ./Vlnk/vs/VentoyVlnk/Release/VentoyVlnk.exe | build | Build Instructions: ./Vlnk/vs/VentoyVlnk/VentoyVlnk.sln |
| ./VtoyTool/vtoytool/00/vtoytool_32 | build | Build Instructions: ./VtoyTool/build.sh |
| ./VtoyTool/vtoytool/00/vtoytool_64 | ||
| ./VtoyTool/vtoytool/00/vtoytool_aa64 | ||
| ./VtoyTool/vtoytool/00/vtoytool_m64e | ||
| ./VtoyTool/vtoytool/01/vtoytool_64 | ||
| ./VtoyTool/vtoytool/02/vtoytool_64 | ||
| ./ZSTD/zstdcat | build | Build Instructions: ./ZSTD/build.txt |
| ./ZSTD/zstdcat64 | ||
| ./ZSTD/zstdcataa64 | ||
| ./IMG/cpio_x86/ventoy/busybox/busybox32 | build | Build Instructions: ./BUSYBOX/build.txt full busybox |
| ./IMG/cpio_x86/ventoy/busybox/busybox64 | ||
| ./IMG/cpio_x86/ventoy/busybox/xzcat32_musl | ||
| ./IMG/cpio_x86/ventoy/busybox/xzcat64_musl | ||
| ./IMG/cpio_arm64/ventoy/busybox/busyboxaa64 | ||
| ./IMG/cpio_mips64/ventoy/busybox/busyboxm64e | ||
| ISNTALL/ventoy/7z/64/7za.exe | upstream | Download from 7z project. URL & File Hash documented in ./DOC/BuildVentoyFromSource.txt 5.12 |
| ISNTALL/ventoy/7z/32/7za.exe | ||
| ./INSTALL/ventoy/wimboot.i386.efi | build | Build Instructions: ./wimboot/build.sh |
| ./INSTALL/ventoy/wimboot.x86_64 | ||
| ./Unix/ventoy_unix/ClonOS/geom_ventoy_ko/13.x/64/geom_ventoy.ko | build | Build Instructions: ./Unix/BuildUnixKmod.txt |
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/10.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/10.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/11.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/11.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/12.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/12.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/13.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/13.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/14.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/14.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/9.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/FreeBSD/geom_ventoy_ko/9.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/MidnightBSD/geom_ventoy_ko/11.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/MidnightBSD/geom_ventoy_ko/11.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/MidnightBSD/geom_ventoy_ko/2.x/32/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/MidnightBSD/geom_ventoy_ko/2.x/64/geom_ventoy.ko | ||
| ./Unix/ventoy_unix/pfSense/geom_ventoy_ko/14.x/64/geom_ventoy.ko |
- Can github provide build environment for FreeBSD and Windows ?
Yes.
- Can github provide build environment for Linux with glibc/musl ?
Yes.
Thanks for adressing this :)
In my opinion if removing the BLOBs would significantly hinder your general progress on the project then please keep it as a "side project" to slowly remove them, but not as a main focus. Maybe mention them in the main readme and link to the sources you used so others can check them out.
Github build environments also have caching. The compile binaries of build processes that are time consuming or do not change often can be stored as an output artifact as the input for updated Ventoy builds. This may eliminate to create subrepos.
Much of the legitimate concern in #2795 stems from needing to trust Ventoy.
While Ventoy is open source, as @fnr1r has been discovering in https://github.com/fnr1r/ventoy-cpio, the build process is very complicated. Documenting sources of binary files, scripts used in the build process and within the runtime environment would go a long way. The easier it is audit what it takes to build Ventoy, the easier it is to trust Ventoy.
While all binaries and blobs may not be able to be built from source, those that remain should be clearly stated with a reason why they cannot be built from source. For example, if it is proprietary and the source is not available, assuming there is a matching checksum to the original input, it then that puts the need for trust on this third party.
It is also a very difficult goal, but I would strong suggest striving to make Ventoy a reproducible build. In the simplest terms, anyone can build from the source and create the same bit for bit binary every time. Reproducible builds are the gold standard for trusting a built shared binary.
Github build environments also have caching. The compile binaries of build processes that are time consuming or do not change often can be stored as an output artifact as the input for updated Ventoy builds. This may eliminate to create subrepos.
Yes, this is just what I thought.
Much of the legitimate concern in #2795 stems from needing to trust Ventoy.
While Ventoy is open source, as @fnr1r has been discovering in https://github.com/fnr1r/ventoy-cpio, the build process is very complicated. Documenting sources of binary files, scripts used in the build process and within the runtime environment would go a long way. The easier it is audit what it takes to build Ventoy, the easier it is to trust Ventoy.
While all binaries and blobs may not be able to be built from source, those that remain should be clearly stated with a reason why they cannot be built from source. For example, if it is proprietary and the source is not available, assuming there is a matching checksum to the original input, it then that puts the need for trust on this third party.
It is also a very difficult goal, but I would strong suggest striving to make Ventoy a reproducible build. In the simplest terms, anyone can build from the source and create the same bit for bit binary every time. Reproducible builds are the gold standard for trusting a built shared binary.
reproducible build is perfect in theory, but it is very difficult in practice.
For example, if the C code use __DATE__ macro, then the build output will be different each time.
Ventoy use many open source projects, so it can not change all the code to match the requirement of reproducible build.
All the binary files in Ventoy have the corresponding source code. In the document https://github.com/ventoy/Ventoy/blob/master/DOC/BuildVentoyFromSource.txt I have list all the binary files, where it get from or how to build it as follows:
For example, Ventoy directly use some busybox binary files downloaded from busybox official website. BusyBox is a very famous and widely used open-source project. So I think it's OK to directly use their pre-built binaries, if we find that the version Ventoy use has some security problem, this should have been fixed in new BusyBox version, so Ventoy can just use their new fixed version and no need to build it from source.
Can github provide build environment for FreeBSD and Windows?
FreeBSD support in vmactions has been mentioned already. There's also an open issue for FreeBSD support in GitHub actions: https://github.com/actions/runner/issues/385
It would also be possible to add a FreeBSD port for the Ventoy kernel module, which would automatically make it available for all supported FreeBSD versions on an ongoing basis (currently 13.5, 14.2, 15.0). Ventoy could then obtain it from there.
Longer term I have some work in progress for the FreeBSD boot which would avoid the need for a special kernel module.
reproducible build is perfect in theory, but it is very difficult in practice.
Many projects have invested a lot of effort into reproducible builds over the last 10+ years and it is not so difficult to achieve now. Packages for the FreeBSD base system are (nearly) all reproducible. I don't have recent stats on our 3rd party software collection, but Debian's 30000+ package collection is over 95% reproducible.
Many projects have invested a lot of effort into reproducible builds over the last 10+ years and it is not so difficult to achieve now. Packages for the FreeBSD base system are (nearly) all reproducible. I don't have recent stats on our 3rd party software collection, but Debian's 30000+ package collection is over 95% reproducible.
For example, Ventoy use official grub2/busybox/... project source code, I think we should firstly push the upstream project to achieve reproducible build.
Take busybox as example, Venoty directly use its binary function, and I'm not familar with its code and build progress, so if it does not support reproducible build, it's hard for Ventoy to do this.
Agreed that this relies on the upstream project, it's not something Ventoy should need to do itself. In the case of busybox it is already reproducible in Debian's CI (except for i386): https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/busybox.html
In #2795 there are some discuss about the BLOBs in Ventoy.
For a long time, I devoted my limited spare time to adding new features and fixing bugs and didn't get around to considering this.
It should be noted firstly that Ventoy is 100% open souce and everything is transparent. Some binary files are directly get from other open source projects and Ventoy directly use them and don't need any change by default.
Now that someone cares about these BLOBs, so I think we can discuss about this and find a better way and use it step by step in Ventoy. This will definitely take a long time and need some help but I think it's good for Ventoy.
One approach I'm thinking of right now is that, for these BLOBs, I create some sub-repos in github and build them by github CI just once, and use the binary file in Ventoy. If we find some problem in these tools, we can update the source code and build again and use the new binary files in new Ventoy version.
What do you think about this?
For this, I want to confirm that
1. Can github provide build environment for FreeBSD and Windows ? 2. Can github provide build environment for Linux with glibc/musl ?
You can cross-build into your desired platform.
Many projects have invested a lot of effort into reproducible builds over the last 10+ years and it is not so difficult to achieve now. Packages for the FreeBSD base system are (nearly) all reproducible. I don't have recent stats on our 3rd party software collection, but Debian's 30000+ package collection is over 95% reproducible.
For example, Ventoy use official grub2/busybox/... project source code, I think we should firstly push the upstream project to achieve reproducible build. Take busybox as example, Venoty directly use its binary function, and I'm not familar with its code and build progress, so if it does not support reproducible build, it's hard for Ventoy to do this.
You might want to use Nix the package manager. Nixpkgs is currently the biggest Linux package repository. Although versions may differ, you can always lock which revision of nixpkgs goes into building the project for maximum reproducibility (flakes makes this much easier, but it's a great way to burn yourself out if you've never used Nix before).
You can find what's available on both the stable and unstable branch here. Don't hesitate to ask for help around Nix if you get around to use it.
Thanks for adressing this :)
In my opinion if removing the BLOBs would significantly hinder your general progress on the project then please keep it as a "side project" to slowly remove them, but not as a main focus. Maybe mention them in the main readme and link to the sources you used so others can check them out.
It's a matter of time till it becomes the main focus whether or not you like it
Now that someone cares about these BLOBs, so I think we can discuss about this and find a better way and use it step by step in Ventoy.
I think this is a great idea!
This will definitely take a long time and need some help but I think it's good for Ventoy.
I can help, I have worked with CI/CD before and can make a few suggestions. What if you make Ventoy an organization to help separate dependencies from the main project and manage permissions for those who are able to contribute?
One approach I'm thinking of right now is that, for these BLOBs, I create some sub-repos in github and build them by github CI just once, and use the binary file in Ventoy.
Your approach is good. We could fork the dependencies into a Ventoy organization and manage it as a separate repository. We can set up some automation to watch the upstream projects and pull in new releases as they update. From there, the dependencies would start a GitHub actions build and cross-compile as needed. The Ventoy build scripts would pull the artifacts from GitHub so the binaries would never have to be checked into the main Ventoy repository.
This would have to happen slowly since Ventoy is complex and cross-compiling can be tricky. Let me know how I can support this awesome project!
It is also a very difficult goal, but I would strong suggest striving to make Ventoy a reproducible build. In the simplest terms, anyone can build from the source and create the same bit for bit binary every time. Reproducible builds are the gold standard for trusting a built shared binary.
reproducible build is perfect in theory, but it is very difficult in practice. For example, if the C code use
__DATE__macro, then the build output will be different each time. Ventoy use many open source projects, so it can not change all the code to match the requirement of reproducible build.
However the __DATE__ macro is a solved problem, please refer to SOURCE_DATE_EPOCH. Honestly there are some challenges in achieving reproducible builds, but they are NOT too difficult in practice. Debian has more than 95% of packages reproducible on x86_64 and aarch64 [1], and its installer images are reproducible [2][3]; Arch Linux has about 85% of packages reproducible [4].
[1] https://reproduce.debian.net/ [2] https://wiki.debian.org/ReproducibleBuilds [3] https://lwn.net/Articles/985739/ [4] https://reproducible.archlinux.org/
For some files (e.g. BusyBox) since Ventoy directly use its binary file, so maybe I can directly get one from Debian/Arch pacckage which is already reproducible.
For some busybox binaries, can we confirm that the one used in Ventoy now (get from busybox official website) is already reproducible?
For some busybox binaries, can we confirm that the one used in Ventoy now (get from busybox official website) is already reproducible?
Reproducibility nowadays depends on the project's dependencies for the most part (although many projects do things that completely break it, like attempting to connect to the internet, needing the system time for whatever reason at compile time, etc.).
Try looking into Nix or Docker. I personally prefer Nix but you might have to rely on the community's help (documentation leaves a lot to be desired)
There is some Windows EXE file, how to process this?
7-zip releases their source code, so you can download the source for the exact release needed. I have only built 7-zip on Linux with gcc-mingw-w64-x86-64, but you should be able to build it on a Windows GitHub runner through GitHub actions.
There is some Windows EXE file, how to process this?
7-zip releases their source code, so you can download the source for the exact release needed. I have only built 7-zip on Linux with
gcc-mingw-w64-x86-64, but you should be able to build it on a Windows GitHub runner through GitHub actions.
Can anyone help to do this? Ventoy only need the final exe file, nothing special.
There is some Windows EXE file, how to process this?
7-zip releases their source code, so you can download the source for the exact release needed. I have only built 7-zip on Linux with
gcc-mingw-w64-x86-64, but you should be able to build it on a Windows GitHub runner through GitHub actions.Can anyone help to do this? Ventoy only need the final exe file, nothing special.
I will set up a repository for this and get it building and then we can discuss how you want to include this in Ventoy.
Can anyone help to do this? Ventoy only need the final exe file, nothing special.
FWIW, there is an example of cross-compiling 7z in Docker: https://github.com/skeeto/w64devkit/commit/ecc5c65615a64a68e58dc7aaa53a483bc0d5b694.
@ventoy
You can add this github workflow to your repository and manually run the action(or modify it to run whenever)
name: Compile Windows 7-Zip
on:
workflow_dispatch:
permissions: {}
jobs:
build:
runs-on: ubuntu-24.04
steps:
- name: Checkout repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Install Nix
uses: cachix/install-nix-action@526118121621777ccd86f79b04685a9319637641 # v31
with:
extra_nix_config: |
experimental-features = nix-command flakes
sandbox = true
- name: Build Windows 64-bit binary
id: result64
run: |
nix build \
"github:NixOS/nixpkgs/09e24d201f135a1f1751e8c9682ed2ba4564c696#pkgsCross.mingwW64._7zz" \
--option restrict-eval true
echo "bin64=$(readlink -f result/bin/7zz.exe)" >> "$GITHUB_OUTPUT"
- name: Upload 64-bit binary artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: 7zip-win64
path: ${{ steps.result64.outputs.bin64 }}
if-no-files-found: error
- name: Build Windows 32-bit binary
id: result32
run: |
nix build \
"github:NixOS/nixpkgs/09e24d201f135a1f1751e8c9682ed2ba4564c696#pkgsCross.mingw32._7zz" \
--option restrict-eval true
echo "bin32=$(readlink -f result/bin/7zz.exe)" >> "$GITHUB_OUTPUT"
- name: Upload 32-bit binary artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: 7zip-win32
path: ${{ steps.result32.outputs.bin32 }}
if-no-files-found: error
Which you then can run https://github.com/ventoy/Ventoy/actions
When updating, you just get a newer commit from master and replace this one: 09e24d201f135a1f1751e8c9682ed2ba4564c696
And here you can see the package definition on how it's built: https://github.com/NixOS/nixpkgs/blob/09e24d201f135a1f1751e8c9682ed2ba4564c696/pkgs/by-name/_7/_7zz/package.nix
For more inspiration, you can look here: https://github.com/NixOS/nixpkgs/tree/master/.github/workflows
Which you then can run https://github.com/ventoy/Ventoy/actions When updating, you just get a newer commit from master and replace this one:
09e24d201f135a1f1751e8c9682ed2ba4564c696
@Saterfield990 thanks for your suggestion on forking rather than mirroring, I did not know that the maintainer had their own GitHub mirror these days.
Your proposed solution is over complicated, it introduces 2 new dependencies: the nix packaging process and the 7-zip url. If either are unavailable building will not be possible. It does not increase transparency, if anything the current @Ventoy solution of including the url and hash is more transparent.
It would be harder to get notified of new 7-zip releases and more annoying since you have to change a hash in multiple places. Cross compiling in this case is unnecessary since we can use the GitHub Windows runner, and the 7-zip maintainer recommends using MSVC in the README.md.
@ventoy everyone seems to be flipping out about this and I would suggest maybe adding a update to readme about the issue, reiteratting your open source stance and a small statement about what you are doing to fix this and maybe add todo check list up to people's reasonable expectations.
Just a simple suggestion.
Which you then can run https://github.com/ventoy/Ventoy/actions When updating, you just get a newer commit from master and replace this one:
09e24d201f135a1f1751e8c9682ed2ba4564c696@Saterfield990 thanks for your suggestion on forking rather than mirroring, I did not know that the maintainer had their own GitHub mirror these days.
Your proposed solution is over complicated, it introduces 2 new dependencies: the nix packaging process and the 7-zip url. If either are unavailable building will not be possible. It does not increase transparency, if anything the current @ventoy solution of including the url and hash is more transparent.
It would be harder to get notified of new 7-zip releases and more annoying since you have to change a hash in multiple places. Cross compiling in this case is unnecessary since we can use the GitHub Windows runner, and the 7-zip maintainer recommends using MSVC in the README.md.
This is a Nix runner. It's not on Windows natively.
You need a proper system that can get you from A to B regardless of how many or how little you have.
@ventoy everyone seems to be flipping out about this and I would suggest maybe adding a update to readme about the issue, reiteratting your open source stance and a small statement about what you are doing to fix this and maybe add todo check list up to people's reasonable expectations.
Just a simple suggestion.
I will create a new branch to process this. But before that, I need to fully discuss about every details.
You can add this github workflow to your repository and manually run the action(or modify it to run whenever)
name: Compile Windows 7-Zip
on: workflow_dispatch:
permissions: {}
jobs: build: runs-on: ubuntu-24.04
steps: - name: Checkout repository uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install Nix uses: cachix/install-nix-action@526118121621777ccd86f79b04685a9319637641 # v31 with: extra_nix_config: | experimental-features = nix-command flakes sandbox = true - name: Build Windows 64-bit binary id: result64 run: | nix build \ "github:NixOS/nixpkgs/09e24d201f135a1f1751e8c9682ed2ba4564c696#pkgsCross.mingwW64._7zz" \ --option restrict-eval true echo "bin64=$(readlink -f result/bin/7zz.exe)" >> "$GITHUB_OUTPUT" - name: Upload 64-bit binary artifact uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: 7zip-win64 path: ${{ steps.result64.outputs.bin64 }} if-no-files-found: error - name: Build Windows 32-bit binary id: result32 run: | nix build \ "github:NixOS/nixpkgs/09e24d201f135a1f1751e8c9682ed2ba4564c696#pkgsCross.mingw32._7zz" \ --option restrict-eval true echo "bin32=$(readlink -f result/bin/7zz.exe)" >> "$GITHUB_OUTPUT" - name: Upload 32-bit binary artifact uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: 7zip-win32 path: ${{ steps.result32.outputs.bin32 }} if-no-files-found: errorWhich you then can run https://github.com/ventoy/Ventoy/actions When updating, you just get a newer commit from master and replace this one:
09e24d201f135a1f1751e8c9682ed2ba4564c696And here you can see the package definition on how it's built: https://github.com/NixOS/nixpkgs/blob/09e24d201f135a1f1751e8c9682ed2ba4564c696/pkgs/by-name/_7/_7zz/package.nix
For more inspiration, you can look here: https://github.com/NixOS/nixpkgs/tree/master/.github/workflows
Thank you for your effort. As we rebuild 7zip with MinGW not native VS, can we guarantee its functionality?
You can add this github workflow to your repository and manually run the action(or modify it to run whenever)
name: Compile Windows 7-Zip
on: workflow_dispatch:
permissions: {}
jobs: build: runs-on: ubuntu-24.04
steps: - name: Checkout repository uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install Nix uses: cachix/install-nix-action@526118121621777ccd86f79b04685a9319637641 # v31 with: extra_nix_config: | experimental-features = nix-command flakes sandbox = true - name: Build Windows 64-bit binary id: result64 run: | nix build \ "github:NixOS/nixpkgs/09e24d201f135a1f1751e8c9682ed2ba4564c696#pkgsCross.mingwW64._7zz" \ --option restrict-eval true echo "bin64=$(readlink -f result/bin/7zz.exe)" >> "$GITHUB_OUTPUT" - name: Upload 64-bit binary artifact uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: 7zip-win64 path: ${{ steps.result64.outputs.bin64 }} if-no-files-found: error - name: Build Windows 32-bit binary id: result32 run: | nix build \ "github:NixOS/nixpkgs/09e24d201f135a1f1751e8c9682ed2ba4564c696#pkgsCross.mingw32._7zz" \ --option restrict-eval true echo "bin32=$(readlink -f result/bin/7zz.exe)" >> "$GITHUB_OUTPUT" - name: Upload 32-bit binary artifact uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: 7zip-win32 path: ${{ steps.result32.outputs.bin32 }} if-no-files-found: errorWhich you then can run https://github.com/ventoy/Ventoy/actions When updating, you just get a newer commit from master and replace this one:
09e24d201f135a1f1751e8c9682ed2ba4564c696And here you can see the package definition on how it's built: https://github.com/NixOS/nixpkgs/blob/09e24d201f135a1f1751e8c9682ed2ba4564c696/pkgs/by-name/_7/_7zz/package.nix
For more inspiration, you can look here: https://github.com/NixOS/nixpkgs/tree/master/.github/workflows
Thank you for your effort. As we rebuild 7zip with MinGW not native VS, can we guarantee its functionality?
As 7z and busybox, I think the current solution is already very transparent.
I document about where the 7z exe and busybox binary files are downloaded (from the upstream open source project official website). and give the file hash value. We can prove by the file hash to confirm that the files are indeed downloaded from upstream, not unknown.
These upstream projects have complicated code and build system. Ventoy only use their binary function. It's a lot of work for Ventoy to rebuild all of them and also we can not guarantee the functionality after rebuild.
Both busybox and 7z are much more popular open source project and used by much more people than Ventoy. If we find any of them has security problem we can replace with the latest version or use other method.
Do you think so?