Ventoy [issue]: vlnk implementation should be thought-over-again and redone , error Disk in vlnk not found

Official FAQ

[X] I have checked the official FAQ.

Ventoy Version

1.0.79

What about latest release

Yes. I have tried the latest release, but the bug still exist.

BIOS Mode

UEFI Mode

Partition Style

GPT

Disk Capacity

512gb

Disk Manufacturer

samsung

Image file checksum (if applicable)

Yes.

Image file download link (if applicable)

https://na

What happened?

A bug happened! ventoyvlnk.sh -v -l path_to_vlnkfile gave error "Disk in vlnk not found"

After tracing through the both ventoyvlnk.sh shell-script and main_linux.c C-code, I inferred that it could be because it identifies 32bit disk_signature as 0x00000000. I have 3 internal drives in 3 bays (1ssd+2hdd). Ventoy is on the internal drive sdc second partition (sdc2) Each of the internal drives has GPT with protective MBR. I used Linux commands dd and od to see hexdump of first 512 bytes of hard drives. The first 512 bytes on sda seem more null than the other two, possibly because it was zeroed some time in distant past. The 4 consecutive bytes at 0x1b8 are indeed null (0x00000000) on all 3 drives. Perhaps the implementation cannot uniquely identify drive by such null signature.

The following disadvantages in the current approach to implementing vlnk

vlnk file is binary and not human read-writable plain text
requires an elaborate script to create the vlnk file
vlnk files will need to be regenerated and overwritten if disk is replaced and disk signatures change. Sometimes, stray disaster recoveries, like overwriting zero blocks, restoring boot-sectors and recreating boot-records might regenerate disk signatures or zero it out.
vlnk file is very large 32768 bytes for the tiny information it contains
drive signature cannot be relied upon, it could be null also, what if drives are cloned, hardware or by software
as mentioned, where and what a reliable disk-signature might depend on the partition-table format (DosMBR/GPT/otherMBR)

I think maybe this is wrong way to identify drives. The signatures for GPT and MBR could be in different places, GPT has its own disk-GUID. GPT is not obligated to fill all fields of protective mbr, and gdisk only fills a partition-entry-slot if protective-mbr is used. Perhaps identifying drives should be done the same way grub does it. If it were that easy to identify drives by signature, grub might have done it that way. The bios/firmware initialization does not guarantee that drives are enumerated the same on each boot, and the user may also swap drives in bays. Ex, in grub, during runtime, the below is used to search for pathname in (hd1,2) and set a variable like root to that value search --file --set var pathtofilename https://www.gnu.org/software/grub/manual/grub/html_node/search.html Perhaps grub's-implementation can be reused by ventoy. That is, ventoy should also, just like grub, infer drive by searching for given filepath on all partitions in every drive at runtime.

logs

[root@sirius transcend]# ( cd  ~/tmp/ventoy-1.0.79/; src="/mnt/a16" ; flname="m01_lnx.raw" ; ext="img" ; dst="/vol/15KJ_Images/transcend" ;./VentoyVlnk.sh -v -c "${src}/${flname}.${ext}" ; mv -f "${flname}.vlnk.${ext}" "${dst}/" ; ls -l "${dst}/${flname}.vlnk.${ext}" )
Create vlnk for /mnt/a16/m01_lnx.raw.img
check vlnk
file size 49861885952 is not a vlnk file size
Full file path is /mnt/a16/m01_lnx.raw.img
IMGPATH=/m01_lnx.raw.img IMGFILE=m01_lnx.raw.img
mountpoint is /mnt/a16  dev 259:0
File system of /dev/sda16 is exfat
FS OK
Major=259 Minor=0
disk partition is sda16
PARTSTART=500117504 DISK=/dev/sda
Output VLNK file is m01_lnx.raw.vlnk.img
create vlnk
====== Vlnk file create success =========

-rw-r--r--. 1 root root 32768 Aug 11 17:56 /vol/15KJ_Images/transcend/m01_lnx.raw.vlnk.img



[root@sirius transcend]# ( cd  ~/tmp/ventoy-1.0.79/; flname="m01_lnx.raw" ; ext="img" ; dst="/vol/15KJ_Images/transcend" ; ./VentoyVlnk.sh -v -l "${dst}/${flname}.vlnk.${ext}" )
Parse vlnk for /vol/15KJ_Images/transcend/m01_lnx.raw.vlnk.img
check vlnk
Vlnk data check OK
parse vlnk
disk_signature:00000000
file path:</m01_lnx.raw.img>
part offset: 256060162048
Disk in vlnk not found!

Aug 11 '22 08:08 hgkamath

vlnk file is in binary format
vlnk file is 32KB size
vlnk file use disk signature to identify disk. All the above are for simplify code implementation in Ventoy. (Ventoy will ignore files < 32KB by default) We have to process duplicate pathname if we only record a simple file pathname. Maybe we can report a warning when disk signature is zero in non-Windows system. In your case, I think give each disk a uniq disk signature should work.

Aug 11 '22 09:08 ventoy

thats what I plan to do. Please tell me, how ( command) is the easiest way to insert a 32bit disk signature in there? other than dd-od-edit-od-dd ?

Aug 11 '22 09:08 hgkamath

Maybe, if disk signature is 0, you can give warning when making vlnk and then search all drives for the path specified and use the first one you find? e.g. when making vlnk file: 'Warning: Disk has no signature bytes, first file matching path 'xxx/xxx/xxx' will be used by Ventoy. Make sure only one file exists on all disks.' and modify Ventoy to search all partitions in system (hd, usb, not cd/floppy?) if there was no disk signature on the target disk. Is this possible?

fdisk can change signature: While in fdisk, enter extra commands section with X, then you can change the Disk ID using the I command.

Aug 11 '22 10:08 steve6375

thats what I plan to do. Please tell me, how ( command) is the easiest way to insert a 32bit disk signature in there? other than dd-od-edit-od-dd ?

dd if=/dev/random of=/dev/sdX bs=1 count=4 seek=440 should work. (sdX is your disk (e.g. sda sdb sdc ...)

Aug 11 '22 11:08 ventoy

Maybe, if disk signature is 0, you can give warning when making vlnk and then search all drives for the path specified and use the first one you find? e.g. when making vlnk file: 'Warning: Disk has no signature bytes, first file matching path 'xxx/xxx/xxx' will be used by Ventoy. Make sure only one file exists on all disks.' and modify Ventoy to search all partitions in system (hd, usb, not cd/floppy?) if there was no disk signature on the target disk. Is this possible?

fdisk can change signature: While in fdisk, enter extra commands section with X, then you can change the Disk ID using the I command.

Maybe we can automatically give the disk a signature when 0.

Aug 11 '22 11:08 ventoy

Wouldn't that affect the checksum of the MBR? Maybe it would cause a problem? What about previously made vlnk files - would these have to be redone if their target device signature is no longer 0 0 0 0 ?

Aug 11 '22 11:08 steve6375

It doesn't matter, actually When install Ventoy, a random disk signature will be written to the disk. If there is only one disk that has 0 disk signature in the computer, it doesn't matter. It can identify the disk If there are more that one disks that has 0 disk signature, the previously made vlnk file may have not work.

Aug 11 '22 11:08 ventoy

But write any data to user's disk is not a good idea under any circumstances. So I think a warning is enough. (at most plus some tips for how to give a disk a signature)

Aug 11 '22 11:08 ventoy

A search path would be more useful than using disk signature anyway. For instance, vlnk file only works on that specific target disk. This means users cannot simply copy the vlnk files and it will work on their other systems - they could have several PCs/notebooks or give out Ventoy to use on many different systems. Having two types of vlnk file (one which uses disk signature and one uses search path) would be a good option?

Aug 11 '22 11:08 steve6375

If you connect any drive to a Windows Vista+ system, a disk signature of 0 0 0 0 is automatically updated to a new (random) signature. So the issue is with Linux-only systems.

Aug 11 '22 11:08 steve6375

That make sense. But since user can just use F2 to browse all the disks/partitions to boot, so vlnk file is not needed for most users and most cases. So I think current implement is OK.

Aug 11 '22 11:08 ventoy

OK - so using linux ventoyvlnk.sh script - just warn if target disk has 0 0 0 0 disk signature bytes? Maybe tell user command to fix it or provide the option to fix it (with warning that previously made .vlnk files will no longer work)? Note that Windows BCDs contain the disk signature, so dangerous to change disk signature on a Windows disk.

Aug 11 '22 11:08 steve6375

Yes. It's a linux-only problem. So some warning/tip message in the ventoyvlnk.sh is enough. After all, this is not often the case.

Aug 11 '22 11:08 ventoy

Using fdisk may be dangerous, what if it makes GPT not recognizable? I did a test. od diff , 512 bytes dd-ed from /dev/sda, and diffed changes between before and after

[root@sirius transcend]#  od -A x -t x1z -v qq1 > qw1
before:
0001a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
0001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
0001c0 02 00 ee ff ff ff 01 00 00 00 af 12 9e 3b 00 00  >.............;..<
after:
0001a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
0001b0 00 00 00 00 00 00 00 00 01 ba ba ba 00 00 00 00  >................<
0001c0 02 00 ee ff ff ff 01 00 00 00 00 00 00 00 00 00  >................<

In the above, we see that fdisk also modifies some bytes on the next line

I first dd-ed 2048 512b-blocks from the disk
used fdisk to modify identifier as mentioned by steve6475
noted that there are bytes other than identifier bytes that are also changed.
fdisk only does MBR view if only 1-block of 512-bytes is dd-ed off.

So one is worried if fdisk/gdisk and other partition tools clobber more than required. GPT, MBR may also keep/overwrite the backupGPT, backupMBR also saved inside un-allocated blocks at the end of disk. They probably also also detect and update disk-geometry (#sectors ).

shell scriptlet to modify the disk-identifier bytes directly

note the little-endian byte-order

dd if=/dev/sda of=./qq3 bs=512 count=1
printf '\x01\xab\xab\xab' | dd of=qq3 bs=1 seek=440 count=4 conv=notrunc
dd if=./qq3 of=/dev/sda

[root@sirius transcend]# fdisk ./qq3
Welcome to fdisk (util-linux 2.38).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
GPT PMBR size mismatch (1000215215 != 0) will be corrected by write.
Command (m for help): p
Disk ./qq3: 512 B, 512 bytes, 1 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xababab01
...

The above can be checked using od hexdump, and diff can be used to verify that only the 4 bytes of concern are modified. When using fdisk directly on disk it first recognizes GPT. One then has to goto advanced-edit(m) followed by protectiveMBR-edit(M) to view the identifier.

To answer the question, How does one get here?

The following disk operations may be typical to many advanced users

Upon purchasing new machine
shrink OEM windows and other large partitions down to 60 Gb
use clonezilla to backup partitions EFISYS, Win10, WinRE, OEM_RE to an external backup-drive somewhere
move the Win10, WinRE partitions to the end of drive.
create data partition, Linux partitions etc
make as dual boot Linux/windows10 system

As machine ages, disks need to be upgraded to newer higher capacity disk,

Format partition table as GPT on new disk
copy partitions over from old disk, make other partitions
Reinstall boot-loaders, grub and win10-BCD

Over the course of usage

disk will have many times be formatted, re-partitioned, experimented with.
reinstall windows or Linux many times over.
recover/restore partition-tables and volumes from dd and or clonezilla-backups.

This is common usage to advanced users, and so one should not assume that this is a rare event.

The case against modifying disk-identifiers

Firstly, until this point, in all my years of disk-management, I never needed to do this. I personally don't use any backup-software that uses disk-identifiers. So I can modify disk identifier any time if needed, and I don't care about duplicates as long as long as at any given time, the connected active disks can be made to have unique disk-identifiers such as 0xababab01 0xababab02 0xababab03. But, it is conceivable that there are disk and volume backup software tools such as clonezilla that record everything. changing disk identifier manually at will might trip them up. These disk identifiers are also meant for such kind of archival and update tracking by such backup-software, as disk-volumes are shifted around. So when restoring from an old dd/clonezilla backup, one needs to remember and set these disk-identifiers. I cannot find any spec file, that mentions if the 0x00000000 disk-identifier in protective-MBR of GPT is intentional. Perhaps its set to zero with good reason. One cannot know in advance if any tool, such as DISKPART or 3rd party boot-loader will behave differently if it were set to something.

I strongly feel the correct way to do linking

is by at boot-run-time read-parsing a plaintext link-file, followed by drive-detection using exhaustive-path-search, just like how GRUB does it.
and not by pre-detecting and hard-coding the disk signature in the vlnk-link-file at creation-time.
if path cannot be resolved, its ok to give Oops msg at boot-run-time, saying "multiple image locations found error" or "image location not found error" as may be the case and bail out.

This way,

it won't require any disk-identifier management on part of user, one-time or in the long-term.
it is also intuitive to those who know how grub works, no need for extra learning
easy to edit plain text file
The 32Kb file size can also be achieved by with padding, ignoring content after say a final line. Padding could be vlnk/documentation below certain line.

You are correct in asserting that using the F2 browser approach obviates most use cases of vlnk-links. A use case of vlnk-links that remains is to put all the vlnk-links in one folder such as E:\transcend, so that one does not have to remember where each image is, and spend time to go searching for it.
Further down the line, it may be possible to configure custom image boot options that are stored in different vlnk-link files to the same image-file, like booting with different efi-bootvars set ... who-knows?
Another way to create ventoy links is through a json config, similar to how ventoy config-options are done. Ventoy grub just reads this vlnk-json-config from a fixed location (sdc1/vtoyvlnk.json) at boot time and just populates the menu. I think the image list plugin (https://www.ventoy.net/en/plugin_imagelist.html) sort of does this. Currently, the path list , could be a "white list" or a "black list" for paths in the ventoy partition, but, its syntax could be expanded for specifying an "additional path list" of images along with their partition-info, which just prepends to the auto-search populated menu.

logs

[root@sirius transcend]# ( cd  ~/tmp/ventoy-1.0.79/; src="/mnt/a16" ; flname="m01_lnx.raw" ; ext="img" ; dst="/vol/15KJ_Images/transcend" ;./VentoyVlnk.sh -v -c "${src}/${flname}.${ext}" ; mv -f "${flname}.vlnk.${ext}" "${dst}/" ; ls -l "${dst}/${flname}.vlnk.${ext}" ) 
Create vlnk for /mnt/a16/m01_lnx.raw.img
check vlnk
file size 49861885952 is not a vlnk file size
Full file path is /mnt/a16/m01_lnx.raw.img
IMGPATH=/m01_lnx.raw.img IMGFILE=m01_lnx.raw.img
mountpoint is /mnt/a16  dev 259:0
File system of /dev/sda16 is exfat
FS OK
Major=259 Minor=0
disk partition is sda16
PARTSTART=500117504 DISK=/dev/sda
Output VLNK file is m01_lnx.raw.vlnk.img
create vlnk
====== Vlnk file create success =========

-rw-r--r--. 1 root root 32768 Aug 11 19:01 /vol/15KJ_Images/transcend/m01_lnx.raw.vlnk.img


[root@sirius transcend]# ( cd  ~/tmp/ventoy-1.0.79/; flname="m01_lnx.raw" ; ext="img" ; dst="/vol/15KJ_Images/transcend" ; ./VentoyVlnk.sh -v -l "${dst}/${flname}.vlnk.${ext}" ) 
Parse vlnk for /vol/15KJ_Images/transcend/m01_lnx.raw.vlnk.img
check vlnk
Vlnk data check OK
parse vlnk
disk_signature:ABABAB01
file path:</m01_lnx.raw.img>
part offset: 256060162048
Disk is <sda>
Find correct partition </dev/sda16>
moutpoint of /dev/sda16  is </mnt/a16>
Vlnk Point: /mnt/a16/m01_lnx.raw.img
File Exist: YES

my system is GPT/UEFI. I hope windows won't notice and barf, when I boot into windows next time.

Updates

Direct boot of windows and Linux worked
Ventoy - direct boot of linux uefi image worked
Ventoy - boot via vlnk of same linux uefi image also worked The following I found a bit strange. I understand if one says image won't boot without vtoyboot.sh being run. The documentation on vtoyboot.sh states that booting up has certain probability. What causes this uncertainty/probability of boot? I experienced the following. I first booted into a image on sda16 directly using F2 browser, just to confirm img.vtoy is bootable. Then I rebooted again via the vlnk. It gave a message about Uefi not loading, and boot halted. Then I rebooted directly into img.vtoy image on sda16 using F2 browser. I presumed that for that image I had not run vtoyboot.sh. and so ran it again and regenerated the iniramfs-s. Then rebooted again, but this time with vlnk and it worked. That raises question to me, How was the image, at first, able to be directly booted even before running vtoyboot.sh. It should not have booted with certainity if it needed vtoyboot. Ref: https://www.ventoy.net/en/plugin_vtoyboot.html ... You need to run the vtoyboot script again after the update or there is a certain probability that the vDisk will not boot next boot.

Aug 11 '22 13:08 hgkamath