udftools
udftools copied to clipboard
pali
udffsck 1.00-beta
I just finished udffsck 1.00-beta. It is first open source implementation for Linux, which is capable of either check and fix UDF file system.
It is able to check and fixes UDF up to version 2.01, just like rest of package.
Some random notes:
- Squash fixup commits. Commits which fixes just another commits in this pull requests are not useful in git history.
- Correct UDF is also with just only 2 AVDPs. Therefore missing third AVDP should not be reflected as error or warning. Also missing (not broken) third AVDP should not be automatically created. In some cases there can be another (non-UDF) data.
- Current implementation recursively traverse file hierarchy. How fast it is and what is time complexity? Specially what would happen on 512GB UDF HDD full of files?
- Linux kernel udf driver set integrityType to LVID_INTEGRITY_TYPE_OPEN when mounting device and set to CLOSE when unmounting. Therefore integrityType is handled as "dirty" bit. udffsck should implement quick/fast mode when it just checks headers (AVDP, VRS, LVID) if integrityType is closed, similar like e2fsck when fs is not dirty.
- Travis tests for fsck should be implemented as new job Travis Matrix (currently there is merge conflict)
- ANSI terminal colors should be used only when printing on terminal. Lot of times fsck output is forwarded to some text log file, specially when running at boot time.
- Definition of flags which enable C99 compiler should be done by autoconf (IIRC there is some M4 macro for it).
- Check that first 8 (or better 16) chars of volsetid are all hexadecimal. It is required by UDF specification and blkid uses it as disk UUID.
- Missing race condition free check that disk is not mounted before running udffsck
- mmap on >= 4GB disk image on 32bit machine would fail and therefore udffsck would not work
- [ ] Squash fixup commits. Commits which fixes just another commits in this pull requests are not useful in git history. TBD
- [x] Correct UDF is also with just only 2 AVDPs. Therefore missing third AVDP should not be reflected as error or warning. Also missing (not broken) third AVDP should not be automatically created. In some cases there can be another (non-UDF) data. Fix: Third AVDP is marked as broken only when checksum and identifier passes (e.g. only error at CRC and position are possible for third AVDP)
- [ ] Current implementation recursively traverse file hierarchy. How fast it is and what is time complexity? Specially what would happen on 512GB UDF HDD full of files? Comment: Hard to say, I haven't tested on larger FS than 16GB flash drive. It is not super fast, it take some time to check it. It really depends on amount of files rather than their size.
- [x] Linux kernel udf driver set integrityType to LVID_INTEGRITY_TYPE_OPEN when mounting device and set to CLOSE when unmounting. Therefore integrityType is handled as "dirty" bit. udffsck should implement quick/fast mode when it just checks headers (AVDP, VRS, LVID) if integrityType is closed, similar like e2fsck when fs is not dirty. Fixed.
- [ ] Travis tests for fsck should be implemented as new job Travis Matrix (currently there is merge conflict) Comment: I understand what you want, but I don't know how to do that. Can you please provide some example? Please notice there are some parts before and also run script is pretty long too due disk size limitations on Travis. ANSI terminal colors should be used only when printing on terminal. Lot of times fsck output is forwarded to some text log file, specially when running at boot time. Fixed. I added flag for coloring.
- [x] Definition of flags which enable C99 compiler should be done by autoconf (IIRC there is some M4 macro for it). Fixed with AC_PROG_CC_C99
- [x] Check that first 8 (or better 16) chars of volsetid are all hexadecimal. It is required by UDF specification and blkid uses it as disk UUID. Fixed.
- [x] Missing race condition free check that disk is not mounted before running udffsck. Fixed: I added check against /etc/mtab. It works for /dev/sdX but not for images, because they mount as /dev/loopX and I compare given path with mtab entry.
- [x] mmap on >= 4GB disk image on 32bit machine would fail and therefore udffsck would not work Fixed: It was struggle, but it seems it works (tested on 32bit virtual machine)
- /etc/mtab does not have to be up-to-date, use /proc/mounts instead
- on Linux 2.6+ you can atomically check if block device is not mounted && open it via O_EXCL flag (see documentation)
- test for 32bit can be done also on 64bit system, just compile & link with flag -m32
- travis would be hard & tricky (and you cannot use there sudo!), so look at it after everything else would be done
- [x] /etc/mtab does not have to be up-to-date, use /proc/mounts instead Fixed
- [ ] on Linux 2.6+ you can atomically check if block device is not mounted && open it via O_EXCL flag (see documentation) Comment: This would be useful for reading block devices only, beacuse O_EXCL must be used with O_CREAT, otherwise is behaviour undefined. As you write, there is exception for Linux 2.6+, but it is only for block devices and image is not block device (correct me if I am wrong). Because of this, I think it is wise to keep actual design instead of this.
- [x] test for 32bit can be done also on 64bit system, just compile & link with flag -m32 TBD: I plan to add this as Travis job to ensure testing for 32 bit and 64 bit since there is lot of pitfalls.
- [ ] travis would be hard & tricky (and you cannot use there sudo!), so look at it after everything else would be done Comment: Why not sudo? I agree with rest, but I can't see why you can't use sudo...
- [ ] There is testing of large FS going on. I found some nasty bugs there in FE/FID parsing, so there is some work to do.
O_EXCL should be used only for block devices on Linux. Use stat() for checking if path is block device, then do open() with or without O_EXCL (based on fact if is block device) and after open via fstat() verify that fd is of same type (block or non-block device) as in stat() call to prevent race condition.
O_EXCL is the only race-free way on Linux to ensure that file system on block device is not mounted.
Ad sudo, when is not used then travis jobs are running in docker instead of full VM. Full VM has slow startup. Increasing number of jobs cause too much total time spent in running tests. So so it should be switched to docker which has fast startup, but due to security Travis disabled sudo in them.
Avoiding usage of sudo should be simple there. Packages can be installed via apt plugin (see current .travis.yml) and cmocka could be installed somewhere into user home dir (for which is root not needed).
Another thing which fsck should do: check if d-string and d-characters values are valid (+ ability to fix it).
Ad O_EXCL: mkudffs now has this race-free implementation, see commit https://github.com/pali/udftools/commit/6fb2b8f7a8d522040d97be9a835b2d71671e5028
Ad color support, e.g. ls --color=auto uses check based on isatty(). But I'm not sure what is expected or correct default behavior for udffsck.
As you are using mmap(), you should register handler for SIGBUS to catch unexpected errors like for SIGGEGV.
Also you should use sigaction() instead of signal().
Ad blocksize: On UDF disk it is stored in Logical Volume Descriptor in field logicalBlockSize. But LVD is not possible to parse without knowing blocksize, so after blocksize is detected, udffsck should check & verify that it matches with what is stored in LVID.logicalBlockSize. If it does not match then it should throw error and do not continue.
TODO
- [ ] check if d-string and d-characters values are valid (+ ability to fix it).
- [ ] Ad blocksize: On UDF disk it is stored in Logical Volume Descriptor in field logicalBlockSize. But LVD is not possible to parse without knowing blocksize, so after blocksize is detected, udffsck should check & verify that it matches with what is stored in LVID.logicalBlockSize. If it does not match then it should throw error and do not continue.
- [x] As you are using mmap(), you should register handler for SIGBUS to catch unexpected errors like for SIGGEGV. Also you should use sigaction() instead of signal().
DONE
- [x] Travis testing for x86 and x86_64
- [x] Overflow bugs at x86 platform and some other minor bugs
- [x] Output coloring switch was added
- [x] Ad O_EXCL: mkudffs now has this race-free implementation, see commit 6fb2b8f
- [x] Fixing extra tests (they were out for long time, found some issues after recent bugfixes)
@pali Please verify if this list is complete for finishing udffsck 1.0. I would love to finish that.
For 1.0 todo there should be also:
- documentation which disk is (or is not) udffsck able to repair... (e.g. from code I can see it does not support disks with Space Table type)
- handling different partition types... currently there is not support for Type 2 partitions (Sparable, VAT or Metadata partition), right? so code should check and do not touch such UDF disks
- check for supported partition access type, possible we do not want to run fsck on readonly or writeonce media access type... (maybe for future it would make sense to add some switch which ignore access type, in udflabel I added --force, see https://github.com/pali/udftools/blob/master/doc/udflabel.8#L63-L73)
And later fsck should be able to change Volume Set Identifier if its first 16 are not hexadecimal values, but in way that would not change generated UUID. Algorithm for UUID is described here: https://github.com/pali/udftools/blob/master/doc/udflabel.8#L129-L170
Btw, I'm not sure if autodetection of UDF blocksize is working correctly in all cases. When I was working on udfinfo, I needed to extend code more times and now it is huge.. https://github.com/pali/udftools/blob/master/udfinfo/readdisc.c#L76-L478 I will test some UDF images with different blocksizes if are correctly detected by current udffsck.
Anyway, now I'm finishing with changes to udftools and in few days I want to release a new version 2.0. There is a new tool udflabel for changing existing UDF label on disks up to UDF 2.60 which do not use VAT or Metadata 2.50 partition. And new tool udfinfo which show various information about UDF disks up to UDF 2.60 excluding Metadata 2.50 partition, useful for GUI tools which show label, uuid, free/used space, etc... Should I wait for finishing udffsck, or release later new udftools version when udffsck is ready?
Is there some fixed date when will be version 2.0 released? I'll try to catch that date, but I won't promise that. I have few relatively free days now, so I can work on this and it doesn't seems to be lot of work to finish.
Basically now I have prepared everything, including updates to manual pages. If I do not find any other bugs then udftools 2.0 is ready.
Mhm... I think it would be better to release new udftools and once fsck is ready then release new version again. In my opinion it is not a good idea to strictly define deadline. Deadline should be: once it is finally ready.
Another thing which fsck could do: instead erasing, put incomplete/broken files into /lost+found/
Another thing which fsck could do: instead erasing, put incomplete/broken files into /lost+found/
I wanted avoid that in version 1.0 because of writing back to file system. There is quite difficult logic for that and it would resulted in lot of work on TODO list. If you insist on that, fine, but I would rather keep it as it is for now.
Release TODO list
- [ ] check if d-string and d-characters values are valid (+ ability to fix it).
- [x] Ad blocksize: On UDF disk it is stored in Logical Volume Descriptor in field logicalBlockSize. But LVD is not possible to parse without knowing blocksize, so after blocksize is detected, udffsck should check & verify that it matches with what is stored in LVID.logicalBlockSize. If it does not match then it should throw error and do not continue. Update: Implemented here: 716b9a1, but in fact, it die earlier during AVDP loading, because when wrong blocksize is used, AVDPs are not loadable and therefore is not possible to check against LVD. But it effectively detects wrong value in LVD->blocksize.
- [ ] documentation which disk is (or is not) udffsck able to repair... (e.g. from code I can see it does not support disks with Space Table type)
- [ ] handling different partition types... currently there is not support for Type 2 partitions (Sparable, VAT or Metadata partition), right? so code should check and do not touch such UDF disks
Sparable partition is used by RW optical discs, you can generate empty images by mkudffs (media-type) -m cdrw or -m dvdrw and fill them by operating system (linux kernel and also windows can do it).
Virtual partition (VAT) is used by recordable discs, for generating you need mkudffs 2.0 and use media-type cdr, dvdr or bdr.
Metadata partition is used on any UDF 2.50 or UDF 2.60 revision which do not use VAT. mkudffs does not support it yet. But you can use e.g. Windows format.exe, Nero Burning ROM (via Image Recorder) or Mac OS X newfs_udf. The only open source implementation which can generate UDF 2.50 images is NetBSD's newfs_udf. I do not know any Linux implementation.
- [ ] check for supported partition access type, possible we do not want to run fsck on readonly or writeonce media access type... (maybe for future it would make sense to add some switch which ignore access type, in udflabel I added --force, see https://github.com/pali/udftools/blob/master/doc/udflabel.8#L63-L73)
- [ ] VSD at VRS size: https://github.com/pali/udftools/pull/7#pullrequestreview-85642832
- [ ] Ability to open, check and fix also block device which is mounted in read-only mode (this would need to extend exclusive open)
- [ ] Switch default mode to skip checking data (files/directories) when lvid integrity is closed -- just to prevent scanning whole clean disk on boot time
Sparable partition is used by RW optical discs, you can generate empty images by mkudffs (media-type) -m cdrw or -m dvdrw and fill them by operating system (linux kernel and also windows can do it).
Virtual partition (VAT) is used by recordable discs, for generating you need mkudffs 2.0 and use media-type cdr, dvdr or bdr.
Metadata partition is used on any UDF 2.50 or UDF 2.60 revision which do not use VAT. mkudffs does not support it yet. But you can use e.g. Windows format.exe, Nero Burning ROM (via Image Recorder) or Mac OS X newfs_udf. The only open source implementation which can generate UDF 2.50 images is NetBSD's newfs_udf. I do not know any Linux implementation.
Based on linux-fs-devel discussion another two things are needed for udffsck:
- Ability to open, check and fix also block device which is mounted in read-only mode (this would need to extend exclusive open)
- Switch default mode to skip checking data (files/directories) when lvid integrity is closed -- just to prevent scanning whole clean disk on boot time
@pali Dstring error detection algorithm was finished (a99362d), hopefully all tests will pass. If you are ok with this implementation, I'll implement correction algorithm.
current version of udffsck segfaults on image created by mkudffs from master branch:
$ ./mkudffs/mkudffs udf 89
$ ./udffsck/udffsck udf
[FATAL] Unexpected error (SEGV), please report it. More info at man page. Exiting.
Yes, but I am quite busy last few months. I hope I'll have some more time during May. Odesláno prostřednictvím aplikace BlackBerry Hub pro Android Od: [email protected]áno: 22. dubna 2018 15:55Komu: [email protected]ěď komu: [email protected]: [email protected]; [email protected]ředmět: Re: [pali/udftools] udffsck 1.00-beta (#7) Do you have any update or plan for fixing problems and finishing it?
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or mute the thread.
current version of udffsck segfaults on image created by mkudffs from master branch:
$ ./mkudffs/mkudffs udf 89 $ ./udffsck/udffsck udf [FATAL] Unexpected error (SEGV), please report it. More info at man page. Exiting.
I see what issue is there. Size of device is under spec requirements. I am thinking what is correct solution for this. From my point of view, I am unable to fix filesystem which is in conflict with spec, therefore I'll drop some message and die. I agree segfault is really bad tho :)
About TCC: is there some real reason why support this compiler? Upstream seems to be dead (https://bellard.org/tcc/)
I am asking because I am having some trouble with compiling cmocka sources with tcc (or better to say running resulting binary). I'll see if latest hotfix fixes that but I am not sure about that.