UsbFile delete() corrupts drive

[cumbie]

I noticed that there are some similar closed issues (Issue #5, Issue #9), but in my tests, I am still able to consistently corrupt the USB Drive after calling UsbFile.delete(). FatFile and FatDirectory are the subclasses that I'm using when calling delete(). I'm using the recent 0.5.5 libaums build, and to corrupt the drive, I simply do the following:

1. Copy a test file or folder to the root of the USB drive.
2. Insert drive in Android phone, accept permission.
3. Use UsbFile.search() to get the test file or folder.
4. Call UsbFile.delete(), which afterwards from the phone's file browser does show the file deleted.
5. Remove from phone and insert into my Windows PC
6. I immediately get the dialog: "You need to format the disk in drive E: before you can use it"

When it is in this state, the drive behaves unpredictably when I try to use it in the phone. And the odd behaviors occur if I format or not, implying that a PC format doesn't necessarily fix drive corruption. Specific odd-behavior include the drive not mounting after accepting the permission request and also the libaums Partition.FileSystem.FreeSpace property returning -32768 (my ChunkSize is 32768, not sure if that is a coincidence here).

To uncorrupt the drive, I will have to do one or more of the following:

1. format (PC or phone)
2. do a checkdisk on PC and fix any found issues
3. open the drive in the Android Settings->Storage page, or File Browse on the Android file browser.
4. if my app can mount the drive, doing some libaums file I/O calls sometimes uncorrupts the drive.

Though any one of these methods don't seem to work consistently every time, so I have to just try a few each time before the drive seems back to normal. I've confirmed this issue exists on a Samsung Galaxy J7 (OS 7) and a Samsung Galaxy S5 (OS 5).

[cumbie]

Here is a stack trace of the error when deleting a folder:

java.lang.OutOfMemoryError: Failed to allocate a 83070912 byte allocation with 16773096 free bytes and 58MB until OOM
	at java.util.Arrays.copyOf(Arrays.java:3231)
	at java.util.Arrays.copyOf(Arrays.java:3204)
	at java.util.ArrayList.grow(ArrayList.java:249)
	at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:223)
	at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:215)
	at java.util.ArrayList.add(ArrayList.java:441)
	at com.github.mjdev.libaums.fs.fat32.FAT.getChain(FAT.java:127)
	at com.github.mjdev.libaums.fs.fat32.ClusterChain.<init>(ClusterChain.java:64)
	at com.github.mjdev.libaums.fs.fat32.FatDirectory.init(FatDirectory.java:156)
	at com.github.mjdev.libaums.fs.fat32.FatDirectory.delete(FatDirectory.java:599)

[magnusja]

Does that mean this error while deleting occurs always and afterward the drive is corrupted?

The error is interesting. It seems that it somehow oversees the end of the cluster chain, but I am currently not sure why. What is pretty odd is that it seems to crash before any drive happens to the device, so I am not sure why the file should be corrupted afterward.

What would be amazing if you could upload an image from the device and then give some code how I can reproduce this error.

[cumbie]

Oh no, sorry I should have been more clear on that. My drive can get corrupt after a delete() even without this error. Sometimes I get this error after deleting. I guess a corrupt drive could encompass many things (1 or more files corrupt, 1 or many bad sectors, etc... I'm not too familiar with low level drive details). Maybe a certain type of corruption triggers this error?

But it seems to be if I delete(), then move the drive to my PC, I will see the "you need to format the drive" popup, pretty much every time.

[magnusja]

But it gets corrupted after the crash for sure? Are you doing some write operations after catching that exception or is the app just crashing? Because there is no write operation before this crash. Does this crash happen with a clean USB drive, meaning you did not do any write operations with libaums prior to that crash? Does the drive still work with libaums if you get the format message? Which version of Windows are you using? Can you check if Linux or macOS can still mount the drive?

[cumbie]

Yes it gets corrupted after the crash.

There are no write operations before or after with the test that I'm doing. My test simply calls delete() on the root drive to remove the test file(s)/folder(s). And in my app if I do just libaums reads or writes I don't see the format message when moving the drive to the PC. It's something with delete() that causes the corruption. Actually if I delete() then do some reads/writes, the drive doesn't always get corrupted, as if read/write operations possibly "fix" the corruption somehow. And for read/writes I am using the libausm UsbFileOutputStream and UsbFileInputStream.

I just tested on the following platforms with each having the same problem: Windows (7, 10), Mac OS X (El Capitan, Mavericks).

Also I made sure the drive was clean in each case, with the same problem. In my case "clean" means, formatting on my PC, copied files to delete to the drive on my PC, inserted into my phone, ran tests with no libaums read or writes, and saw same issue.

And yes, libaums works after corruption if I reattach the drive to the phone, but only sometimes. And this is when I sometime see the FileSystem.FreeSpace property returning -32768. Other failed times, it just won't let me connect to the drive with libaums after accepting the permission request.

[magnusja]

All right I see. Is it possible that you upload an image and give me a specified sequence of actions I need to execute, which will reproduce this error with a high probability?

[cumbie]

Let me know if this works for you to test. I didn't get any images cause I think this way makes more sense. I added a method to your UsbFileman test app that will delete files on the root. If you call this method and nothing else, and then move the drive to a PC/Mac, etc, you should see that the drive is corrupt. Let me know if you have any questions.

Update app::MainActivity.java:

1. add this method:

private void deleteRootFiles() {
        AlertDialog.Builder builder = new AlertDialog.Builder(this);
        builder.setTitle("Delete Results");

        builder.setPositiveButton("Ok", new DialogInterface.OnClickListener() {
            @Override
            public void onClick(DialogInterface dialog, int whichButton) {
                dialog.dismiss();
            }

        });
        builder.setCancelable(false);

        try {
            if (currentFs == null) {
                throw new Exception("No USB device detected... please insert it.");
            }

            UsbFile root = currentFs.getRootDirectory();
            UsbFile[] files = root.listFiles();
            String msg = "Files to delete ("+ String.valueOf(files.length) +"):\r\n";

            for (UsbFile file : files) {
                msg += " - '"+ file.getName() +"'\r\n";
                file.delete();
            }
            msg += "SUCCESS!";

            builder.setMessage(msg);
        }
        catch (Exception x) {
            builder.setMessage(x.getMessage());
        }
        finally {
            builder.create().show();
        }
    }

2. To call, I just wired it to the Options menu "Run Tests" button (edit the case in MainActivity.onOptionsItemSelected):

public boolean onOptionsItemSelected(MenuItem item) {
 ...
case R.id.run_tests:
  //startActivity(new Intent(this, LibAumsTest.class));
  deleteRootFiles();
  return true;

3. Rebuild Usbfileman and deploy to your phone/tablet

Prepare a Clean a USB Drive

Note: I used Windows 7, so just do the equivalents on whatever OS you use.

1. Insert USB drive into PC and format (I used allocation size of 32KB and FAT32).
2. After format completes, right-click drive and select Properties
3. Select Tools tab
4. Click "Check now" to run a check disk on the drive
5. On Check Disk popup, select both checkboxes "Automatically fix file system errors" and "Scan for and attempt recovery of bad sectors"
6. If errors were found, repeat steps 1-5, until no errors exist
7. Copy a file to the drive's root.
8. Right-click drive and select "Eject" to properly eject

Test with the App to Corrupt Drive

1. Start the updated UsbFileman
2. Insert USB drive
3. Accept all permission requests
4. Select options menu, and tap "Run Tests"
5. If successful, a popup will say so.
6. Remove drive
7. Insert back into PC, and the drive should be corrupt (get the format message and/or drive is inacccesible)

[magnusja]

I only have a mac and can only do this via command line.

NEWFS_MSDOS(8)            BSD System Manager's Manual           NEWFS_MSDOS(8)

NAME
     newfs_msdos -- construct a new MS-DOS (FAT) file system

SYNOPSIS
     newfs_msdos [-N] [-B boot] [-F FAT-type] [-I volid] [-O OEM] [-S sector-size] [-a FAT-size] [-b block-size] [-c cluster-size] [-e dirents] [-f format] [-h heads] [-i info] [-k backup] [-m media] [-n FATs] [-o hidden] [-r reserved]
                 [-s total] [-u track-size] [-v volume-name] special [disktype]

DESCRIPTION
SYNOPSIS
     newfs_msdos [-N] [-B boot] [-F FAT-type] [-I volid] [-O OEM] [-S sector-size] [-a FAT-size] [-b block-size] [-c cluster-size] [-e dirents] [-f format] [-h heads] [-i info] [-k backup] [-m media] [-n FATs] [-o hidden] [-r reserved]
                 [-s total] [-u track-size] [-v volume-name] special [disktype]

DESCRIPTION
     The newfs_msdos utility creates a FAT12, FAT16, or FAT32 file system on device special, using disktab(5) entry disktype to determine geometry, if required.

     The options are as follows:

     -N      Don't create a file system: just print out parameters.

     -B boot
             Get bootstrap from file.

     -F FAT-type
             FAT type (one of 12, 16, or 32).

     -I volid
             Volume ID.

     -O OEM  OEM string (up to 8 characters).  The default is "BSD  4.4" (with two spaces).

     -S sector-size
             Number of bytes per sector.  Acceptable values are powers of 2 in the range 128 through 32768.

     -a FAT-size
             Number of sectors per FAT.

     -b block-size
             File system block size (bytes per cluster).  This should resolve to an acceptable number of sectors per cluster (see below).

     -c cluster-size
             Sectors per cluster.  Acceptable values are powers of 2 in the range 1 through 128.

What do you mean exactly with allocation size? bytes/sector or sectors/cluster or a combination?

Interesting is that the manpage says I can specify up to 32768 bytes for sector size, but when I execute the command it says 4096 is the maximum. https://github.com/st3fan/osx-10.9/blob/master/msdosfs-198/newfs_msdos.tproj/newfs_msdos.c#L90

EDIT

Seems to work?

sudo newfs_msdos -F 32 -b 32768 -v "TEST VOLUME" /dev/disk3
newfs_msdos: warning: /dev/disk3 is not a character device
512 bytes per physical sector
/dev/disk3: 15724736 sectors in 245699 FAT32 clusters (32768 bytes/cluster)
bps=512 spc=64 res=32 nft=2 mid=0xf0 spt=32 hds=255 hid=0 drv=0x00 bsec=15728640 bspf=1920 rdcl=2 infs=1 bkbs=6

[magnusja]

Okay, so I played around a little and I, unfortunately, cannot reproduce this. I am using El Capitan and I just sometimes get a LOST.DIR folder when I plug it into the mac again (might be an indicator for some kind of failure?). Do you get that only with 32K block size? Maybe you can give me an image from a corrupted device and I try to analyze that.

[cumbie]

I think your format routine results are similar my steps in Windows, so that should be fine to get the drive "clean". You could also use the Disk Utility app on your Mac (goto Finder->Go->Utilities). In there, highlight your drive and choose Erase to format it. Select MS-DOS (FAT) as the format type and Master Boot Record as the scheme. There doesn't seem to be extra options for specifying cluster size, etc.

The LOST.DIR folder is not a problem. That is something Android creates on the drive automatically for some internal drive management.

I have been working with 32K block size only because it was the default that the Windows format chose. Let me try with some other sizes and see if that makes a difference.

So did you test with the deleteRootFiles() method that I provided?

Also, I'll go ahead and corrupt a drive and get you that image shortly...

[cumbie]

Here's an image of a corrupt 16GB USB drive. I zipped it down to a ~60MB file. Here's a link to it on my dropbox:

https://www.dropbox.com/sh/135j6ewubswlz4k/AADIjbnfwySLxdYoxuIjOBAWa?dl=0

Let me know if you have problems downloading it.

So before drive corruption, I had 1 text file on it at the root when I called delete().

[magnusja]

All right, I wrote the image to a drive and I cannot mount it with my macOS. In addition, I get this libaums exception:

09-26 07:25:19.626 22512-22512/com.github.mjdev.usbfileman E/MainActivity: error setting up device
                                                                           com.github.mjdev.libaums.partition.PartitionTableFactory$UnsupportedPartitionTableException
                                                                               at com.github.mjdev.libaums.partition.PartitionTableFactory.createPartitionTable(PartitionTableFactory.java:76)
                                                                               at com.github.mjdev.libaums.UsbMassStorageDevice.setupDevice(UsbMassStorageDevice.java:233)
                                                                               at com.github.mjdev.libaums.UsbMassStorageDevice.init(UsbMassStorageDevice.java:192)
                                                                               at com.github.mjdev.libaums.usbfileman.MainActivity.setupDevice(MainActivity.java:799)
                                                                               at com.github.mjdev.libaums.usbfileman.MainActivity.access$000(MainActivity.java:99)
                                                                               at com.github.mjdev.libaums.usbfileman.MainActivity$2.onReceive(MainActivity.java:166)
                                                                               at android.app.LoadedApk$ReceiverDispatcher$Args.run(LoadedApk.java:885)
                                                                               at android.os.Handler.handleCallback(Handler.java:739)
                                                                               at android.os.Handler.dispatchMessage(Handler.java:95)
                                                                               at android.os.Looper.loop(Looper.java:234)
                                                                               at android.app.ActivityThread.main(ActivityThread.java:5526)
                                                                               at java.lang.reflect.Method.invoke(Native Method)
                                                                               at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
                                                                               at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)

Same for you?

[cumbie]

When I write the image to a drive, I can't mount it on my Mac or Windows. And it's not recognized by my app. But I have not seen that particular exception. Are you able to repeat and get that same exception? And do you have any ideas yet on how delete() may be causing the corruption?

[magnusja]

Hey there, sorry for the late response, I am quite busy these days. I noticed you used some tool called PassMark to create the image. Are you using that to write the image to disk again? Can you maybe create an image using dd? That is the tool I use, which just creates a file with only the contents of the disk. I am not sure what PassMark exactly does, but it looks like it uses some file format, that's why the boot sector is already invalid I think.

EDIT: Instructions on using dd: https://github.com/magnusja/libaums/issues/86#issuecomment-312145961

[cumbie]

Not a problem. Yes I used the PassMark app to image, it was just something I googled, so not a problem to recreate one with dd. I did so at the command line, following the details at the website you referenced, using a block size of 64k (bs=64k). Here it is zipped and uploaded to my dropbox:

https://www.dropbox.com/s/zi01ujxvkpvce5g/bad_usb.dd.zip?dl=0

Also, I noticed the behavior of the corrupt drive is a little different on my Mac versus Windows. Windows will immediately popup saying the drive is corrupt and must be formatted. Whereas the Mac will just not auto-mount it when inserting the USB. And it seems like when I mount it on the Mac, the drive becomes accessible and no longer corrupt.

Hopefully you can find some clues with this image. Let me know if you need anything else.

[Islam-Darwish]

don't delete "System Volume Information" folder from root directory

[magnusja]

Fixed with v0.9.4