Broken .tar file after unpacking

[clhunsen]

I have run the following commands to backup my Android N device (7.1.1, N4F26T) and to check the files in the backup using ABE (a checkmark ✔ after any command indicates exit code 0):

[ /media/data/backups/files ] ✔
$ adb backup '-all -apk -shared -nosystem'
[ /media/data/backups/files ] ✔
$ head -n4 backup.ab
ANDROID BACKUP
4
1
AES-256
[ /media/data/backups/files ] ✔
$ java -jar ../android-backup-extractor/build/libs/abe-all.jar unpack backup.ab backup.tar
This backup is encrypted, please provide the password
Password: 
Calculated MK checksum (use UTF-8: true): [hash]
2381763584 bytes written to backup.tar.
[ /media/data/backups/files ] ✔
$ 

So far, so good. Now, the following command should give me a list of all files in the backup (.tar) file, but unfortunately, after a long list files, it gives me an error:

$ tar -tf backup.tar
apps/cc.dict.dictcc/_manifest
apps/cc.dict.dictcc/a/base.apk
[...]
shared/0/WhatsApp/Media/WhatsApp Video/VID-20161224-WA0001.mp4
shared/0/WhatsApp/Media/WhatsApp Video/VID-20161209-WA0001.mp4
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Am I doing anything the wrong way? Or is this a bug in ABE when unpacking the .ab file?

[SquallATF]

new android version backup struct changed, try backup with -nocompress

[clhunsen]

try backup with -nocompress

Thank you, that solved my problem. At least, for the moment.

As ABE should be able to handle compressed backup files of newer formats, I leave this issue open.

[nelenkov]

Any idea what the exact change is? This seems to only happen when -shared is specified.

[superbonaci]

I've found similar problem, is the same problem or is not?

Sony Xperia Z5 Compact fails to adb backup shared storage

[nelenkov]

Does -nocompress help? If so, might be related.

[egalanos]

I've also got this exact problem. Both star and tar are getting the same error.

My phone is now dead so can't generate a new .ab. Any ideas on how I can get abe to work?

[egalanos]

My phone worked again for a few hours today. I created a new encrypted (is forced on) backup using -nocompress.

The abe unpack command completed without error. The tarball is still broken.

[superbonaci]

@egalanos what is the error of the tar?

[egalanos]

% ls -l adb-nocompress-unpacked.tar 
-rw-rw-r--. 1 user user 4440329216 Jul  9 14:33 adb-nocompress-unpacked.tar

tar:

% tar tf adb-nocompress-unpacked.tar
<file list>
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

star:

star tvf adb-nocompress-unpacked.tar -debug
<file list>
star: Tar file too small (amount: 0 bytes).
star: Unexpected EOF on input.
star: Cannot recover from error - exiting.
star: 433625 blocks + 9216 bytes (total of 4440329216 bytes = 4336259.00k).

[nelenkov]

Make a backup without excluding external storage (-sdcard), it seems the error happens during unpacking the SD card data.

[superbonaci]

@egalanos file a bug here: https://issuetracker.google.com/

Similar issues: https://issuetracker.google.com/issues/36987260 https://issuetracker.google.com/issues/38290747 https://issuetracker.google.com/issues/37150449 https://issuetracker.google.com/issues/37130390 https://issuetracker.google.com/issues/37115971 https://issuetracker.google.com/issues/37103304 https://forum.xda-developers.com/showpost.php?p=20965332&postcount=25 https://issuetracker.google.com/issues/37096097

Just do a search on android bugs.

[gsgrippal]

@clhunsen Probably it's an issue with your phone. Try running adb logcat while backup and watch for letter capital E.

• adb logcat -s BackupManagerService

Open another console and backup

• adb backup -apk -shared -all -system

[M0rtale]

same error here... The unpacked .tar file is damaged.

[gsgrippal]

@M0rtale android has unfixed bugs

[clhunsen]

@clhunsen Probably it's an issue with your phone. Try running adb logcat while backup and watch for letter capital E.

• adb logcat -s BackupManagerService

Thanks for that hint, @gsgrippal! I found that the Amazon Prime Video app threw some errors. After resetting it (and removing all stored local data for it), the backup ran flawlessly.

[gsgrippal]

@clhunsen glad that you found the culprit. Thanks also for the heads up, as you can see adb backup still need a lot of improvement to be user friendly.

[mjakubicek]

Any chance to recover a backup that produces a tar which fails like this (I cannot make a new one)?

tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now

Even just a partial recovery of a particular file (I know the paths) would be very helpful.

[nelenkov]

Getting the files using a binary editor, or try using a scripting language to try the read specifics files could work. Assuming the files you need were backed up correctly.

[gsgrippal]

@mjakubicek what is the size of your tar file?

[hungerburg]

Seeing tar: Unexpected EOF in archive I tried adb backup -nocompress … - Now the tar of the backup contains shared stuff too, not just apps. But trying to extract still reports errors.

The first tar error happens after a file that is more than 2GB in size. Maybe that gives a pointer to the problem source? The list of files in the tar from the compressed backup ends with this very same file, up to there, both lists are fully identic. Option -i with tar does not help.

2124660736 2020-02-24 16:04 apps/…/bigFile
tar: Skipping to next header
503800 2020-03-09 21:12 shared/0/DCIM/…

UPDATE - adb shell ls bigFile also reports 212…, and that is still less than 2^31 ~~Note: native android Files app reports the size of bigFile as 2.12GB, while tar shows it as 1.98GB (212… above)~~

Note2: As the shared files are only a fifth of the full ab backup, the difference in ab file size maybe due to compression alone?

[gsgrippal]

@hungerburg don't include shared/* on any adb backup, it's broken and will always be. Use -noshared to make adb backup. You can backup your internal sdcard by copying everything to USB OTG. Using MTP may cause some files to be skipped and not be properly copied to your computer. Alternatively user FTP client on phone and server on your computer. Include all hidden files.

[mjakubicek]

@mjakubicek what is the size of your tar file?

5.2 GB (yes, with -shared, I didn't know that would be so broken)

[hungerburg]

@gsgrippal I made a comparison: find . on the MTP mount produces the exact same listing as a adb shell find /sdcard0. The tar extracted from ab file indeed is missing some files from that: namely everything below /Android and curiously also /Movies. The rest though looks fine.

Looking further, some of the stuff below MTP/Android is in TAR/apps. So likely it is only the /Movies I should take extra care (I don't :) Additionally, a single command for restore is nice, and modification dates on pictures are nice too (MTP kills those).

Remains the question: what is "tar tf" complaining about?

[gsgrippal]

@gsgrippal I made a comparison: find . on the MTP mount produces the exact same listing as a adb shell find /sdcard0. The tar extracted from ab file indeed is missing some files from that: namely everything below /Android and curiously also /Movies. The rest though looks fine.

Looking further, some of the stuff below MTP/Android is in TAR/apps. So likely it is only the /Movies I should take extra care (I don't :) Additionally, a single command for restore is nice, and modification dates on pictures are nice too (MTP kills those).

Remains the question: what is "tar tf" complaining about?

adb unpack what shows? As nelenkov said before, your adb backup was truncated at creation time, then tar is corrupt and truncated as well.

Each time you make some adb backup, test the tar with tar tf. If fails, repeat until works, or eventually use another method.

Ii's not that difficult to understand that android has bugs, in this case is unrelated to android-backup-extractor.

[egandrew]

Thank you for the hard work and open source tools. I have a rooted pixel3a on Android 10 and the tar from 'abe-all.jar' (20181012025725-d750899). tar gives no output. winrar say tar is invalid. I haven't figured out good error/trace info. Will try some of the techniques above for watching logs during the backup and see if anything comes out there. I am only backing up one app that has allow-backups true. ( ANDROID BACKUP 5 0 none ). Maybe it's just empty?

[nelenkov]

Apps can forbid backup, so in that case behavior is somewhat undefined. How big are the .ab and .tar files? Try looking at contents with editor/cat to get some idea.

[hungerburg]

@gsgrippal […]

Each time you make some adb backup, test the tar with tar tf. If fails, repeat until works, or eventually use another method.

An adb backup done yesterday: The tar received from the extractor tested without error. If I fully understand, the tar is created by adb or even on device, and abe only decrypts the stream from the .ab file.

This is not the forum for android bugs, still of note probably, just for reference:

The tar error was reported immediately following a (very close to) 2GB size file below /apps/, see above "tar: Skipping to next header"; that file meanwhile shrunk considerably; no more error but some more files in /apps get listed.

[radscience]

I got the same error. Using

bsdtar

instead gave decompression without error.

[gsgrippal]

I got the same error. Using

bsdtar

instead gave decompression without error.

That doesn't even mean anything. Were the files there? It could just exit without message even if file is damaged.

[Niko-sk2x]

Is there any way to restore such broken backup? I had no idea it could be broken and this is the only backup I have before unlocking bootloader and installing lineageos. Restoring it as is doesn't really restore anything, despite taking significant amount of time, and trying to unpack it and pack it again, it "completes" almost immediately and does nothing.