Downloading playlist as an archive ends up with corrupted zip file

[kli0rf]

Some files in music.zip file gets corrupted after download hence cannot be extracted. For exapmle, extracting files with 7zip gives me something like: "0 CRC failed in '08. Never Say.mp3'. File is broken." This is actual file, as you can see there's no non ascii chars in it's filename.

Running latest (i guess) CherryMusic Server 0.36.0 on Debian Jessie (8.3) x86.

[6arms1leg]

Hmm... never seen this before, but there are many similar reports about extraction errors with 7-Zip. Since CherryMusic uses the standard GNU/Linux tools as backend to create *.zip files, I doubt it's a bug in CherryMusic. Could you please copy such a *.zip file, that 7-Zip reports to be broken, back to your CherryMusic server and check if extraction works there under Debian? Please test the archive with: $ unzip -vt music.zip and $ gunzip -Vvt music.zip and $ bunzip2 -vt music.zip

I predict that all methods will report the archive to be OK (which would make this an issue with 7-Zip, not CherryMusic), but if they don't please post the output here so we can investigate further.

[kli0rf]

Here's the output:

kli0rf@studio:~/tmp$ unzip -vt music.zip
Archive:  music.zip
    testing: 01. It Isn't Meant To Be.mp3   OK
    testing: 02. Desire Be Desire Go.mp3   bad CRC da5f2336  (should be 25a0dcca)
    testing: 03. Alter Ego.mp3        OK
    testing: 04. Lucidity.mp3         OK
    testing: 05. Why Won't You Make Up Your Mind.mp3   OK
    testing: 06. Solitude Is Bliss.mp3   bad CRC e34a5bfd  (should be 1cb5a403)
    testing: 07. Jeremy's Storm.mp3   bad CRC a89f9ea7  (should be 57606159)
    testing: 08. Expectation.mp3      bad CRC 933cb972  (should be 6cc3468e)
    testing: 09. The Bold Arrow Of Time.mp3   OK
    testing: 10. Runway, Houses, City, Clouds.mp3   OK
    testing: 11. I Don't Really Mind.mp3   OK
At least one error was detected in music.zip.
kli0rf@studio:~/tmp$ gunzip -Vvt music.zip
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Jean-loup Gailly.

kli0rf@studio:~/tmp$ gunzip -vt music.zip
music.zip:      gzip: music.zip has more than one entry--rest ignored
 OK

kli0rf@studio:~/tmp$ bunzip2 -vt music.zip
  music.zip: bad magic number (file not created by bzip2)

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

EDIT: Formatted code properly. Please use Markdown next time. -- 6arms1leg

[kli0rf]

And here's another one:

kli0rf@studio:~/tmp/CD I$ ls -l
total 125072
-rw-r--r-- 1 kli0rf kli0rf 12874693 Apr  3 19:50 01. It Isn't Meant To Be.mp3
-rw-r--r-- 1 kli0rf kli0rf 10651148 Apr  3 19:50 02. Desire Be Desire Go.mp3
-rw-r--r-- 1 kli0rf kli0rf 11510034 Apr  3 19:50 03. Alter Ego.mp3
-rw-r--r-- 1 kli0rf kli0rf 10879958 Apr  3 19:50 04. Lucidity.mp3
-rw-r--r-- 1 kli0rf kli0rf  7978325 Apr  3 19:50 05. Why Won't You Make Up Your Mind.mp3
-rw-r--r-- 1 kli0rf kli0rf  9439062 Apr  3 19:50 06. Solitude Is Bliss.mp3
-rw-r--r-- 1 kli0rf kli0rf 13142175 Apr  3 19:50 07. Jeremy's Storm.mp3
-rw-r--r-- 1 kli0rf kli0rf 14520389 Apr  3 19:50 08. Expectation.mp3
-rw-r--r-- 1 kli0rf kli0rf 10603088 Apr  3 19:50 09. The Bold Arrow Of Time.mp3
-rw-r--r-- 1 kli0rf kli0rf 17407477 Apr  3 19:50 10. Runway, Houses, City, Clouds.mp3
-rw-r--r-- 1 kli0rf kli0rf  9040961 Apr  3 19:50 11. I Don't Really Mind.mp3

kli0rf@studio:~/tmp/CD I$ zip -v test.zip ./*
  adding: 01. It Isn't Meant To Be.mp3 .        (in=12874693) (out=12740592) (deflated 1%)
  adding: 02. Desire Be Desire Go.mp3 . (in=10651148) (out=10580343) (deflated 1%)
  adding: 03. Alter Ego.mp3 .   (in=11510034) (out=11461613) (deflated 0%)
  adding: 04. Lucidity.mp3 .    (in=10879958) (out=10807623) (deflated 1%)
  adding: 05. Why Won't You Make Up Your Mind.mp3       (in=7978325) (out=7906298) (deflated 1%)
  adding: 06. Solitude Is Bliss.mp3     (in=9439062) (out=9384092) (deflated 1%)
  adding: 07. Jeremy's Storm.mp3 .      (in=13142175) (out=12980682) (deflated 1%)
  adding: 08. Expectation.mp3 . (in=14520389) (out=14363629) (deflated 1%)
  adding: 09. The Bold Arrow Of Time.mp3 .      (in=10603088) (out=10448407) (deflated 1%)
  adding: 10. Runway, Houses, City, Clouds.mp3 .        (in=17407477) (out=17280542) (deflated 1%)
  adding: 11. I Don't Really Mind.mp3   (in=9040961) (out=8933725) (deflated 1%)
total bytes=128047310, compressed=126887546 -> 1% savings

kli0rf@studio:~/tmp/CD I$ unzip -vt test.zip
Archive:  test.zip
    testing: 01. It Isn't Meant To Be.mp3   OK
    testing: 02. Desire Be Desire Go.mp3   OK
    testing: 03. Alter Ego.mp3        OK
    testing: 04. Lucidity.mp3         OK
    testing: 05. Why Won't You Make Up Your Mind.mp3   OK
    testing: 06. Solitude Is Bliss.mp3   OK
    testing: 07. Jeremy's Storm.mp3   OK
    testing: 08. Expectation.mp3      OK
    testing: 09. The Bold Arrow Of Time.mp3   OK
    testing: 10. Runway, Houses, City, Clouds.mp3   OK
    testing: 11. I Don't Really Mind.mp3   OK
No errors detected in compressed data of test.zip.

[6arms1leg]

Well, then I was wrong and we can't blame 7-Zip for it. I see, manually creating *.zip files from the command line works fine. In your initial post you wrote

Some files in music.zip file gets corrupted [...]

Are they always the same files, that end up corrupted? Let's say you would do the same procedure again, starting with downloading the exact same playlist in CherryMusic etc.... Do the same files end up corrupted?

[kli0rf]

Yup, always the same files. I just tried the same procedure on another CherryMusic instance running on another Jessie with the same content...

[6arms1leg]

What's the command line output from CherryMusic when you download a playlist? Otherwise, I guess I'm out of ideas. CherryMusic uses ZipStream to create the archives and I remember we had an issue with that in the past (see #405), which sparked three new Issues/Pull requests over at their GitHub project page (see the references at the end of the page in #405). But this issue doesn't seem to be related so far...

[kli0rf]

The only output is:

[160410-16:25] 127.0.0.1 - - [10/Apr/2016:16:25:22] "POST /api/heartbeat HTTP/1.1" 200 14 "http://cherry.domain.tld/" "Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101 Firefox/45.0"
[160410-16:25] 127.0.0.1 - - [10/Apr/2016:16:25:46] "POST /api/downloadcheck HTTP/1.1" 200 14 "http://cherry.domain.tld/" "Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101 Firefox/45.0"
[160410-16:25] 127.0.0.1 - - [10/Apr/2016:16:25:46] "POST /download HTTP/1.1" 200 - "http://cherry.domain.tld/" "Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101 Firefox/45.0"
[160410-16:25] 127.0.0.1 - - [10/Apr/2016:16:25:52] "POST /api/heartbeat HTTP/1.1" 200 14 "http://cherry.domain.tld/" "Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101 Firefox/45.0"

I didn't see anything like "-v" flag in the help page, to make the output more verbose. I don't know, i guess i can provide you an access to my cherrymusic instance if that would help.

[devsnd]

Hey @kli0rf,

Thanks for the report. CherryMusic uses a patched python 2 module to create the zip files. It might as well contain a bug, as it already did in the past!

Can you please send me the file in question, so that I can create a testcase to recreate this error? thanks!

[kli0rf]

Hi @devsnd,

do you mean a problematical zip file? If so, here's a file on my debian box (since it's too large to attach it to this conversation).

[kli0rf]

Hi @devsnd,

i've removed the file from my server already. Hope you've successfully downloaded it. Are there any updates on this case?

[devsnd]

I tried to reproduce it but I was not successful... It seems there is still a subtle bug in the zipstream module.

also, do you use python 2 or 3? I'm using python 3 and never have experienced any of those issues.

BTW: I love that album.

[kli0rf]

I use python 2.7.9... So that's why probably i'm getting this error. Ok, let me try to install python 3. I will let You know.

[kli0rf]

Yup, You're absolutely right. With python3 everything's ok. Thank You! :)