"Content download not correct size" for 0005000e10112200

[JohnVeness]

When trying to download this title (an update) I get the following output:

$ python FunKiiU.py -onlinetickets -title 0005000e10112200
*******
FunKiiU 2.2 by cearp and the cerea1killer
*******

Downloading/updating data from redacted
-Downloading titlekeys.json.
-File size is None.
-File in disk is 258382.
Downloaded data OK!
Starting work in: "install/0005000e10112200 _ Update"
Downloading TMD...
-Downloading install/0005000e10112200 _ Update/title.tmd.
-File size is None.
-File in disk is 0.

This is an update, so we are getting the legit ticket straight from Nintendo.
-Downloading install/0005000e10112200 _ Update/title.tik.
-File size is None.
-File in disk is 0.
Downloading Contents...
Total size is 176.31 MB

Downloading 1 of 18.
-Downloading install/0005000e10112200 _ Update/00000000.app.
-File size is 32768.
-File in disk is 0.
Download complete: 32.00 KB.15 KB of 32.00 KB, 50%                    
                                        
Downloading 2 of 18.
-Downloading install/0005000e10112200 _ Update/00000001.app.
-File size is 32769.
-File in disk is 0.
Content download not correct size of 32.00 KB, 50%                    

*Attempt 2 of 4
-Downloading install/0005000e10112200 _ Update/00000001.app.
-File size is 32769.
-File in disk is 32784.
Content download not correct size of 32.00 KB, 50%                    

*Attempt 3 of 4
-Downloading install/0005000e10112200 _ Update/00000001.app.
-File size is 32769.
-File in disk is 32784.
Content download not correct size of 32.00 KB, 50%                    

*Attempt 4 of 4
-Downloading install/0005000e10112200 _ Update/00000001.app.
-File size is 32769.
-File in disk is 32784.
Content download not correct size of 32.00 KB, 50%                    

ERROR: Could not download content file... Skipping title

It seems to be expecting a file of size 32769 but getting one of size 32784.

[JohnVeness]

Same with title ID 0005000e1010f500.

[JohnVeness]

I've just done: wget http://ccs.cdn.wup.shop.nintendo.net/ccs/download/0005000e10112200/00000001 and wget http://ccs.cdn.wup.shop.nintendo.net/ccs/download/0005000e1010f500/00000001 and in both cases the file is size 32784. In other words, that seems to truly be the file size on the CDN, so your code isn't downloading it wrongly. The problem seems to be that the TMD says it is 32769 which doesn't match, assuming your code is reading the TMD correctly. I haven't checked the TMD parsing separately.

So the question is, what is the correct thing to do in this case? You could truncate the file to match the size the TMD says (or only download the first X bytes based on what the TMD says). Or you could just not bother checking the file sizes, maybe as an optional switch?

[JohnVeness]

The file size mismatch problem has been discussed in https://gbatemp.net/threads/funkiiu-incompatibility-with-0005000e10102100-nintento-land-update.464552/ where it has been pointed out that the file size in the TMD is the file size of the decrypted content, not the encrypted content downloaded from the CDN. When decrypted, the file sizes are correct apparently, although I realise that FunKiiU doesn't do decryption so cannot verify that.

https://gbatemp.net/threads/funkiiu-verify-function-to-list-all-incompatible-game.464904/ has a modified version of your code with a nosizecheck parameter which would help with these awkward files.

Arguably, though, you shouldn't be using the TMD to verify sizes of downloaded encypted content at all, in other words, nosizecheck should be the default state, even though the size check does help detect incomplete downloads.

Tricky to know what to suggest, other than adding decryption as a verification step.

[llakssz]

Thanks, I never knew this problem. I'll see what we can do. But no decryption :)

[JohnVeness]

I said previously that a "no size check" mode should be the default, as it would be more "correct", based on the principal that the size in the TMD isn't actually what one thinks it is.

Thinking about this further, given that 99% of titles have sizes in the TMD which do match the encrypted file sizes, I now think it actually would be good to check the sizes (to detect partial downloads) by default.

The question then is what to do about the titles with incorrect sizes in the TMD.

Maybe you could detect this at runtime, e.g. if after N download attempts the file size is wrong (but consistently wrong to the same size each time), go into a "no size check" mode.

Alternatively you could have a hardcoded list of title IDs with incorrect sizes, and use that to go into the different mode. Or try and persuade the titkekey site to store a flag against the title ID in their database, exposed via the JSON.

It might be worth bearing in mind that presumably the incorrect TMDs may be fixed by Nintendo in future, at least for patches and DLC if not for base titles, so a "blacklisted" title ID might not stay like it forever.

[aphirst]

In light of the recent discontinuation of Wii U USB Helper, is there any likelihood that this issue could get re-explored?

It seems the downstream developers are aware of this issue too: https://github.com/dojafoja/FunKii-UI/issues/4

[V10lator]

You can get the correct size with code like this (inside of download_file) but that's quick&dirty / there might be a way without opening two connections to the NUS (but I also can't re-use the connection reading the content-length as that would destroy download resuming).

            infile = urlopen(url)
            cl = infile.headers['Content-Length']
            if cl == None:
                expected_size = None
            else:
                expected_size = int(cl)
            infile.close()