OSX Error When Dumping Patch - Could not extract file "".

[coryshaw1]

Before I begin I must note I had to change a few things in the script to support OSX.

1. Replace all instances of mkdir --parents with just mkdir -p
2. dd does not have a --version flag on OSX so I just updated L870 to dd --version 2> /dev/null | grep GNU > /dev/null

And finally for the error. I ran in debug mode to get more logging:

./ftpdump 192.168.1.250:2121 -p -r --debug

\e[95mDEBUG: \e[35mNon-GNU dd detected - using regular dd options (slow!).\e[39m
\e[95mDEBUG: \e[35mUsing current directory as output directory.\e[39m
\e[95mDEBUG: \e[35mOptions: dump_app=, dump_patch=1, dump_dlc=, dump_keystone=, dump_sflash=, dump_appdb=, dump_path=, decrypt=1, debug=1, debug_pfs=, curl_options=, curl_verbose=, wget_options=, wget_verbose=, pfs_extraction_enabled=\e[39m
\e[95mDEBUG: \e[35mUsing FTP server "ftp://192.168.1.250:2121".\e[39m
\e[95mDEBUG: \e[35mChecking FTP connection ...\e[39m
\e[95mDEBUG: \e[35mGoldHEN 2.x FTP server detected.\e[39m
\e[95mDEBUG: \e[35mServer-side SELF decryption is enabled.\e[39m
\e[95mDEBUG: \e[35mChecking if patch data exists.\e[39m
Dumping CUSA02084 patch data (17048 MB):
\e[95mDEBUG: \e[35mDumping directory "/mnt/sandbox/pfsmnt/CUSA02084-patch0/", cutting 4 directories.\e[39m
eboot.bin
libfmod.prx
...
sce_sys/trophy/
Preparing extraction of patch.pkg ...
\e[95mDEBUG: \e[35mDumping 48 bytes from patch.pkg to get body size.\e[39m
\e[95mDEBUG: \e[35mBody offset: 8192\e[39m
\e[95mDEBUG: \e[35mBody size: 21946368\e[39m
\e[95mDEBUG: \e[35mDumping body from patch.pkg (21954560 bytes).\e[39m
\e[95mDEBUG: \e[35mID: 0xffffffffe6697f70\e[39m
\e[95mDEBUG: \e[35mFilename offset: 1691452777\e[39m
\e[95mDEBUG: \e[35mData offset: -610120600\e[39m
\e[95mDEBUG: \e[35mSize: 1671949259\e[39m
Extracting
Could not extract file "".
Error encountered, script aborted.

I'm guessing something with the partial file download is failing on OSX using dd.

Any ideas here?

Editing with an example from attempting to dump a base game with app.pkg:

./ftpdump 192.168.1.250:2121 -r --debug

\e[95mDEBUG: \e[35mNon-GNU dd detected - using regular dd options (slow!).\e[39m
\e[95mDEBUG: \e[35mUsing current directory as output directory.\e[39m
\e[95mDEBUG: \e[35mNo dump options specified - enabling app, patch, and DLC dumping.\e[39m
\e[95mDEBUG: \e[35mOptions: dump_app=1, dump_patch=1, dump_dlc=1, dump_keystone=, dump_sflash=, dump_appdb=, dump_path=, decrypt=1, debug=1, debug_pfs=, curl_options=, curl_verbose=, wget_options=, wget_verbose=, pfs_extraction_enabled=\e[39m
\e[95mDEBUG: \e[35mUsing FTP server "ftp://192.168.1.250:2121".\e[39m
\e[95mDEBUG: \e[35mChecking FTP connection ...\e[39m
\e[95mDEBUG: \e[35mGoldHEN 2.x FTP server detected.\e[39m
\e[95mDEBUG: \e[35mServer-side SELF decryption is enabled.\e[39m
Dumping CUSA02084 app data (6438 MB):
\e[95mDEBUG: \e[35mDumping directory "/mnt/sandbox/pfsmnt/CUSA02084-app0/", cutting 4 directories.\e[39m
eboot.bin
main_ps4.hdr
main_ps4_0.ark
...
sce_sys/trophy/
Preparing extraction of app.pkg ...
\e[95mDEBUG: \e[35mDumping 48 bytes from app.pkg to get body size.\e[39m
\e[95mDEBUG: \e[35mBody offset: 8192\e[39m
\e[95mDEBUG: \e[35mBody size: -1646592\e[39m
\e[95mDEBUG: \e[35mDumping body from app.pkg (-1638400 bytes).\e[39m
"/user/app/CUSA02084/app.pkg": cURL reported error 1 (see https://curl.se/libcurl/c/libcurl-errors.html).
Error encountered, script aborted.

[hippie68]

Thanks for reporting this. I will add the changes to the script.

It seems the offsets are totally wrong (they are negative). Either MacOS dd behaves differently than expected, or the PKG is unusual. I neither have a Mac nor that specific game, so I can't test this.
You could try changing the MacOS dd options from "ibs=1" to "bs=1" to see if it helps (just a wild guess, as apparently it makes no difference on GNU/Linux dd). To make sure neither the PKG is special nor the PKG reading function needs to change, you could try running the script on Linux or try to install GNU dd.
You can comment the if blocks at line 1072/1107 to skip right to the PKG extraction part.

[coryshaw1]

I did some more research, and read that brew install coreutils actually prepends all of the the GNU commands with a g. So dd would become gdd. I’ll try some more stuff around that to see if that’s the cause. I’ll get a pull request up for all around OSX support including my other changes if I can get it working.

[coryshaw1]

So I did some more testing. gdd did nothing more than fix the dd --version error. I set up WSL on my VM and it worked flawlessly. Something is definitely strange on OSX. Calculating negative offsets is very strange. Not sure what to do to fix OSX issues though.

[hippie68]

Maybe the Bash version is too old, missing or misinterpreting/ignoring some features that are used. I saw there's a brew Bash, too. perhaps it is more up-to-date and would fix the issues.

[coryshaw1]

That actually did fix the offset issues! And it fixed the Debug output colors. I didn't get any errors when dumping the app with -r, I'm going to try and redump everything altogether. Dumping the patch I am getting CRITICAL: Could not find unencrypted version of "trophy00.trp". Not sure if that will break anything.

[hippie68]

Dumping the patch I am getting CRITICAL: Could not find unencrypted version of "trophy00.trp". Not sure if that will break anything.

That should not have happened. It might not be related, but please be aware with GoldHEN 2.0's FTP server (at least pre-2.0b2, not sure about the current version), there is a chance resuming might fail for decrypted files. The updated FTP server payload at https://github.com/hippie68/ps4-ftp should resume properly.

I have tried to add your changes and Bash version checking to the current script, which is attached here: ftpdump.zip Please let me know if it works. I did not change anything related to trophies, so if there is a bug it needs to be fixed separately.

[NikolausC]

Hey, I've run into these same issues on macOS using the latest release of ftpdump. To fix the Could not extract file "" issue, the only modification I made to the script was to change the bash path on the first line to point to the newer version I installed via brew. This fixed that error, and now I get the same trophy error:

DEBUG: ID: 0x1400
DEBUG: File name offset: 161
DEBUG: Data offset: 754672
DEBUG: Size: 7639552
DEBUG: File name: "trophy/trophy00.trp"
Extracting trophy/trophy00.trp
Replacing npbind.dat and nptitle.dat ...
npbind.dat
nptitle.dat
DEBUG: Trophy directories found in npbind.dat:
Checking trophy files for encryption ...
File "trophy00.trp" is encrypted.
CRITICAL: Could not find unencrypted version of "trophy00.trp".
Done.
1 critical message(s).

[NikolausC]

Ah, it seems the problem is that readarray isn't included on macOS, that's why there are no directories on this line: DEBUG: Trophy directories found in npbind.dat:

Seeing if there's anything I can do.

[hippie68]

readarray is a Bash built-in, which should be available in Bash 4+. The Homebrew Bash is probably 5+. As the script did not output an error message while trying to read npbind.dat, I think the file is there but was not downloaded correctly. Perhaps it has a size of 0 bytes or is incomplete?

[NikolausC]

I've been playing around with this since then and while I don't yet have a solution, I do have some more info.

The npbind.dat which is dumped here

if [[ -f npbind.dat ]]; then
  echo "Replacing npbind.dat and nptitle.dat ..."
  dump_file "$root/system_data/priv/appmeta/$1/npbind.dat"
  dump_file "$root/system_data/priv/appmeta/$1/nptitle.dat"
  replace_encrypted_trophies
fi

is not the same npbind.dat which is meant to be grepped in replace_encrypted_trophies. I went to "$root/system_data/priv/appmeta/$1/npbind.dat" and downloaded the file myself to confirm. The files are the same size, though. I'm not totally familiar with this but I think the dumped one may still be encrypted.

I experimented with the npbind.dat that I manually pulled from the machine, and I can clearly see the NPWR... string that should be getting grepped out. On a Mac, this is the command I was able to use with the good npbind.dat cat npbind.dat | strings | grep -E NPWR\[0-9\]{5}_\[0-9\]{2} to extract the trophy directory.

readarray definitely isn't present even in Bash 5 on macOS, but even if it were, it seems it's failing before getting there.

[NikolausC]

Okay I see part of the problem now. On the the first download, it downloads the correct decrypted versions, but when resuming it looks like it's overwriting it with the encrypted one.

[NikolausC]

Opened #14 to address the issue extracting trophies. Think the resume issue is beyond my abilities.

[NikolausC]

Opened #15 to address the issue involving npbind.dat when resuming downloads.

[hippie68]

Thanks, I think #15 is the true bug and makes #14 obsolete: If Bash does not understand readarray or any other command, it outputs an error message. But in your log there is no error message. So an explanation would be the Homebrew Bash version you used at that point executes readarray, but the script failed because npbind.dat had not been replaced.

Please run the updated attached script to confirm. You can comment line 597, "dump_dir ..." in function "dump_app_data()", to skip right to where it matters. ~ftpdump.zip~

Current version: ftpdump.zip

[hippie68]

The previously attached file was not working correctly. I have replaced it with a new version.

[NikolausC]

Sorry about the red herring with readarray on macOS: as you say it is present in bash. I was only checking for it with 'which'.

Will be a day or so before I'm able to test and confirm against a real PS4. However, I've got a decrypted npbind.dat so I've tested the problematic grep line on that. Turns out it was due to another outdated version shipped in macOS by default (2.6.0). brew install grep can fix it, but also required updating the 'grep' call to 'ggrep'.

[hippie68]

The options that are used in that grep line (-aoE) are listed here as supported: https://www.unix.com/man-page/osx/1/grep/. I don't know how accurate that page is though.

[NikolausC]

I've been able to test this now (I used the script on main branch) and unfortunately it still isn't able to find the trophy directories to copy the trophies, despite having the unencrypted npbind.dat. Installing the newer version of grep via brew, and updating the ftpdump script to use ggrep (installed by brew) fixes the problem.

bash-5.1$ ggrep -aoE 'NPWR[0-9]{5}_[0-9]{2}' npbind.dat
NPWR07862_00
NPWR07892_00


bash-5.1$ grep -aoE 'NPWR[0-9]{5}_[0-9]{2}' npbind.dat


bash-5.1$ ggrep --version
ggrep (GNU grep) 3.7
Packaged by Homebrew
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others; see
<https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.


bash-5.1$ grep --version
grep (BSD grep, GNU compatible) 2.6.0-FreeBSD

[hippie68]

This code tries to make the script use ggrep and gdd, if available, on macOS:

# Check for gdd and ggrep on macOS
if [[ $(uname) == Darwin ]]; then
  if gdd --version &> /dev/null; then
    [[ $debug ]] && debug_message "gdd detected - using gdd instead of dd."
    alias dd=gdd
  fi
  if ggrep --version &> /dev/null; then
    [[ $debug ]] && debug_message "ggrep detected - using ggrep instead of grep."
    alias grep=ggrep
  elif [[ $(grep --version) != *GNU* ]]; then
    warning_message "On macOS, GNU grep (ggrep) may be required to automatically replace encrypted trophies (\"\$ brew install grep\")."
  fi
  shopt -s expand_aliases
fi

If anyone who has a Mac (I don't) wants to test the code: it can be pasted to line 1455 (right after the Bash version check). Unless there are objections, I am going to include it in an upcoming script update.

[hippie68]

For completeness: the code was added on Apr 15, 2023 and there have been no complaints since then. I assume everything works well now for macOS.