webMAN-MOD
webMAN-MOD copied to clipboard
aldostools
[slaunch] Move slaunch icons into subdirectory
Hi,
would it be possible to move the location for icons/thumbnails/covers for slaunch into a subdirectory, e.g. wmtmp/icon (as named in the slist), wmtmp/covers or wmtmp/art?
This would make deleting all of them at once through through FTP or webMAN's Delete Cache files functionality in XMB much easier in order to re-download after edits.
Regards, Darjan
webMAN MOD searches for the covers in multiple paths as it is explained in the wiki: https://github.com/aldostools/webMAN-MOD/wiki/Game-Paths-&-Covers
One of the paths searched is /dev_hdd0/tmp/wmtmp
e.g. /dev_hdd0/tmp/wmtmp/BLES12345.JPG
This path has the lowest priority. That means that the covers are searched first in the other paths.
If you clear the temporary folder, all the covers will be removed as you suggested.
The path with highest priority is /dev_hdd0/game/BLES80608/USRDIR/covers. This path is used by multiMAN, IRISMAN, webman, sman and other backup managers.
I think I wasn't very clear with my issue.
I was going at Movie DVD/BD covers that are put next to ISOs on the ps3netsrv, which webMAN MOD downloads all into the /dev_hdd0/tmp/wmtmp directory that could be put in a further subdirectory so they can be deleted all at once, leaving last_games.bin, slist.bin, slaunch.cfg intact if one chooses so.
Deleting of covers on the PS3 to trigger re-downloading them would then be possible through XMB easily:
I even resorted to writing my own C program to generate the slist.bin with the covers/icons referenced in the subdirectory and copy slist.bin + covers over with FTP to manage them.
On another note: Thanks for the effort keeping this project up! And just a heads up, in the upcoming days I'll come at you with some pull requests with new features and performance improvements for ps3netsrv on Linux and slaunch.
You can try this test build: webftp_server.zip Extract the zip and copy webftp_server.sprx to /dev_hdd0/plugins Put your covers in /dev_hdd0/tmp/wmtmp/covers, reboot and re-scan the content.
It searches for covers (title id or iso name) in /dev_hdd0/tmp/wmtmp/covers NOTE: The folder has to be created manually to enable the search in that folder.
I'll check it out over the weekeend!
While you're at it working on the webftp_server.sprx managing downloading .jpg/.png files, would you add preliminary support for downloading custom simple uncompressed covers with an .argb extension? I'll have a pull request for slaunch with the feature and a png2argb converter tool coming in by Monday.
That extension is very uncommon, so it would be unuseful for most users. I could consider adding it later if it is a demanded feature.
Can talk about it in the pull request, you might like it along with other improvements.
So, forgot another issue which was why I resorted to managing covers manually (also because 1100+ DVDs/BDs), at some point webMAN MOD tried to access cover files that don't exist (happened again with this new webftp_server.sprx):
ps3netsrv stdout:
[0] Connection from 192.168.178.109
open dir /PS3ISO
open dir /BDISO
open dir /DVDISO
file error: opening "/media/RAID/ps3netsrv/DVDISO/<non_existent_movie_cover>.jpg"
open error on "/DVDISO/<non_existent_movie_cover>.jpg" (viso=0)
When this happens, the search stops and XMB restarts (with Refresh webMan Games & XMB).
So, kinda tough to test. Could reduce the number of ISOs though, will try next.
Could you check if wMM is trying to copy a remote cover containing invalid characters like *, ?, , ", |, > or <
How can I check that? Enabling debug printf's in ps3netsrv?
Also I don't use many non-alphanumerical characters in filenames, only '.', '-', '#' and '%', which wMM usually manages fine.
In ps3netsrv the name displayed in "file error: opening"
For the error (viso=0) it looks like it's trying to open the .jpg as a folder, I will try to investigate.
I'll try a way to reproduce this with a clean ps3netsrv directory, do you use Linux? Could write a little bash script that generates a lot of dummy ISOs, and covers for most of them.
Ok, then bash should work.
Unpack and run the generate_dummy_dvdiso.sh script, which creates a DVDISO dir with 1000 dummy ISOs and every 10th with a missing cover.
Deleted wmtmp and getting this stdout/stderr log on Refresh webMan Games & XMB:
$ ps3netsrv ps3netsrv_debug
ps3netsrv build 20200708 (mod by aldostools)
Path: ps3netsrv_debug
Host IP #1: 192.168.178.24:38008
Waiting for client...
[0] Connection from 192.168.178.109
open dir /DVDISO
file error: opening "ps3netsrv_debug/DVDISO/dummy_1.jpg"
open error on "/DVDISO/dummy_1.jpg" (viso=0)
Hope this helps.
Thanks. The file name doesn't contain invalid characters.
Currently I'm checking the code to validate that the file exists before try to open it.
This is a new test build that should fix the issue trying to mount non-existing remote files webftp_server.zip
My ps3netsrv segfaults with that one soon after
open dir /DVDISO
used the 1k dummy DVDISO directory for testing.
Stacktrace with -g build, null pointer dereference of client->dirpath:
$ gdb -ex start --args ps3netsrv_20200708/ps3netsrv/ps3netsrv ps3netsrv_debug
[...]
Reading symbols from ps3netsrv_20200708/ps3netsrv/ps3netsrv...
Temporary breakpoint 1 at 0x404d6a: file src/main.cpp, line 1660.
Starting program: /media/2x16TB_RAID0/ps3netsrv_20200708/ps3netsrv/ps3netsrv ps3netsrv_debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Temporary breakpoint 1, main (argc=2, argv=0x7fffffffd578) at src/main.cpp:1660
1660 {
(gdb) c
Continuing.
ps3netsrv build 20200708 (mod by aldostools)
Path: ps3netsrv_debug
Host IP #1: 192.168.178.24:38008
Waiting for client...
[0] Connection from 192.168.178.109
[New Thread 0x7ffff7cf7640 (LWP 27312)]
open dir /DVDISO
Thread 2 "ps3netsrv" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7cf7640 (LWP 27312)]
0x0000000000465925 in __strlen_avx2 ()
(gdb) bt
#0 0x0000000000465925 in __strlen_avx2 ()
#1 0x0000000000403863 in process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1258
#2 0x0000000000404d0d in client_thread (arg=0x5036e0 <clients>) at src/main.cpp:1629
#3 0x0000000000417b99 in start_thread ()
#4 0x0000000000478b73 in clone ()
(gdb) f 1
#1 0x0000000000403863 in process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1258
1258 char *path = (char*)malloc(MAX_PATH_LEN + root_len + strlen(client->dirpath + root_len) + MAX_FILE_LEN + 2);
(gdb) info locals
items = 0
result = {
dir_size = 0
}
dir_entries = 0x7ffff0009520
path = <optimized out>
st = {
file_size = 6566335474611867209,
mtime = 5641090987094996321,
ctime = 1643970633,
atime = 1644010356,
mode = 33188
}
entry = <optimized out>
d_name_len = <optimized out>
dirpath_len = <optimized out>
slen = <optimized out>
ini_file = <optimized out>
p = <optimized out>
fd = <optimized out>
(gdb) p client->dirpath
$1 = 0x0
@darjankrijan Thank you for the crash report. Could you test using an older version of ps3netsrv?
Maybe the crash is specific to that linux build. The main change in the latest build of wMM is that now performs a remote stat before try to open or copy a file. The change should not affect the listing of remote directories.
I tested the Windows release of build 20200708 and it listed 3075 items without any crash:
BTW a new build of wMM was released today fixing an issue related PSN connection. The fix should be unrelated to this issue with ps3netsrv.
How far back should I go? Can say ps3netsrv_20200529 also crashes.
Edit: I'll just debug this, find a fix and pull request.
Edit2: It seems like that stat structure is uninitialized from the large numbers, will start there.
Did you try using the docker version by @shawly ? Follow the readme at https://github.com/shawly/docker-ps3netsrv
I'm using Win10 and I am not getting that error. So this issue can be difficult to debug.
Does this crash happen only in the directory with thousands of files or in all directories?
Nah I refuse to use any of that docker stuff, as said in the edits I'll just fix this myself hopefully tonight.
It could be a memory allocation error in this line: https://github.com/aldostools/webMAN-MOD/blob/master/Projects/ps3netsrv/src/main.cpp#L1255
I'll make an update and upload here so you can test it.
Will try, just dumping what I got so far:
$ gdb -ex start --args ps3netsrv_20200708/ps3netsrv/ps3netsrv ps3netsrv_debug
[...]
Reading symbols from ps3netsrv_20200708/ps3netsrv/ps3netsrv...
Temporary breakpoint 1 at 0x404d6a: file src/main.cpp, line 1660.
Starting program: /media/2x16TB_RAID0/ps3netsrv_20200708/ps3netsrv/ps3netsrv ps3netsrv_debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Temporary breakpoint 1, main (argc=2, argv=0x7fffffffcbd8) at src/main.cpp:1660
1660 {
(gdb) break main.cpp:1255
Breakpoint 2 at 0x40382b: file src/main.cpp, line 1255.
(gdb) c
Continuing.
ps3netsrv build 20200708 (mod by aldostools)
Path: ps3netsrv_debug
Host IP #1: 192.168.178.24:38008
Waiting for client...
[0] Connection from 192.168.178.109
[New Thread 0x7ffff7cf7640 (LWP 85939)]
open dir /DVDISO
[Switching to Thread 0x7ffff7cf7640 (LWP 85939)]
Thread 2 "ps3netsrv" hit Breakpoint 2, process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1255
1255 netiso_read_dir_result_data *dir_entries = (netiso_read_dir_result_data *) malloc(sizeof(netiso_read_dir_result_data) * MAX_ENTRIES);
(gdb) n
1256 memset(dir_entries, 0, sizeof(netiso_read_dir_result_data) * MAX_ENTRIES);
(gdb) n
1258 char *path = (char*)malloc(MAX_PATH_LEN + root_len + strlen(client->dirpath + root_len) + MAX_FILE_LEN + 2);
(gdb) n
1260 if ((!client->dir) || (!client->dirpath) || (!dir_entries) || (!path))
(gdb) p client->dirpath
$1 = (client_t *) 0x5036e0 <clients>
(gdb) p *client
$2 = {
s = 4,
ro_file = 0x0,
wo_file = 0x0,
dir = 0x7ffff00011c0,
dirpath = 0x7ffff0000d70 "ps3netsrv_debug/DVDISO/",
buf = 0x7ffff7cf8010 "",
connected = 1,
ip_addr = {
s_addr = 1840425152
},
thread = 140737350956608,
CD_SECTOR_SIZE = 2352
}
(gdb) c
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 2, process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1255
1255 netiso_read_dir_result_data *dir_entries = (netiso_read_dir_result_data *) malloc(sizeof(netiso_read_dir_result_data) * MAX_ENTRIES);
(gdb) p *client
$3 = {
s = 4,
ro_file = 0x0,
wo_file = 0x0,
dir = 0x0,
dirpath = 0x0,
buf = 0x7ffff7cf8010 "",
connected = 1,
ip_addr = {
s_addr = 1840425152
},
thread = 140737350956608,
CD_SECTOR_SIZE = 2352
}
dir and dirpath seem to get set to 0 somehow on the 2nd time this function enters.
Same thing:
$ gdb -ex start --args ./ps3netsrv_20220206_bugfix/ps3netsrv ps3netsrv_debug
[...]
Reading symbols from ./ps3netsrv_20220206_bugfix/ps3netsrv...
Temporary breakpoint 1 at 0x404d98: file src/main.cpp, line 1668.
Starting program: /media/2x16TB_RAID0/ps3netsrv_20220206_bugfix/ps3netsrv ps3netsrv_debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Temporary breakpoint 1, main (argc=2, argv=0x7fffffffcbd8) at src/main.cpp:1668
1668 {
(gdb) break main.cpp:1264
Breakpoint 2 at 0x403860: file src/main.cpp, line 1264.
(gdb) c
Continuing.
ps3netsrv build 20220206 (mod by aldostools)
Path: ps3netsrv_debug
Host IP #1: 192.168.178.24:38008
Waiting for client...
[0] Connection from 192.168.178.109
[New Thread 0x7ffff7cf7640 (LWP 87675)]
open dir /DVDISO
[Switching to Thread 0x7ffff7cf7640 (LWP 87675)]
Thread 2 "ps3netsrv" hit Breakpoint 2, process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1264
1264 char *path = (char*)malloc(MAX_PATH_LEN + root_len + strlen(client->dirpath + root_len) + MAX_FILE_LEN + 2);
(gdb) p *client
$1 = {
s = 4,
ro_file = 0x0,
wo_file = 0x0,
dir = 0x7ffff00011c0,
dirpath = 0x7ffff0000d70 "ps3netsrv_debug/DVDISO/",
buf = 0x7ffff7cf8010 "",
connected = 1,
ip_addr = {
s_addr = 1840425152
},
thread = 140737350956608,
CD_SECTOR_SIZE = 2352
}
(gdb) c
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 2, process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1264
1264 char *path = (char*)malloc(MAX_PATH_LEN + root_len + strlen(client->dirpath + root_len) + MAX_FILE_LEN + 2);
(gdb) p *client
$2 = {
s = 4,
ro_file = 0x0,
wo_file = 0x0,
dir = 0x0,
dirpath = 0x0,
buf = 0x7ffff7cf8010 "",
connected = 1,
ip_addr = {
s_addr = 1840425152
},
thread = 140737350956608,
CD_SECTOR_SIZE = 2352
}
(gdb) n
Thread 2 "ps3netsrv" received signal SIGSEGV, Segmentation fault.
0x0000000000465955 in __strlen_avx2 ()
In this code the variables (!client->dir) || (!client->dirpath) are checked before malloc of path.
ps3netsrv_20200206_src3.zip
I don't think it has to do with malloc on that line.
$ gdb -ex start --args ./ps3netsrv_20220206_bugfix/ps3netsrv ps3netsrv_debug
[...]
Reading symbols from ./ps3netsrv_20220206_bugfix/ps3netsrv...
Temporary breakpoint 1 at 0x404d98: file src/main.cpp, line 1668.
Starting program: /media/2x16TB_RAID0/ps3netsrv_20220206_bugfix/ps3netsrv ps3netsrv_debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Temporary breakpoint 1, main (argc=2, argv=0x7fffffffcbd8) at src/main.cpp:1668
1668 {
(gdb) break main.cpp:1264
Breakpoint 2 at 0x403860: file src/main.cpp, line 1264.
(gdb) break main.cpp:171
Breakpoint 3 at 0x40214a: file src/main.cpp, line 171.
(gdb) break main.cpp:212
Breakpoint 4 at 0x402386: file src/main.cpp, line 212.
(gdb) break main.cpp:1045
Breakpoint 5 at 0x402b34: file src/main.cpp, line 1045.
(gdb) break main.cpp:1130
Breakpoint 6 at 0x404436: file src/main.cpp, line 1130.
(gdb) break main.cpp:1130
Note: breakpoint 6 also set at pc 0x404436.
Breakpoint 7 at 0x404436: file src/main.cpp, line 1130.
(gdb) break main.cpp:1158
Breakpoint 8 at 0x40447f: file src/main.cpp, line 1158.
(gdb) break main.cpp:1264
Note: breakpoint 2 also set at pc 0x403860.
Breakpoint 9 at 0x403860: file src/main.cpp, line 1264.
(gdb) c
Continuing.
ps3netsrv build 20220206 (mod by aldostools)
Path: ps3netsrv_debug
Host IP #1: 192.168.178.24:38008
Waiting for client...
[0] Connection from 192.168.178.109
Breakpoint 3, initialize_client (client=client@entry=0x5036e0 <clients>) at src/main.cpp:172
172 client->connected = 1;
(gdb) c
Continuing.
[New Thread 0x7ffff7cf7640 (LWP 88501)]
[Switching to Thread 0x7ffff7cf7640 (LWP 88501)]
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb) c
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb) c
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb) c
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb)
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb)
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb)
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb)
Continuing.
open dir /DVDISO
Thread 2 "ps3netsrv" hit Breakpoint 2, process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1264
1264 char *path = (char*)malloc(MAX_PATH_LEN + root_len + strlen(client->dirpath + root_len) + MAX_FILE_LEN + 2);
(gdb) p *client
$1 = {
s = 4,
ro_file = 0x0,
wo_file = 0x0,
dir = 0x7ffff00011c0,
dirpath = 0x7ffff0000d70 "ps3netsrv_debug/DVDISO/",
buf = 0x7ffff7cf8010 "",
connected = 1,
ip_addr = {
s_addr = 1840425152
},
thread = 140737350956608,
CD_SECTOR_SIZE = 2352
}
(gdb) c
Continuing.
Thread 2 "ps3netsrv" hit Breakpoint 5, process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
1045 client->dirpath = NULL;
(gdb) bt
#0 process_open_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1045
#1 0x0000000000404d2b in client_thread (arg=0x5036e0 <clients>) at src/main.cpp:1633
#2 0x0000000000417bc9 in start_thread ()
#3 0x0000000000478ba3 in clone ()
(gdb) n
1047 normalize_path(dirpath, true);
(gdb)
1048 client->dir = opendir(dirpath);
(gdb)
1049 if(!client->dir)
(gdb)
1052 result.open_result = BE32(NONE);
(gdb) p dirpath
$2 = 0x7ffff0000f90 "ps3netsrv_debug/DVDISO [auto]"
(gdb) n
1065 if(!client->dirpath)
(gdb)
1067 free(dirpath);
(gdb)
1070 ret = send(client->s, (char *)&result, sizeof(result), 0);
(gdb)
1071 if(ret != sizeof(result))
(gdb)
client_thread (arg=0x5036e0 <clients>) at src/main.cpp:1657
1657 if(ret != SUCCEEDED)
(gdb)
1584 ret = recv_all(client->s, (void *)&cmd, sizeof(cmd));
(gdb)
1585 if(ret != sizeof(cmd))
(gdb)
1590 switch (BE16(cmd.opcode))
(gdb)
77 return ret;
(gdb)
1637 ret = process_read_dir_cmd(client, (netiso_read_dir_entry_cmd *)&cmd);
(gdb)
Thread 2 "ps3netsrv" hit Breakpoint 2, process_read_dir_cmd (client=client@entry=0x5036e0 <clients>, cmd=cmd@entry=0x7ffff7cf71b0) at src/main.cpp:1264
1264 char *path = (char*)malloc(MAX_PATH_LEN + root_len + strlen(client->dirpath + root_len) + MAX_FILE_LEN + 2);
(gdb)
Thread 2 "ps3netsrv" received signal SIGSEGV, Segmentation fault.
0x0000000000465955 in __strlen_avx2 ()
process_open_dir_cmd sets client->dirpath to NULL here between two process_read_dir_cmd calls.
In this code the variables (!client->dir) || (!client->dirpath) are checked before malloc of
path. ps3netsrv_20200206_src3.zip
This one seems to at least not crash now, but there are weird g, ng, o, so files in the slist now, including that non-existent DVDISO [auto] directory I also encountered in the debug dump above (should be 1100 dummy entries but they stop at entry 518/1504):
** deleted previous zip **
I added a check to verify that the remote folder exists before try to open it. It should avoid the issue with the non-existing "DVDISO [auto]"
EDIT: Here is new build of both programs with additional check that remote folders exist. ps3netsrv_20220206_src4.zip
EDIT2: Found an issue mounting remote folders. Reviewing this issue.
EDIT3: Released a fix for the issue mounting remote folders. https://github.com/aldostools/webMAN-MOD/releases
Better now, it still stops after 504 entries from the DVDISO directory, think that also was an issue before the patches. Also doesn't download any covers, with and without the /dev_hdd0/tmp/wmtmp/covers directory.
Would be great if that could be resorted as well. Here an ls of the DVDISO directory and the slist.bin:
ps3netsrv_debug_dvdiso_slist.zip
Seems like some entries are left out randomly every 10~20 entries as well.
The number of items is limited by the VSH memory allocated in /setup.ps3 You can increase the number of items selecting the option 3072KB [MC] = 4-bg and selecting the Plugin memory usage: Max (1280KB).
The entries left out could be the order of the files read from the OS.
I'm glad that this build works better. However I have to fix the mount of remote folder that got broken with one the recent changes. The build from Dec 28 mount fine though.
EDIT: The issue mounting remote folders was fixed. https://github.com/aldostools/webMAN-MOD/releases