m4b-tool Just the last 10 files are used when merging an Audiobook with 42 files

Hello Andreas, First of all thanks a lot for this wonderfull application. its helping me alot :).

My scenario: I am merging 42 files to an m4b audiobook, but m4b-tool is just using the last 10 files. The weird thing, if I put the last 10 files to another folder and try it again, all 33 Files are merged. Don't really understand what is going on here. Regarding the files there is no difference in coding and tags, as they are all ripped with the same coding configuration and I have put the same tagging information to all files.

System: I am using debian buster in a wsl 2 environment on Windows 10 Education 2004, docker for windows in wsl2 and the docker container build with the provided code here on the github

I have attached some pictures.

These are the files, that are merged Screenshot 2020-12-05 132042 with this command: FOLDER=Ausgefressen m4b-tool merge ./"$FOLDER/" --jobs 4 --use-filenames-as-chapters --no-chapter-reindexing --no-conversion --force --output-file="$FOLDER.m4b" (the absolute path is a folder in my home directory, if you need it tell me)

Result respectively generated chapters.txt file Screenshot 2020-12-05 131947 This is the second scenario, where all 33 files in the folder are merged: Screenshot 2020-12-05 133710 Same command as above: m4b-tool merge ./"$FOLDER/" --jobs 4 --use-filenames-as-chapters --no-chapter-reindexing --no-conversion --force --output-file="$FOLDER.m4b" Result: Screenshot 2020-12-05 133911

Maybe you can help? If you need anything else tell me I will try to provide any helping information.

Update: Just tried another file. Same Results. But here the Chapters go from 50 to 73.

Dec 05 '20 12:12 dmuiX

Thanks for reporting so detailed... Which version are you using? If you are using 0.4.2 you could try the latest pre-release.

Currently I'm working on a complete rewrite of the chapter logic, so this also might fix this issue.

Dec 05 '20 16:12 sandreas

Can you try with --debug? Also, technically, the command with variable should be like: ./"$FOLDER"/ and --output-file="$FOLDER".m4b, though I doubt that would cause this specific of an issue.

Dec 07 '20 07:12 djdembeck

Hey there. I'll close this ticket in about 3 days on no further feedback. Feel free to reopen if the problem persists.

Dec 12 '20 19:12 sandreas

I'm having this same issue, using the latest commit in master I built my own docker image, and some books it only grabs the last 10 files, and I even have one that it only grabs one file and it's not even the first/last it's just a random file in the middle... The book has 66 files total and it only converts Chapter 51.

I used --debug and I don't see anything useful in the output that gives a clue as to why it's only grabbing one file...

Command used: docker run -it --rm -v D:/Workareas/Workarea-Books/!AudioConversion:/mnt incryptosoft/m4b-tool merge -v --debug --jobs=4 --output-file="output/" --use-filenames-as-chapters --batch-pattern="input/%a/%s/%p - %n/" --batch-pattern="input/%a/%n/" "input/"

MP3 Tags: I wiped all tags from the MP3 files and only have the following set...

ALBUM = Children of Captain Grand
ARTIST = Jules Verne
YEAR = 1868
TRACK = <01-66>
TITLE = Chapter <01-66>

UPDATE: I ran another book that has 266 files in it, it merged 225 of them and skipped the rest... very odd behavior, it doesn't seem to be a certain number that its skipping, I've gone through and deleted the id3 tags from the files and re-written them from scratch, tried regenerating the file names to make sure there wasn't something weird with an invalid character. I'm at a loss.

Log Output:

Dec 29 '20 05:12 nerd-king

Sorry I have just forgotten completely about my report. I think I have solved it by using the latest pre-release version. Well actually I am not completely sure as I have used your tool on other audiobooks than the one I have reported, but there everything went well. If I have time I will try it again on the reported audiobook and report again

@incryptosoft A solution could be to use the latest pre-release version: docker build . --build-arg M4B_TOOL_DOWNLOAD_LINK=https://github.com/sandreas/m4b-tool/files/5683619/m4b-tool.tar.gz -t m4b-tool This was the command I used.

Dec 31 '20 15:12 dmuiX

@dmuiX Thanks for giving this advice... like I already said: It has been a long time since the last stable release and I try to finish the next release as soon as possible, but unfortunately I'm very busy atm.

BTW: Others reported that the latest pre-release did NOT FIX it, but this needs careful investigation, which may take some time.

Dec 31 '20 18:12 sandreas

Thanks a lot for your comment sandreas. Yeah this was my thought, too there has not been an release for a long time :D. No Problem.

Interesting. I will try it again and see what happens. For the last 3 Audiobooks I have created everything went well with chapters...

Another solution for anybody who is looking for creating audiobooks might be to use this application. audiobookcreator Unfortunately its just for Windows PC. Don't know if its working in Wine.

@sandreas if you don't want that kind of advertisement, i can understand that, you can say it and i delete it. But i thought it could be helpful for people coming here ;). This was my solution for the problems I had the first time.

Dec 31 '20 21:12 dmuiX

Additional Information:

I messed around with various combinations trying to determine what the issue might be. What I came up with is that any combination that I do where the input folder is inside the folder that was mounted (e.g. my command contains -v /d/convert:/mnt) it fails to find all of the files. If I first copy the input folder to another location INSIDE the container it works fine (with and without patterns)...

So I start the docker container and change the entrypoint to /bin/sh, copy /mnt/input to /tmp/input, run my conversion which finds all files and converts perfectly fine (with /mnt/output/ as the output folder), delete /tmp/input/ and exit, and it's perfect.

I am going to test this on a linux box to see if it's something specific to reading the files from a mounted folder on windows or on any distro.

I am running the latest pre-release that is posted.

Jan 01 '21 12:01 nerd-king

Don't know if it helps you, but I just write what am I using at the moment:

For me everything is working fine with this command in my .zshrc on a Mac OSX: alias m4b-tool='docker run -it --rm -u $(id -u):$(id -g) -v "$(pwd)":/mnt m4b-tool'

I put every m4a file written like Kapitel 01.m4a, Kapitel 02.m4a, ... in a folder and then create the audiobook with this command: m4b-tool merge "./" --jobs 4 --use-filenames-as-chapters --no-chapter-reindexing -vvvv --no-conversion --output-file "Somebook.m4b"

Haven't tried it on Windows though...last time it was not working for me, but there I haven't used the pre-release. So could be a Windows Problem. Good Luck on Linux :)

Jan 01 '21 17:01 dmuiX

On my maschine the problem is solved with the use of the current pre-release docker version. :) Have tried converting the audiobook from my original post and it is created correctly.

Jan 14 '21 08:01 dmuiX

I'm still seeing this issue actually. Running a docker build on windows using the latest master commit: https://github.com/sandreas/m4b-tool/commit/1c33feb3600415ceced69da42891812196b436a3

Trying to combine 102 M4B files and it's only grabbing the last ~68 of them.

docker run -it --rm -v `"D:\Media\iTunes\iTunes Media\Audiobooks\Cornelia Funke`":/mnt m4b-tool merge . --no-conversion --output-file="Dragon Rider.m4b" --use-filenames-as-chapters --name="Dragon Rider")

Side note, the ability to use the --merge-similar in the merge operation would be handy for this particular book, but when I try to run it as a part of the chapters operation, I'm getting this: Command:

docker run -it --rm -v `"D:\Media\iTunes\iTunes Media\Audiobooks\Cornelia Funke`":/mnt m4b-tool chapters --merge-similar ".\Dragon Rider.m4b")

Result:

PHP Fatal error:  Uncaught TypeError: Argument 3 passed to M4bTool\Chapter\ChapterMarker::guessChaptersBySilences() must be an instance of Sandreas\Time\TimeUnit, null given, called in phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Command/ChaptersCommand.php on line 177 and defined in phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Chapter/ChapterMarker.php:33
Stack trace:
#0 phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Command/ChaptersCommand.php(177): M4bTool\Chapter\ChapterMarker->guessChaptersBySilences(Array, Array, NULL)
#1 phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Command/ChaptersCommand.php(128): M4bTool\Command\ChaptersCommand->buildChapters(Array)
#2 phar:///usr/local/bin/m4b-tool/vendor/symfony/console/Command/Command.php(255): M4bTool\Command\ChaptersCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 phar:///usr/local/bin/m4b-tool/vendor/symfony/console/Application.php(908): Symfony\Component\Console\Command\Command-> in phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Chapter/ChapterMarker.php on line 33
an error occured, that has not been caught:
Array
(
    [type] => 1
    [message] => Uncaught TypeError: Argument 3 passed to M4bTool\Chapter\ChapterMarker::guessChaptersBySilences() must be an instance of Sandreas\Time\TimeUnit, null given, called in phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Command/ChaptersCommand.php on line 177 and defined in phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Chapter/ChapterMarker.php:33
Stack trace:
#0 phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Command/ChaptersCommand.php(177): M4bTool\Chapter\ChapterMarker->guessChaptersBySilences(Array, Array, NULL)
#1 phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Command/ChaptersCommand.php(128): M4bTool\Command\ChaptersCommand->buildChapters(Array)
#2 phar:///usr/local/bin/m4b-tool/vendor/symfony/console/Command/Command.php(255): M4bTool\Command\ChaptersCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 phar:///usr/local/bin/m4b-tool/vendor/symfony/console/Application.php(908): Symfony\Component\Console\Command\Command->
    [file] => phar:///usr/local/bin/m4b-tool/src/library/M4bTool/Chapter/ChapterMarker.php
    [line] => 33
)

Feb 02 '21 05:02 Yukigamine

Ok, since this seems to be a problem, I'll reopen the issue and try to investigate further... although I don't think that it will be easy to reproduce / fix, since this seems to be very random.

Feb 02 '21 09:02 sandreas

Appreciate it! Let me know if I can help in any way. I'm not super skilled in PHP, but I'm good at testing stuff and retrieving logs!

Feb 02 '21 16:02 Yukigamine

Ok, I improved the logging with the latest pre-release. Would be awesome, if someone could try to reproduce the problem with --debug and provide the logfile.

Jun 04 '21 11:06 sandreas

Thanks @incryptosoft for the workaround idea. Worked great.

@sandreas attached a m4b-tool.log. Skips the first 35 (out of 90) files in my case. Running on docker (through Docker for Windows). Moving the files to the container proper (copied from the volume) worked as noted above.

docker run -it --rm -v C:\...\workdir:/mnt m4b-tool merge --skip-cover --jobs=16 -vv --debug . -o Malice.m4b

m4b-tool.log

Jul 20 '21 05:07 cridenour

I have the exact same behavior with my files as well. Can run some logs if you want me to.

Dec 19 '21 16:12 RemonVanNieuwenhuizen

Yeah logs would be good to have, although I don't believe that I can do something about it. What we could try is to find a generic solution without mounting the device via docker, but via SMB, SSHFS or any other fs layer

Dec 19 '21 17:12 sandreas

I was having the same issue on windows. I managed to fix it by turning off "Use the WSL 2 based engine". Since doing that its working perfectly

Mar 10 '22 11:03 kaldigo

I was having the same issue on windows. I managed to fix it by turning off "Use the WSL 2 based engine". Since doing that its working perfectly

Great find. Thank you for reporting this.

Mar 10 '22 12:03 sandreas

Using what nerd-king stated earlier as a starting point I did a test as well. Initially, I mounted the following Volumes. These /mnt/media is actually a SMB Volume mounted to my Docker Host VM. When I do this, the merge is unable to merge all of the mp3s and will even get some out of order for whatever reason.

  - /mnt/media/_todo/audiobookconvert/temp:/temp
  - /mnt/media/_todo/audiobookconvert/original:/original
  - /mnt/media/_todo/audiobookconvert/backup:/backup

I then decided to try manipulating the original mount point by mounting the original folder to a local location on the Docker Host VM as opposed to a mounted SMB Volume on the Docker Host VM and everything just started working. My SMB mount Volume mounted to /mnt/media is a TrueNAS server. Do not know if I am running into a potential IOPS issue, a SMB protocol issue, a mv command issue within the auto-m4b-tool.sh script, or some oddball issue.

  - /mnt/media/_todo/audiobookconvert/temp:/temp
  - /home/myuser/test:/original
  - /mnt/media/_todo/audiobookconvert/backup:/backup

So I have ran several tests and they have been successful. I first filezilla my book with mp3s to my /home/myuser/test folder on my Docker Host VM and I let the m4b-tool Docker VM do its thing where it copies a backup to the mounted SMB Volume on my TrueNAS and then I get the converted m4b file from the TrueNAS mounted SMB Volume.

Mar 15 '22 13:03 tboucher204

Maybe I will introduce a --copy-source-file-to-tmp boolean option to just copy the source file to the tmp path BEFORE converting and try if this fixes the problem... but it's pretty hard to fix this without the ability to reproduce it.

I tried my best, also on Windows, but I did not reproduce the problem so far... I leave this open just to have some options to try for the people expieriencing this....

Mar 15 '22 16:03 sandreas

I don't know about anyone else but this is a cool tool. Now that I have a solution/work-around to the issue, it definitely makes my life easier and will definitely be using it more. I literally just ran across this tool yesterday and started using it today. Thanks for the work you put into it. I was just trying to help everyone try to narrow down the issue.

Mar 15 '22 16:03 tboucher204

Maybe I will introduce a --copy-source-file-to-tmp boolean option to just copy the source file to the tmp path BEFORE converting and try if this fixes the problem... but it's pretty hard to fix this without the ability to reproduce it.

I tried my best, also on Windows, but I did not reproduce the problem so far... I leave this open just to have some options to try for the people expieriencing this....

When trying to reproduce, is your docker on WSL2? I think that may be the underlying issue (much slower drive access maybe?)

Mar 15 '22 17:03 cridenour

When trying to reproduce, is your docker on WSL2? I think that may be the underlying issue (much slower drive access maybe?)

I basically tried EVERYTHING except having a slower harddisk / SSD (All of my systems use NVMe)... While that might be the problem in reproducing this error, I really tried on Linux, MacOS, Windows 10 with WSL1 and WSL2 and I've put A LOT of effort into this. The only things I can think of are new specific settings or retry mechanisms in m4b-tool to workaround the problem - not directly fixing it. I'm pretty sure this is a docker thing, but hey, if I can help, I'll try. Maybe in the future I be able to reproduce... For now, I'll stick with the hints in this ticket.

Mar 15 '22 21:03 sandreas

When trying to reproduce, is your docker on WSL2? I think that may be the underlying issue (much slower drive access maybe?)

I basically tried EVERYTHING except having a slower harddisk / SSD (All of my systems use NVMe)... While that might be the problem in reproducing this error, I really tried on Linux, MacOS, Windows 10 with WSL1 and WSL2 and I've put A LOT of effort into this. The only things I can think of are new specific settings or retry mechanisms in m4b-tool to workaround the problem - not directly fixing it. I'm pretty sure this is a docker thing, but hey, if I can help, I'll try. Maybe in the future I be able to reproduce... For now, I'll stick with the hints in this ticket.

Strange! Mine was happening on a nvme drive so don’t think it’s hdd related. That said , I agree it’s a docker or WSL problem. Will try the workaround option next time I have a m4b to merge and report back.

Mar 15 '22 21:03 cridenour

Will try the workaround option next time I have a m4b to merge and report back.

I'm not sure when it is gonna happen... the workaround is planned, but does not exist atm.

Mar 15 '22 22:03 sandreas

Well then I’ll submit a PR next time!

Mar 15 '22 22:03 cridenour

I was having a heck of a time with this issue as well. Running on Windows 11 targeting WSL2. No matter what I tried certain input files would get skipped. I had two books that had 94 and 95 chapters, but whenever I ran the script from a windows drive (D: in my image below) it would drop chapters 1-42. Every time....

What worked for me was migrating fully to the WSL2 distribution (Ubuntu). I created the directory structure directly on the WSL filesystem and copied my script (largely based on this one) with the minor path updates. Now when I run the script all chapters are included in the output.

Ubuntu filesystem

Sep 11 '22 03:09 Mercerbearman

Ok, I'll try to provide a workaround in the next days here. I experimented with integrating ls and find to get a list of the files and I don't know, if this will fix the problem, but at least it is worth a try to use a native file listing utility to check the file list and do extensive logging.

Mar 10 '23 08:03 sandreas

Probably I'll provide a flag --file-listing-provider=find or something... stay tuned.

Mar 25 '23 10:03 sandreas

m4b-tool m4b-tool copied to clipboard

Just the last 10 files are used when merging an Audiobook with 42 files

m4b-tool
m4b-tool copied to clipboard