phiola icon indicating copy to clipboard operation
phiola copied to clipboard

Overwriting during Internet recording

Open Ducke1001 opened this issue 11 months ago • 19 comments

If I record from the Internet and the file already exists, the recording stops. This happens particularly often with the Gingles. But even when a new song comes on, the recording is not continued. Unfortunately, the "-f" option for overwriting is not available here.

Ducke1001 avatar Jan 22 '25 08:01 Ducke1001

Hi!

But even when a new song comes on, the recording is not continued.

Next file should be created as usual, depending on what output file template you're using. Can you attach logs when it is not the case?

Unfortunately, the "-f" option for overwriting is not available here.

When using file names from the server's meta data, overwriting your existing files on disk is insecure behaviour -- it's better if you'd use @counter or @nowtime to generate unique file names.

stsaz avatar Jan 23 '25 05:01 stsaz

The command:

"phiola.exe" -Log phi.log "https://play.radioking.io/nordic-chillout-radio/736489" -tee "D:\Records\@artist - @title.aac"

gave the following log:

08:36:05.871 #12576 INFO file-write: *2: D:\Records\Max Cooper - Rule 110 (feat. Joe Mcbride) (Synkro Remix).aac: written 1844KB 08:38:45.725 #12576 INFO file-write: *4: D:\Records\ - Jingle Nordic FeelGood.aac: written 63KB 08:42:23.970 #12576 INFO file-write: *5: D:\Records\Firas Tarhini - So Slowly.aac: written 1704KB 08:48:20.512 #12576 INFO file-write: *7: D:\Records\Cannons - Fire For You.aac: written 1164KB

Phiola overlooks some files. They are created but not written. The file size remains at 0. File 1 was also not written and remained at 0 size. I assume the file was too small because the title had been playing for a while.

When using file names from the server's meta data, overwriting your existing files on disk is insecure behaviour -- it's better if you'd use @counter or @nowtime to generate unique file names.

I see it as an advantage to be able to save by artist and title. Perhaps recordings could automatically add an index if the file already exists.

Ducke1001 avatar Jan 24 '25 08:01 Ducke1001

gave the following log

I would need the full debug log to understand what was happening.

I see it as an advantage to be able to save by artist and title.

You can just use @counter. @artist - @title.aac, for example.

stsaz avatar Jan 25 '25 08:01 stsaz

Here is a debug log: phi.log

All files were written at once only after recording was complete. The file size remained at zero for that long. I hope it helps. Otherwise I'll try again.

You can just use @counter. @artist - @title.aac, for example.

One option would be to set the counter at the end of the file. In the beginning it is unfavorable for sorting. But no matter how, you would have to rename each recording afterwards, since the counter in the file name is not needed.

Ducke1001 avatar Jan 25 '25 12:01 Ducke1001

All files were written at once only after recording was complete.

It's a bug, will be fixed in the next version, thank you!

I see it as an advantage to be able to save by artist and title. Perhaps recordings could automatically add an index if the file already exists.

Well, I currently see two options here:

  • Use the template that will generate unique file names
  • Or just ignore the error messages when the file already exists

Adding file name counter automatically only in case the output file exists won't add any benefit, because the new output file will have the same content as the previous one (e.g. artist - title.aac == artist - title (1).aac), and you will delete the second file anyway.

stsaz avatar Jan 26 '25 07:01 stsaz

It's a bug, will be fixed in the next version, thank you!

Thank you very much for your effort and the active development of Phiola.

Adding file name counter automatically only in case the output file exists won't add any benefit, because the new output file will have the same content as the previous one (e.g. artist - title.aac == artist - title (1).aac), and you will delete the second file anyway.

No, not necessarily. Recordings (songs) from internet radio are never exactly the same length. Sometimes they are mixed with other tracks or are interrupted by jingles. I always look for the best version and then delete the others.

Ducke1001 avatar Jan 26 '25 08:01 Ducke1001

All files were written at once only after recording was complete.

This should be resolved in 2.3-rc8.

Perhaps recordings could automatically add an index if the file already exists.

I thought for a moment about that and couldn't come up with the plan how to properly pass this user's intention via CLI (because this definitely must not be default behaviour). Maybe you can suggest a CLI option how you think it's best to do it?

stsaz avatar Feb 01 '25 08:02 stsaz

I like streamripper's approach here (https://github.com/streamripper/streamripper). It saves incomplete titles in a separate folder and you can decide what you want to do with them. Here are the corresponding options that are offered:

-o (always | never | larger | version) Overwrite tracks in complete directory When streamripper rips tracks they are put into the incomplete directory until they are finished. Normally, they are then moved into the complete directory. However, when the track is already there, can use this option to tell streamripper what you want to do. There are three choices: always, never, and larger. If you dont include any of the -o options on the command line, the default is "-o larger" for version through 1.63.4, and "-o version" starting with 1.64.5. If you use the "-o never" option, this tells streamripper to never overwrite any existing file in the complete directory. If you use the "-o always" option, this tells streamripper to always overwrite any existing file in the complete directory. If you use the "-o larger" option, this tells streamripper to overwrite an existing file in the complete directory if the newer file is larger. If you use the "-o version" option, this tells streamripper to keep both versions, renaming the existing file. -t Dont overwrite tracks in incomplete directory Normally streamripper writes the files in the incomplete directory, and then moves it to the base directory (the complete directory) when it is done. If the file with the name of the track already exists in incomplete, it will overwrite the old track. When you use the -t flag, however, this will tell streamripper to backup the existing file in incomplete (appending a version number), and then create the new file. This is useful for streams that dont have meta-data. Because these streams only have a single file, reconnects will cause overwriting the existing file, which is not desired. -T Truncate completed tracks in incomplete directory When you are not overwriting files in the complete folder, the duplicate files will normally stay in the incomplete folder. This option tells streamripper to truncate the files to zero bytes in the incomplete folder if they are a duplicate.

But of course this is more complex to implement.

Alternatively, an option -add (if the file already exists, create a new file with an index) or -dskip (don't skip the file if it already exists) might be possible.

Ducke1001 avatar Feb 01 '25 09:02 Ducke1001

option -add (if the file already exists, create a new file with an index)

This would require the code that either iterates in a loop to find the correct index or reads the contents of the directory -- in either case several to many system calls in the place where we can/should just create the file in 1 single operation.

Personally, I don't see how this approach is any better than just using @counter or @nowtime at the end of file name, but let's keep this issue open for now...

stsaz avatar Feb 03 '25 05:02 stsaz

This should be resolved in 2.3-rc8.

Sorry, but there is still an error in the recording. The current track is sometimes not saved when a new track starts and the new track is not created. I have a log, but it is 34Mb in size due to the ongoing recording of several tracks.

Ducke1001 avatar Feb 13 '25 17:02 Ducke1001

The current track is sometimes not saved when a new track starts and the new track is not created.

Hmm, OK. The design of this feature is somewhat complex (the data is passed between different tracks), so this is not a surprise. Either my last fix didn't work, or there is something else now...

I have a log, but it is 34Mb in size

Log size should not be a problem -- just compress the file with 7zip/LZMA, and post here.

stsaz avatar Feb 14 '25 04:02 stsaz

Yes, I could have thought of that myself. Here is the log.

phi.zip

Ducke1001 avatar Feb 14 '25 06:02 Ducke1001

I checked your log file and didn't find any problems there -- there were 3 files written successfully (you can also see that yourself with type phi.log | findstr INFO), and the 4th one was in progress when the log abruptly ends (probably you just killed the phiola process?).

stsaz avatar Feb 15 '25 05:02 stsaz

The third file was not saved when the fourth was already playing. This was not the case with the first ones. They were only saved when the recording was completed. I'll try again and record longer.

Ducke1001 avatar Feb 15 '25 07:02 Ducke1001

Ok, here is a new log.

phi.zip

The 4th song (anamē - It Can Be Better Now (feat. Welt).aac) was not saved and the subsequent songs were not created either. After that, phiola no longer had any file i/o in the task manager. The log also seems to end there. After exiting Phiola, the last song was saved up to 1.00 MB.

Ducke1001 avatar Feb 15 '25 08:02 Ducke1001

The 4th song (anamē - It Can Be Better Now (feat. Welt).aac) was not saved and the subsequent songs were not created either.

Again, I don't see any problems in this new log file.

Take a look:

09:16:45.999 #15120 DEBUG  icy: *1: meta: [48] "StreamTitle='Juani Bria - Higher';"
09:20:56.055 #15120 DEBUG  icy: *1: meta: [48] "StreamTitle=' - Jingle Nordic FeelGood';"
09:20:56.067 #15120 INFO   file-write: *2: D:\Programmierung\wxbasic\Projects\dsmPlayer\test\Juani Bria - Higher.aac: written 1969KB
09:21:04.003 #15120 DEBUG  icy: *1: meta: [32] "StreamTitle='Snowk - Like You';"
09:21:04.010 #15120 INFO   file-write: *3: D:\Programmierung\wxbasic\Projects\dsmPlayer\test\ - Jingle Nordic FeelGood.aac: written 63KB
09:24:21.761 #15120 DEBUG  icy: *1: meta: [64] "StreamTitle='Sons Of Maria - All The Things We Had';"
09:24:21.762 #15120 INFO   file-write: *4: D:\Programmierung\wxbasic\Projects\dsmPlayer\test\Snowk - Like You.aac: written 1547KB
09:27:09.769 #15120 DEBUG  icy: *1: meta: [64] "StreamTitle='anamē - It Can Be Better Now (feat. Welt)';"
09:27:09.771 #15120 INFO   file-write: *5: D:\Programmierung\wxbasic\Projects\dsmPlayer\test\Sons Of Maria - All The Things We Had.aac: written 1313KB
...
09:29:25.778 #15120 DEBUG  file-write: *6: D:\Programmierung\wxbasic\Projects\dsmPlayer\test\anamē - It Can Be Better Now (feat. Welt).aac: write: bufferred 1267 bytes @1048576+40691
...

4 files were written successfully. The 5th track It Can Be Better Now is being written, and thus you don't see the file size until it's complete.

stsaz avatar Feb 15 '25 11:02 stsaz

I was already on the 7th song and Phiola no longer had file i/o. The log recording also seems to have stopped.

edit1: I ran phiola normally once and once in recording mode with the volume at 0. Then I compared the displayed titles with the saved titles. It is always the case that after 2 or 3 songs, a new file is no longer created when a new song begins. Until then, everything works as it should. When phiola is then closed, it writes the retained data to the file. But usually no new file is created when a new song starts. It looks like phiola is simply frozen.

edit2: Here are 2 screenshots from phiola: Image Image

In the first picture, phiola is working normally. In the second picture you can see that phiola no longer needs a CPU and no longer has any i/o. Then phiola stopped writing to the disk.

Ducke1001 avatar Feb 15 '25 12:02 Ducke1001

The last line in log is track: *1: tui: in:8192, and most likely phiola is blocked by OS on writing to STDOUT. Your program is the parent of phiola process, and your program must read from STDOUT and STDERR streams, otherwise the system buffer is filled completely, and then phiola process is blocked.

stsaz avatar Feb 16 '25 07:02 stsaz

Oh man, you're right. I read the STDOUT and STDERR from first phiola, which plays the title. I previously had the windows media player library for playback and only once phiola for recording. After switching from the media player to phiola, I now have phiola twice and forgot to read the STDOUT and STDERR on the second call. Thanks for the tip.

Ducke1001 avatar Feb 16 '25 08:02 Ducke1001