Retrieve the final Final URLs automatically and add them to the database

[davidhedlund]

The playlists downloaded from radio-browser.info mostly comes shipped with "Official" URLs, links that are copy/pasted from the website that hosts the radio station. Some of these links will redirect you to another address and this cause problems:

• The official URLs often redirects from "https" to "http", this leads to a false sense of security.
• Redirect URLs don't work with certain applications, like streamripper.

Solution

A pragmatic solution is to make a script that could detect if the URL is a redirect URL, and store the final URL in the database. There are currently 12905 stations in the database and if you estimate that 2/12th of it are redirects as in my example below then you got 2151 (12905÷6) stations to update final URLs on.

Then I suggest that you make it possible to download customized variants of the playlists:

• Playlists with official URLs (long-term): This is the current playlist model.
• Playlists with final URLs (short-term): Auto-indexed by tracking the official URL. This should be added as an extra task to the current cronjob.

Example

There are 2 offical URLs (of 12) that redirects in http://www.radio-browser.info/webservice/pls/stations/bytagexact/psytrance

File4=https://hirschmilch.de/psytrance/listen.pls
Title4=Hirschmilch Radio Psytrance

Investigation
Redirect destination url (showed when played in VLC)=http://hirschmilch.de:7000/psytrance.mp3
HTTPS redirected to HTTP=Yes

$ streamripper https://hirschmilch.de/psytrance/listen.pls -u "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
Connecting...
Segmentation fault
$ streamripper http://hirschmilch.de:7000/psytrance.mp3 -u "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
Connecting...
stream: Hirschmilch Psytrance
server name: SHOUTcast/
declared bitrate: 128
meta interval: 16384

[skipping...   ] Wrecked Machines - Bandbox (Symbolic & Waio Remix) [   400kb]

###################################################

File10=http://schizoid.in/schizoid-psy.pls
Title10=Radio Schizoid - Psychedelic Trance

Investigation
Redirect destination url (showed when played in VLC)=http://88.198.48.36:8000/schizoid
HTTPS redirected to HTTP=No

$ streamripper http://schizoid.in/schizoid-psy.pls -u "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
 streamripper streamripper http://schizoid.in/schizoid-psy.pls
Connecting...

error -6 [SR_ERROR_CANT_RESOLVE_HOSTNAME]
bye..
shutting down
$ streamripper http://88.198.48.36:8000/schizoid -u "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
streamripper http://88.198.48.36:8000/schizoid
Connecting...
stream: Radio Schizoid
server name: Icecast 2.3.2
declared bitrate: 128
meta interval: 16000

[skipping...   ] Archaic - Owl Trip [  360kb]