youtube-dl icon indicating copy to clipboard operation
youtube-dl copied to clipboard

Published 20 hours ago • verified publisher icon ytdl-org

Please add support for https://veeps.com/ <--high quality live concert streams

Open poochbeast57 opened this issue 4 years ago • 61 comments

Checklist

  • [x] I'm reporting a new site support request
  • [x] I've verified that I'm running youtube-dl version 2020.06.16.1
  • [x] I've checked that all provided URLs are alive and playable in a browser
  • [x] I've checked that none of provided URLs violate any copyrights
  • [x] I've searched the bugtracker for similar site support requests including closed ones

Example URLs

  • Single video: https://www.youtube.com/watch?v=BaW_jenozKc
  • Single video: https://youtu.be/BaW_jenozKc
  • Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc

Description

WRITE DESCRIPTION HERE I'm trying to Download Streaming Video that's Divided into Parts in XHR Format. I don't see any files that end with .m3u8 The concert I'm trying to download has sequential files that run from 0.ts to 2452.ts (the concert runs 2:43:28)

Here are some example URLs that's I've tried to add to youtube-dl.exe https://jessemalin.veeps.com/stream/80d09cea10f9 https://fn0pgvgj5615mhof01jc9652a.litix.io/ blob:https://jessemalin.veeps.com/c82b2f01-5915-49a9-8e4f-b66ec7ca04e4

Here are the first, "middle" and last files from the sequence:

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/ipAtw008lecNKzC6Y02jZh38jhCShD01F00BKu5v2Kdq3zN5AwRSA2stzinYxKlUtmdShoKDFYfH5e9dtJRdgqQzpGmhfXfNdAOF/0.ts?skid=default&signature=NWVmNThlZTBfNGUzN2VlOTdkNzBiOTZlODg3OTJhNzc2NGI4MzMwMzMyYjYyNDliN2ZiMzE0NjcxMjMwMTc0Nzk1YzhkZGE3Mg==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/ipAtw008lecNKzC6Y02jZh38jhCShD01F00BKu5v2Kdq3zN5AwRSA2stzinYxKlUtmdShoKDFYfH5e9dtJRdgqQzpGmhfXfNdAOF/1226.ts?skid=default&signature=NWVmNWI5MTBfODU2MDlkZTY5OWZlMDdiZWNiOGEyYmQ3MDExZWY3ZWNhNjNkMjBhNWY1MTZkMmNiNWI2NzhjZjkzZDYxYmRlMQ==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/ipAtw008lecNKzC6Y02jZh38jhCShD01F00BKu5v2Kdq3zN5AwRSA2stzinYxKlUtmdShoKDFYfH5e9dtJRdgqQzpGmhfXfNdAOF/2452.ts?skid=default&signature=NWVmNThlZTBfYTZiMzgwMWFhZTE5Mzg3OTljMDZiMjM2ZmVmYjRmNzBjNzE1ZTllYTczNGYzOTI0ZGQ3ZmVlYWY1ODE3YjgwNg==

This was a paid concert, but I paid for it, so I'd like to be able to download it, not just watch it. Thanks!

poochbeast57 avatar Jun 19 '20 08:06 poochbeast57

Those last URLs 403 now due to an expired signature; removing the signature from the URL also causes a 403. None of the urls that you have added to youtube-dl contain this signature, so I imagine the signature is user-specific and created when a user requests the video, with a finite lifetime. I'll try to take a deeper look into this tomorrow and see if it is possible, but it looks like there needs to be some way of generating a valid signature in order to be able to request the stream.

dunnousername avatar Jun 27 '20 04:06 dunnousername

I'm a fan of musician Jesse Malin, but many other artists use this "veep" platform, including Brandi Carlile. Jesse has been performing concerts live on Thursday evenings, and the replay is available to stream until late Saturday evenings. The average size of each file in the live stream is approx. 1100 KB, but the the average file size of the replay files are only 725 KB. There's a noticeable drop in quality. I'd love to know how to grab the higher quality stream from the replay. The cost for access is as low as $10 per concert. By the time you "take a deeper look," the stream will be gone, so I took the liberty of grabbing a string of URLs from the most recent concert for you to see. To my laymen's eyes, they all seem to have unique signatures...making grabbing them much more difficult. Until a bulk download method is perfected, I've been double-clicking on each URL (.ts file) and downloading them individually. Each concert contains approx. 2000 .ts files, so as you can imagine, this is not fun. Then I join them using TSSplitter (Joiner). During the last concert, I grabbed nearly all of the files during the live broadcast, except for 3 groups of 50. I wasn't happy when I went back to grab them during the replay and I noticed the smaller file sizes...which don't match perfectly to the larger files (the audio is seamless, but the video skips). I haven't checked, but I'm guessing it's the difference between 1080p and 720p. Also, this video can't be played with most of my standard video players, but it does play with VLC media player. What's up with that?

Here's the URL:

https://jessemalin.veeps.com/stream/75193060dd58

Here's are some sample sequential files:

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/260.ts?skid=highwinds&e=1593882000&st=031d9a3bd68892b4bfc12a770fb8e598

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/261.ts?skid=highwinds&e=1593882000&st=222a210ce6d8cfb6cf5f0660ac6fdeaa

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/262.ts?skid=highwinds&e=1593882000&st=27c5ebfe92d8cc749212a13d69406e0b

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/263.ts?skid=highwinds&e=1593882000&st=aff881c1c679967fb54da4ce550042cf

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/264.ts?skid=highwinds&e=1593882000&st=2ce38fd18d6ed5344a1544c859d62000

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/265.ts?skid=highwinds&e=1593882000&st=726e6a8aeb5e04e155e9481f17a5bca0

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/266.ts?skid=highwinds&e=1593882000&st=fc7160f6c72a4957bab6db4ed4109ba4

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/267.ts?skid=highwinds&e=1593882000&st=e8a844bc5dc2209e9e251cd14017faf9

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/268.ts?skid=highwinds&e=1593882000&st=4b45340fa732002bcdbe81fcbe675f47

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/269.ts?skid=highwinds&e=1593882000&st=6944fab671f30080ed76223bc0a80add

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/270.ts?skid=highwinds&e=1593882000&st=efd4bfd7f0a0ec87dd4ed4c73a36e5b4

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/271.ts?skid=highwinds&e=1593882000&st=fe6414e42e4dedc2478a5457ee057bea

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/272.ts?skid=highwinds&e=1593882000&st=e8695494b9f74dc07a0e2b60f40da9e5

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/273.ts?skid=highwinds&e=1593882000&st=5a8bf24536ca327a764dd05729b42a73

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/274.ts?skid=highwinds&e=1593882000&st=eaa6c177d5d6aaeb6980c60d2e2439f6

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/275.ts?skid=highwinds&e=1593882000&st=a67907a92cef33e1dc4b8d7e8c91aca4

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/276.ts?skid=highwinds&e=1593882000&st=1e5729182a973019b431ed4905332bc6

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/277.ts?skid=highwinds&e=1593882000&st=574ac5ba4db94e1ba5ed22789831c0a7

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/278.ts?skid=highwinds&e=1593882000&st=9a2f7ab71f1b8828afb4807669230fb8

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/279.ts?skid=highwinds&e=1593882000&st=641e3f65b645d1096505a441af43df01

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/280.ts?skid=highwinds&e=1593882000&st=df6e69b8ff9abfdf296fa38231a00357

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/281.ts?skid=highwinds&e=1593882000&st=bf98312c02c2be3c2da8cda45fd78a95

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://ps8.pndsn.com/v2/subscribe/sub-c-84ee6f14-961d-11ea-a94f-52daec260573/jessemalin_75193060dd58%2Cjessemalin_75193060dd58_artist/0?heartbeat=300&tt=15932771060886169&tr=2&uuid=c666329bd069&pnsdk=PubNub-JS-Web%2F4.27.6&l_hist=227

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/282.ts?skid=highwinds&e=1593882000&st=b246a5e0ef13fffd7abe8555e1b7ac24

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/283.ts?skid=highwinds&e=1593882000&st=dcb0381eac7c696960daa5e250d7149d

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/284.ts?skid=highwinds&e=1593882000&st=4a1954479320fc1b4b8c10752dd433ef

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://c9q7k6z9.map2.ssl.hwcdn.net/v1/chunk/KsANdJ8ZZ02g9DtlGrZa02dqTDwBvHLK1GjQwDObj7KL2gzJvyZCBIEKmwjgYmIqLsbtLXii02PcFTrURZIZBPU6pvGpgaSoaP01/285.ts?skid=highwinds&e=1593882000&st=508732489964bb766571770f049377a5

Be sure to notice the 2nd URL (bold) after 281.ts

Let me know if I can provide any additional info for your research. Thanks!

poochbeast57 avatar Jun 27 '20 17:06 poochbeast57

Are these the URLs that get requested by the web browser upon viewing? The litix.io URLs seem to just be 1x1 GIF images for some reason. All of the URLs with .ts files appear to work.

Also, as for why VLC can play it and other media players can't - I'm not 100% sure, but it is probably something to do with the container format and the codecs used. VLC, in my experience, is more tolerant to weird formats than the default windows media player, for example. If you want to make it play on more video players, I would try re-encoding it to something like H.264 video with AAC audio on a .mp4 container (a pretty standard combination IIRC) using ffmpeg or a similar tool.

When I go to https://jessemalin.veeps.com/stream/75193060dd58, it asks to me to login. I've tried creating an account, and after logging in it takes me to the webpage to buy tickets without showing any info about the stream. It seems that it serves a different page depending on if you have purchased tickets already or not, so it looks like we need to figure out how to authenticate a user on that site. I can't find anything relating to the .ts URLs on my version of the page. I'm not sure how the youtube-dl community normally goes about site support requests that require authentication due to the sensitivity of information; while I've been developing software for a while, this is one of my first issues on GitHub, so I'm not entirely sure how these types of things are handled.

Finally, I'm still puzzled at how the signatures are determined; it might be some client-side code that runs when you are viewing the stream live, or it might send another HTTP request to some server to get the signatures.

dunnousername avatar Jun 27 '20 18:06 dunnousername

  1. Yes, these are the URLs that get requested by the web browser.
  2. You would know better than me why the 1x1 gif images alternate with the .ts files (but their pattern seems more random).
  3. I'm guessing that the 1 odd file (after 281.ts) re-verifies my subscription?
  4. I'm assuming it's too late to buy a ticket for the previous Jesse Malin concert, and I'm not expecting you to buy one. Tickets for next Thursday's show go on sale on Sunday (again, I'm not asking you to buy one). On the other hand, you might find an artist/band with an upcoming concert for whom you would like to buy a ticket: https://veeps.com/#livestreams Some artists leave their streams on the site indefinitely.

I would gladly share my login info with you, but not here in a public forum. Just promise me that you won't change my password and lock me out of my own account :-) (I can always change my password later after you've finished your investigation). How can I contact you privately? Maybe you could setup a temporary gmail account, post the address here, and then discard later?

Like you, I'm puzzled at how the signatures are determined, but this is really over my head. I can follow instructions to use software you've developed, but I don't possess the knowledge to design the software myself.

Thanks for your help!

poochbeast57

poochbeast57 avatar Jun 27 '20 21:06 poochbeast57

Alright. If you feel comfortable with doing that, I've created a temporary email address at [removed]. The address is brand new and there isn't a recovery email or phone number associated with it; I generated a fresh password for it, but I'll be sure to delete the account later anyway. I won't change your password.

After I get an email I'll probably edit the address out of this post so that it doesn't get used by anyone in the future.

Thanks, dunnousername

dunnousername avatar Jun 27 '20 22:06 dunnousername

OK. I just replied to you at your temporary gmail address. Let me know if you received my reply (either here or directly).

Thank you! poochbeast57

poochbeast57 avatar Jun 27 '20 23:06 poochbeast57

Yes, I've gotten it. I'll take a look at it now. Thanks

dunnousername avatar Jun 27 '20 23:06 dunnousername

Okay, update - I don't know how, but after adding breakpoints on XHR requests using console, I was able to get it to request an M3U8 file. I opened the file with VLC and it appears to contain the entire stream, with all of the signatures. Right now I'm trying to figure out how the file was requested.

https://manifest-gce-us-east1-production.global.ssl.fastly.net/vEskNnyPa9go9GOoiPWjkkzIiXk8fRNZ01VhyDpuQRTygRaxVTteLtM8KLYEeBim0002VFxvx7q02ZGir8htZ2LlO7mdbm6rrGUX/rendition.m3u8?cdn=highwinds&expires=1593907200&skid=default&signature=NWYwMTE4MDBfMjJmZDQzNGE0NTZjZjM4YzUzYjM5NTU0OTVkMDNmZTQ4ZGVmZDM4NjI5YjRmMDdkNWY3ZmI4ODE3ZTQ1M2JiYw==

dunnousername avatar Jun 28 '20 00:06 dunnousername

Alright, I think I've figured it out, minus only the authentication. I'm putting it here in case I am unable to figure this out for some reason, so that someone else can build off of this. If I take the _veeps_session_production cookie from the browser and download the page using wget or any other tool, I can get the same page, but without evaluating js (it ignores user agents as well). Then, one of the div elements (div[data-react-class="containers/StreamEvent"] should select it) contains a ton of html-escaped json in the data-react-props attribute. If I unescape and parse that json object as, say, obj, then obj.event.stored_stream_asset is the m3u8 file. This file is hosted on stream.mux.com, and inside of it are links to nested m3u8 files containing the actual ts files. You can actually run the stream.mux.com m3u8 file from VLC, too. The only challenge now is getting a session cookie.

dunnousername avatar Jun 28 '20 00:06 dunnousername

I'm having trouble with figuring out how getting a session token works; I can't replicate it with wget exactly the way it would work in a browser. It's getting kind of late here, however I can probably figure out how this part of the process works with my own account later/tomorrow.

To recap:

  • I haven't figured out how to log in to veeps yet, but this part most likely doesn't require the user to have purchased tickets
  • logging into veeps yields a session cookie stored in _veeps_session_production.
  • requesting a stream page with this session token when tickets have been purchased gives a page with a video player
  • on this page with the player, a div element given by css selector div[data-react-class="containers/StreamEvent"] has an attribute data-react-props that contains an html-escaped json object
  • inside this json object, object['event']['stored_stream_asset'] contains a url to an m3u8 playlist containing several more playlists, most likely for different resolutions.

dunnousername avatar Jun 28 '20 01:06 dunnousername

https://manifest-gce-us-east1-production.global.ssl.fastly.net/vEskNnyPa9go9GOoiPWjkkzIiXk8fRNZ01VhyDpuQRTygRaxVTteLtM8KLYEeBim0002VFxvx7q02ZGir8htZ2LlO7mdbm6rrGUX/rendition.m3u8?cdn=highwinds&expires=1593907200&skid=default&signature=NWYwMTE4MDBfMjJmZDQzNGE0NTZjZjM4YzUzYjM5NTU0OTVkMDNmZTQ4ZGVmZDM4NjI5YjRmMDdkNWY3ZmI4ODE3ZTQ1M2JiYw==

I'm glad I was able to play a small role (albeit very small) in helping you to figure out help this works. I understand about half of your explanation. ;-)

I'm sure you know this already, but the video stream produced by this link is only 480x270. I'm assuming the higher resolution files are there. My questions are "where?" and "how do we find them?"

poochbeast57 avatar Jun 28 '20 01:06 poochbeast57

I'm having trouble with figuring out how getting a session token works; I can't replicate it with wget exactly the way it would work in a browser. It's getting kind of late here, however I can probably figure out how this part of the process works with my own account later/tomorrow.

To recap:

  • I haven't figured out how to log in to veeps yet, but this part most likely doesn't require the user to have purchased tickets
  • logging into veeps yields a session cookie stored in _veeps_session_production.
  • requesting a stream page with this session token when tickets have been purchased gives a page with a video player
  • on this page with the player, a div element given by css selector div[data-react-class="containers/StreamEvent"] has an attribute data-react-props that contains an html-escaped json object
  • inside this json object, object['event']['stored_stream_asset'] contains a url to an m3u8 playlist containing several more playlists, most likely for different resolutions.

I should have read all of your comments before responding to your first comment. I'm glad you were able to find playlists for different resolutions. I don't mind supporting the artists and buying a ticket. I just want to know how to capture the stream. I don't fully understand all of your instructions. If you're seeing this before you go to bed, is there any chance you could share a link to the highest resolution file, so I can download it before they remove it (if indeed they really do remove it). Also, I wonder if there's a way to find the stream(s) from the previous concert(s)?

Thanks again for all your help!

Poochbeast

PS - Is it too much to ask for you to create a tutorial video? Just askin'? ;-)

poochbeast57 avatar Jun 28 '20 02:06 poochbeast57

I'm planning on adding this to youtube_dl through a pull request; I think I can get this done relatively quickly (next few days likely). A video tutorial won't be necessary, as it'll be available in a youtube_dl update once I get it sorted out. It's also somewhat hard to do by hand, so it'd be easier to do through youtube_dl.

The M3U8 is https://stream.mux.com/wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig.m3u8 However, when I play it in VLC, it seems to be low resolution; This version is supposed to be 1920x1080; I can't tell if it actually is or not: https://manifest-gce-us-east1-production.global.ssl.fastly.net/9VnfMCJ02WjWwgOS01hZWkYM89ZCO00T8sWZ2WYzdAfEwqysblWuDfu47eXHkgUV31sXDmPHzYqwW5Ae02DZbDXEA7m0200uQVsOnW/rendition.m3u8?cdn=fastly&expires=1593914400&skid=default&signature=NWYwMTM0MjBfNmY0ZTMyMzc5YmRhNGE2MTFjNTNhYjJjODViNGIxNGViMjNjZTkwMGZjNTUzOGJkM2FmOTFjNGEyMzA2OGE1MQ==

dunnousername avatar Jun 28 '20 02:06 dunnousername

Excellent! (re: your plan to add it to youtube-dl). I can't wait! Yes, this version is definitely 1920x1080. It will be terrific to be able to download all of the .ts files in bulk rather than one-at-a-time! LOL! I wish I could figure out how to do that now. It's really sleazy of veeps to stream the concert in 1080p and then replay it in only 720p. Thanks again!

poochbeast57 avatar Jun 28 '20 02:06 poochbeast57

I know I said I was going to bed like 3 hours ago, but I realized it'd be much harder to test the script after tonight (and I also had nothing better to do); I wrote a module so that youtube_dl could login and download a stream from veeps. (I'm actually, 100% going to bed now). I'm currently testing it on the 1920x1080 version; it's downloading all the .ts files and re-encoding them, so needless to say it is taking a while and I'm going to leave it running overnight. Tomorrow, if all is well, I'll push the code back to github and try to get it merged into the official repo. A few things of note, which are probably non-issues but might be significant to contributors reading this that are more knowledgeable on youtube_dl than I am:

  • I'm willing to revise my code if it doesn't meet style guidelines; this is the first contribution I've made to a project of this size, so I'm pretty new to this.
  • By default, it tries to download im 480x270, as it believes this is the best quality for some reason. I'm not sure if there is anything I can do about this; however, other resolutions can be listed with -F and then selected with -f. I think -f 4755 worked for me but I can't remember the exact number and I'm not on my desktop.
  • You'll need to provide --username and --password to youtube-dl. Make sure if your password contains special characters, such as exclamation points, you prepend them with a backslash.

When I create the pull request I'll reference it in this issue.

dunnousername avatar Jun 28 '20 05:06 dunnousername

If you want to continue working on this, some artists (like Brandi Carlile) leave their streams on the site to be viewed "On Demand" with no expiration date.

If you would like, I could buy a ticket for one of her past concerts, and you can view it through my account. I'm sure it would be best to try other artists just to be sure there aren't any anomalies between artists' streams.

Also, since my password contains no special characters, I can create one that does. I can send it to you through your temporary gmail account (if you haven't deleted it yet). Let me know if you want me to do this.

As I suspected, even though the link to the Jesse Malin concert has now been removed from the front end of veeps, we can still download it because we know the server address. I wonder if it's possible to use a "list" command to find other hidden concerts on the server? I'd love to get his previous show (as well as some shows by some other artists).

poochbeast57 avatar Jun 28 '20 15:06 poochbeast57

I don't think different channels will cause issues; however, if you'd like, I can try additional channels.

A password containing special characters isn't an issue with youtube_dl or my addition; it is an issue with the command line shell, so it is the case for every program. You don't need to create a password with special characters.

It wouldn't be possible to list other concerts on the server; if that were possible, anyone could download any streams without buying tickets, and this is certainly something veeps developers have considered.

dunnousername avatar Jun 28 '20 17:06 dunnousername

I bought a ticket to the Brandi Carlile concert from June 1. It's in my account if you want to check it out. PS - I used VLC to stream and record the Jesse Malin concert in real time using the 1920x1080 link you provided, but doesn't VLC re-encode the stream? While it looks good, I'm sure it's still even better to be able to download and join the .ts files (which youtube-dl should provide for this site when you've completed your update).

poochbeast57 avatar Jun 28 '20 18:06 poochbeast57

VLC doesn't re-encode the stream; in fact, it uses the raw .ts files to play the video. If you are recording it, though, it might re-encode it, but it is most likely set to be visually lossless by default. In short, it shouldn't make a difference. I'll test the June 1st stream to make sure everything works. I'll also include the logs, minus sensitive data, in the pull request.

dunnousername avatar Jun 28 '20 18:06 dunnousername

Yes, I was referring to recording with VLC. I'm just learning Python, but I still don't understand it well enough to add your Veeps extractor. I guess I'll wait for the update.

poochbeast57 avatar Jun 28 '20 18:06 poochbeast57

Well, if you want to try using my branch, and you have python installed, you can download my branch here before it merges, so you don't have to wait to update: https://github.com/dunnousername/youtube-dl/tree/veeps

Make sure to git checkout veeps afterwards in that directory to get the right branch.

Once you download it, you can change directory into the folder and run python -m youtube_dl and it should run my version. If it doesn't detect stream URLs it might be conflicting with the one installed through pip, so try uninstalling the pip version of youtube_dl.

You can use it like you would with downloading any other stream, except pass --username [email] --password [password] too.

For the June 1st stream for example, run python -m youtube_dl --username [email] --password [password] https://brandicarlile.veeps.com/stream/e5ccef3d7eba -f 2516 to download the highest resolution version (720p)

dunnousername avatar Jun 28 '20 18:06 dunnousername

(Occasional Veeps user here.)

Just some technical stuff to explain: Once you have the link to a working m3u8 playlist (or youtube-dl does), the rest can be handled by youtube-dl. Download the m3u8 and check its text contents, you will see links to new playlists with the various resolutions. Nothing more to worry about.

This is how yotube-dl already can handle it (if you don't need a login, or special cookies/links):

[barsnick@paradise youtube-dl]$ youtube-dl -F https://stream.mux.com/wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig.m3u8                                             [generic] wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig: Requesting header
WARNING: Could not send HEAD request to https://stream.mux.com/wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig.m3u8: HTTP Error 405: Method Not Allowed
[generic] wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig: Downloading webpage
[generic] wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig: Downloading m3u8 information
[info] Available formats for wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig:
format code  extension  resolution note
697          mp4        480x270     697k , avc1.64001e, mp4a.40.2
994          mp4        640x360     994k , avc1.64001f, mp4a.40.2
1640         mp4        960x540    1640k , avc1.640020, mp4a.40.2
2516         mp4        1280x720   2516k , avc1.640020, mp4a.40.2
4755         mp4        1920x1080  4755k , avc1.64002a, mp4a.40.2 (best)

youtube-dl of course still requires explicit veeps support to figure out the playlist URI, the title, and so on (which is what @dunnousername programmed, I guess - if so, good job!).

Regarding re-encoding: Like your manual downloading of the .ts segments, you can let youtube-dl do that for you, and even correctly assemble them in the correct order without any reencoding or even remuxing:

[barsnick@paradise youtube-dl]$ youtube-dl --hls-prefer-native --fixup never https://stream.mux.com/wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig.m3u8
[generic] wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig: Requesting header
WARNING: Could not send HEAD request to https://stream.mux.com/wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig.m3u8: HTTP Error 405: Method Not Allowed
[generic] wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig: Downloading webpage
[generic] wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1843
[download] Destination: wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig-wKk6rMRgprh44VympfJo9jHxI4q01mcuod01UCavzPPig.mp4
[download]   0.1% of ~3.71GiB at  4.86MiB/s ETA 17:48

(Ignore the warning here.)

The resulting file is am MPEG-TS file, which many players won't support (especially if you leave the extension ".mp4", which is incorrect, but shouldn't matter for smart tools). You can re-mux (like repackage) that into an actual MP4 without any loss by reencoding, e.g. with ffmpeg (as youtube-dl also does internally for merging, fix-upping, and so on):

ffmpeg -i blabla.mp4 -c copy proper_file.mp4

(assuming the first file is actually not an MP4, but MPEG-TS).

barsnick avatar Jul 03 '20 11:07 barsnick

Is it my imagination, or has veeps changed their streaming protocols? Here are some samples from this week's Jesse Malin concert. To my untrained eyes, every file looks identical:

https://fn0pgvgj5615mhof01jc9652a.litix.io/

https://manifest-gce-us-east1-production.global.ssl.fastly.net/lAI8saZoyEJZ02FRghxdRIlP1dYHt22ZepaKA007ind9xPN902CzBuHoHSVCJPPn6F7iD9n2FB2029ou02pCvsjuHB0129Nu6OmQt7/rendition.m3u8?cdn=fastly&expires=1594940400&skid=default&live=1&signature=NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA=

https://manifest-gce-us-east1-production.global.ssl.fastly.net/lAI8saZoyEJZ02FRghxdRIlP1dYHt22ZepaKA007ind9xPN902CzBuHoHSVCJPPn6F7iD9n2FB2029ou02pCvsjuHB0129Nu6OmQt7/rendition.m3u8?cdn=fastly&expires=1594940400&skid=default&live=1&signature=NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA=

https://manifest-gce-us-east1-production.global.ssl.fastly.net/lAI8saZoyEJZ02FRghxdRIlP1dYHt22ZepaKA007ind9xPN902CzBuHoHSVCJPPn6F7iD9n2FB2029ou02pCvsjuHB0129Nu6OmQt7/rendition.m3u8?cdn=fastly&expires=1594940400&skid=default&live=1&signature=NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA=

https://manifest-gce-us-east1-production.global.ssl.fastly.net/lAI8saZoyEJZ02FRghxdRIlP1dYHt22ZepaKA007ind9xPN902CzBuHoHSVCJPPn6F7iD9n2FB2029ou02pCvsjuHB0129Nu6OmQt7/rendition.m3u8?cdn=fastly&expires=1594940400&skid=default&live=1&signature=NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA==

https://manifest-gce-us-east1-production.global.ssl.fastly.net/lAI8saZoyEJZ02FRghxdRIlP1dYHt22ZepaKA007ind9xPN902CzBuHoHSVCJPPn6F7iD9n2FB2029ou02pCvsjuHB0129Nu6OmQt7/rendition.m3u8?cdn=fastly&expires=1594940400&skid=default&live=1&signature=NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA==

https://manifest-gce-us-east1-production.global.ssl.fastly.net/lAI8saZoyEJZ02FRghxdRIlP1dYHt22ZepaKA007ind9xPN902CzBuHoHSVCJPPn6F7iD9n2FB2029ou02pCvsjuHB0129Nu6OmQt7/rendition.m3u8?cdn=fastly&expires=1594940400&skid=default&live=1&signature=NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA==

poochbeast57 avatar Jul 09 '20 23:07 poochbeast57

Request URL: https://manifest-gce-us-east1-production.global.ssl.fastly.net/lAI8saZoyEJZ02FRghxdRIlP1dYHt22ZepaKA007ind9xPN902CzBuHoHSVCJPPn6F7iD9n2FB2029ou02pCvsjuHB0129Nu6OmQt7/rendition.m3u8?cdn=fastly&expires=1594940400&skid=default&live=1&signature=NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA== Request Method: GET Status Code: 200 OK Remote Address: 151.101.117.194:443 Referrer Policy: strict-origin-when-cross-origin Accept-Ranges: bytes Access-Control-Allow-Origin: * Age: 0 Cache-Control: no-cache, no-store, must-revalidate Connection: keep-alive Content-Encoding: gzip Content-Length: 922 Content-Type: application/x-mpegURL Date: Thu, 09 Jul 2020 23:16:56 GMT Grpc-Metadata-Content-Type: application/grpc Vary: Accept-Encoding Via: 1.1 varnish X-Cache: MISS X-Cache-Hits: 0 X-CDN: fastly X-Request-ID: 07a81dae-542b-4f67-b776-bfefd31f0e5b X-Served-By: cache-bos4650-BOS X-Timer: S1594336617.886909,VS0,VE26 Accept: / Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,it;q=0.8,it-IT;q=0.7,it-CH;q=0.6,ru;q=0.5,es;q=0.4,es-419;q=0.3,de;q=0.2 Connection: keep-alive Host: manifest-gce-us-east1-production.global.ssl.fastly.net Origin: https://jessemalin.veeps.com Referer: https://jessemalin.veeps.com/ Sec-Fetch-Dest: empty Sec-Fetch-Mode: cors Sec-Fetch-Site: cross-site User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36 cdn: fastly expires: 1594940400 skid: default live: 1 signature: NWYxMGRiZjBfZmQ0MDYxMmQ3YjYxMWE4MDljMjE1ZDVlNTE2ZTI0MGE1MGU3MGM5NzkyZmVhNzhlYjk1Njk1MzE5MmJkZGNhNA==

poochbeast57 avatar Jul 09 '20 23:07 poochbeast57

Weird! Now it's back on track...

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/733.ts?skid=default&signature=NWYxMGRiZjBfM2QwNThiOGQyNjQ4NzU4YmJlYWViYTMwODkyMzE5YTNiYTE1MWQzOWQyMTJkYzU2YTMyN2M5ZmFhMmMyMTIxMQ==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/734.ts?skid=default&signature=NWYxMGRiZjBfZjRkYjM0MDg4NmRhZmFlZWYzNDI4N2Q2ZTMyZWE1ZjYxOWY3ZDlmZjUyYTc4ODIxNTBjNDg5ODAxMzJjZjQ3ZQ==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/735.ts?skid=default&signature=NWYxMGRiZjBfMWEzNzhhYjYyNzExMzE5YjNlMDdjY2U4Njk1OTZjNWIwOTc2MjI1ZTRmMDZiZjdlOTIxMjViZDY0MWVhNWE0Ng==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/736.ts?skid=default&signature=NWYxMGRiZjBfZWM0OGYxYzI4ZTI5YmNjOWY4NDUzMmFkNTBiNThlNTQ3Yjc5ZmUyZmM3Y2NiZDc4MTNhMmVjMjdjMDg0NmE0OA==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/737.ts?skid=default&signature=NWYxMGRiZjBfZTZiYjI4Nzg5MjhhYmE2YmUxMTgwOGFjNTRkMWVlY2E0ZGIzMzY0ZTU0MDI1NzJmMjg3MWJkM2I3OGI5YzlhYw==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/738.ts?skid=default&signature=NWYxMGRiZjBfMGZiNGZjZDE4MTIyNGMyMDQ2NTUxYmQ1MTBjMzNjOTYwNjIzZDQ3N2Q1NDQ4MjBiYjJhY2U4ZGQxOTJlYjZiOA==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/739.ts?skid=default&signature=NWYxMGRiZjBfZDEyNDg0ZjNiYTM4NTc0M2Y0ZTJiMTgzYjI1Nzc4ZmNkNDdkMmQ2ZmUzNTFhYjMxMTZlNjM0MTJjMTZjNzJhMQ==

https://chunk-gce-us-east1-production.global.ssl.fastly.net/v1/chunk/scPBjCZCxWg8024i86L3o8eaBFHdBeGW7eBlZRC0086dZ00Fs1aYj0027ViNvKwhRarTCzf9hxR00bkdivToWYpW8fabp02VoAf1oT/740.ts?skid=default&signature=NWYxMGRiZjBfZjA3YjYzM2RkMThiMzIxOWZlMGVlZDBmY2FmZmJjNmNiNTBlYzgyNTQ2MjgwNDdjNjg5MTk3NzY5MzY0NTMzZA==

poochbeast57 avatar Jul 09 '20 23:07 poochbeast57

Jesse Malin 7/9/2020 veeps livestream URL:

https://jessemalin.veeps.com/stream/1fd5fa819d0d

poochbeast57 avatar Jul 10 '20 04:07 poochbeast57

Hey - sorry, I totally forgot about this issue and haven't been checking github for a while. The pull request is still open, but it is unclear what needs to be fixed. All that was said about the pull request was "You must provide account credentials for review." and "Read coding conventions." How does the review process work, and what specifically needs to be changed? I've looked over the coding conventions and it passes the necessary tests on travis.

dunnousername avatar Jul 31 '20 01:07 dunnousername

Hey, I apologize for my delayed reply too. It was all I could do to figure out how to download and install your branch. As far as what youtube-dl requires before merging your branch, it's a bit over my head. However, I would like to see an official merge, so please try to figure it out what they still need. Thanks again for all the work you put into this. It's much appreciated!

One more favor... while you still had access to my veeps account, by any chance did you happen to download the Jesse Malin show from 7/9/2020? That was a really hot show, and at the time I was still trying to figure out how to install your branch, so I missed downloading it. If you got it, I'd love to get a copy. Maybe you could share it via Google Drive or wetransfer or the file sharing platform of your choice? Thanks in advance!

PS - I'm still kicking myself for not realizing how easy it is to find the m3u8 playlist. D'oh!

poochbeast57 avatar Aug 15 '20 03:08 poochbeast57

Has this been added to youtube-dl? If not, anyone figure out a way to dl from Veeps?

codedmon avatar Dec 18 '20 02:12 codedmon

@dunnousername It seems your branch and fork were taken down with the RIAA DMCA shenanigans last year, and it wasn’t restored when the original youtube-dl repository was. Could you repost it?

rsanchezsaez avatar Dec 18 '21 12:12 rsanchezsaez