f2e-spec
f2e-spec copied to clipboard
alibaba
Content distribution over torrent
Implement in-game song downloads to make it easier to use for new users.
WIP in git branch "torrent".
Text moved here from wiki:
== Performous DLC (Downloadable Content) ==
This features aim to provide ingame track search/download for Performous. Performous Team could for example provides free content (Creative Commons) and other people car use it for personnal use (providing it privatly when showing it to friends without needing to always have all the songs on a removable media).
My (Yoda) proposal is to use bittorrent using libtorrent (Rasterbar) with DHT ([http://www.bittorrent.org/beps/bep_0005.html BEP 05]) and Web Seeding ([http://www.bittorrent.org/beps/bep_0019.html BEP 19]). This way even an non-seeded torrent can successfully be downloaded.
DLC will use feed given in the configuration file with an http link. Here is an example of DLC feed :
<?xml version="1.0" encoding="UTF-8"?>
<performous-dlc version="0.0" name="Official Performous DLC">
<song>
<title>Code Monkey</title>
<artist>Jonathan Coulton</artist>
<content>
<cover>http://performous.org/dlc-cover/joco-code-monkey.png</cover>
<content>magnet:?xt=urn:btih:0123456789012345678901234567890123456789&dn=joco-codemonkey</content>
</content>
</song>
</performous-dlc>
It could be nice if:
- the feed can be shown in a song browser screen similar to the current song screen
- the instrument provided by a song could be shown before the download
- the already downloaded content are marked
- the current downloading songs are marked with a percent of completion (and an ETA)
- there were an easy (GUI) way to add/import sources
- other ideas ...
Things to consider:
- should the files be downloaded to a separate DLC dir?
- should the downloaded files be arranged to folders by feed?
The torrent branch is kept up to date with latest changes to master but development on it has stalled. This is probably post-0.8 feature but even then it would be nice if someone else could pick up the development. I've been too busy with various projects to do anything other than continued merges from master to it.
The current idea summarized: Performous downloads (over HTTP, cpp-netlib?) content catalogs from configured repositories. The catalog is a simple XML on JSON file (we haven't really decided yet, JSON would make more sense) that can be served by any HTTP server (static content). It contains a list of packages to be downloaded, along with their torrent URLs (magnet URLs so that no .torrent files need to be downloaded at any point). Performous then uses libtorrent-rasterbar to download these into its DLC data folder. A specific main menu entry is reserved for download manager that can be used for browsing the packages and for controlling the ongoing transfers.
Current state of implementation:
- libtorrent is up and running
- skeleton downloader screen is implemented and included in main menu
- repository support (HTTP client and JSON/XML parsing) is entirely missing
There are no specific plans for package versioning and updates (different active torrents cannot overwrite each-other's files, libtorrent enforces this restriction). This is not absolutely necessary to solve but it would certainly be helpful if updates could re-use the media files already downloaded when a typo in lyrics is fixed. This also begs the question about how small the torrents should be (one package/album per torrent, one song per torrent, one file of a song per torrent?).
Distribution of content directly from Performous is not planned at this time. Repositories are to be set up and manually maintained by us and our users. Performous will come preconfigured with the official repository only.
I think DLC is a good feature for performous so I wanna help implementing it. I want to discuss about some of DLC characteristic first then go to implementing.
That is what I think of DLC, correct me if I'm wrong please
What is the protocol and how different parts of DLC are communicate to each other
we have a server holding the list of songs (xml and/or JSON and/or DB) and can makes torrent files from a subset of them (files or song folder or album or ...). and we have a client (in-game and/or separated) that is in communication with server under some special circumstances. we can use different protocols for client and server to communicate (HTTP, FTP, bittorrent, ...). all of them are good some of them are better so we can start with an easy one(http) and get to more sophisticated one (bittorrent). if I am right to this point we should have two separate part
- DLC Client
- DLC Server
a simple diagram looks like this:
DIAGRAM
Client side
For client side we can have in-game DLC manager or separate client (Desktop and/or Web application). For any DLC client we might have following characteristic (to name some):
- List the songs and show their details
- Select/Mark one or multiple songs and add them to download List/Basket/Queue
- Download the songs to the specified folder
In-Game DLC manager can list like song browser which we have and that would be nice. and we can mix DLC to local song list!. in this case we have list of all songs available or not and we can decide if we want to download the songs right there. We have also in-game playlist (which is gonna be more sophisticated by getting an embedded server for itself). so far we have List and Select/Mark and all we want is Download capability. It would be nice if we had a download bar showing download progress (both one song and total progress). and we can use a filter for DLC songs to separate them easily.
For an outside game DLC we have some option like:
- Web App (would be nice and challenging)
- Desktop Client (easier than others)
For ease of development someone can make a Desktop Client and use it for development and testing. and we can have Desk client manage the setting. and it is good for someone who don't like to open the performous to download the song. and maybe closing the performous abort the downloads (who knows!).
A Web App for performous DLC? yeah why not. It would be cool.
An Android App? why not :) but HOW??!!!
Server side
Of course it will be some debate about it. but general, as far as I know, we can implement the server side with any existing server technology. so we are not limited to anything specific.
- Java (simple Java SE socket program or more complex ejb/jpa or even spring/hibernate)
- python
- php
- ruby on rails
- C++
- ...
What is important is what we expect the server to provide? Java is big and has a large footprint, maybe harder to develop comparing to others but it gives us so much power to do almost anything we want.
I don't have any experience with other web technologies.
clients might have this ability:
- receiving http requests
- creating package/torrent file (should be decided if we want to do this or just create packages beforehand )
- sending package/torrent file
- log anything happens (it's always good to know what's happening)
- handle uploading new songs
we should also decide on how we can create the packages and store them. DB, simple directory on filesystem, or something else.
on the other hand server can do more than just the above listed items. it can be used to manage high-score table (seems like a cool feature), manage accounts (Do we need accounts? maybe for high-score or uploading to server) but they are all new features and has to be discussed on their specific issue.
It became a large post...sorry...I will appreciate your suggestions and comments and criticism.
@Tronic summarized the idea in the previous comment, but the server side is meant to be "stupid", i.e. we don't have any custom software, but instead use regular webserver (such as apache, lighttpd or nginx) to serve static catalog files that list available songs. These would be called Song/DLC Repositories and contain magnet-links to actual songs. The catalogs are downloaded through regular HTTP protocol, most likely with cppnetlib.
All song data downloading happens through libtorrent-rasterbar (already set up) and magnet URIs. Internally, the URIs might not only contain torrents but "webseeds", i.e. a regular HTTP addresses, but that is handled by the library.
We need to keep the scope manageable, so only integrated client side DLC manager (skeleton already implemented) until the system works well.
This is an example of XML for catalog.
<?xml version="1.0" encoding="UTF-8"?>
<catalog version="1" songs="2" servers="1">
<!-- List of Servers-->
<trackers>
<tracker>
<public>Yes</public>
<url>http://bittorrent-public-server.org<url>
</tracker>
</trackers>
<!-- List of Songs-->
<songs>
<song>
<name>Bulls on Parade</name>
<artist>Rage Against the Machine</artist>
<album>Evil Empire</album>
<year>1996</year>
<lenght>3:51</lenght>
<genre>Rock</genre>
<description>
Bulls on Parade is a song released by Rage Against the Machine from their second album Evil Empire. This song, one of their signature songs, deals with what is commonly referred to as the "military-industrial complex", which is the tendency of industry (the arms industry in particular) to encourage military action in order to gain military contracts, and therefore increase its profits.\n
The song features one of Tom Morello's most famous guitar riffs and a highly unusual solo that sounds more like someone raping a rubber duck than a guitar by making use of the toggle switch and rubbing his hand along the strings parallel to the neck of the guitar. It is usually cited as the most famous example of Tom Morello's extremely innovative guitar playing.
</description>
<downloads>
<song-magnet> <!-- song can be a folder or in any package format like zip, tar.gz, 7z, etc. -->
magnet:?xt=urn:sha1:SADFASFASDFAFWNAE52SJUQCZO5C
</song-magnet>
<album-art>
http://performous.org/DLC/songs/album_arts/r/rage_against_the_machine/evil_empire/bulls_on_parade.png
</album-art>
</downloads>
</song>
<!-- ==========Next Song======== -->
<song>
<name>Second song</name>
<artist>John Doe</artist>
<album>An album with no name</album>
<year>1999</year>
<lenght>5:12</lenght>
<genre>Jazz</genre>
<description>
This is one of the best songs in the world!
</description>
<downloads>
<song-magnet>
magnet:?xt=urn:sha1:YNCKHTQCWBTRNJIV4WNAE52SJUQCZO5C
</song-magnet>
<album-art>
http://performous.org/DLC/songs/album_arts/r/rage_against_the_machine/evil_empire/bulls_on_parade.png
</album-art>
</downloads>
</song>
</songs>
</catalog>
Please give feedback. thanks
XML is good and more human readable but a little fat! JSON is so much faster and better...this is a JSON structure proposal for DLC catalog:
{
"last_update": "30 July 2013 @ 22:45",
"songs_count": 2,
"songs": [
{
"name": "Bulls on Parade",
"artist": "Rage Against the Machine",
"album": "Evil Empire",
"year": 1996,
"length": "3:51",
"genre": "Rap Metal",
"description": "Bulls on Parade is a song released by Rage Against the Machine from their second album Evil Empire. This song, one of their signature songs, deals with what is commonly referred to as the \"military-industrial complex\", which is the tendency of industry (the arms industry in particular) to encourage military action in order to gain military contracts, and therefore increase its profits.\nThe song features one of Tom Morello's most famous guitar riffs and a highly unusual solo that sounds more like someone raping a rubber duck than a guitar by making use of the toggle switch and rubbing his hand along the strings parallel to the neck of the guitar. It is usually cited as the most famous example of Tom Morello's extremely innovative guitar playing.",
"downloads": [
{
"magnetlink": "magnet:?xt=urn:sha1:SADFASFASDFAFWNAE52SJUQCZO5C",
"album-art": "http://performous.org/DLC/songs/album_arts/r/rage_against_the_machine/evil_empire/bulls_on_parade.png",
"direct-link": ""
}
]
},
{
"name": "another song",
"artist": "John Doe",
"album": "An album with no name",
"year": 1999,
"length": "5:12",
"genre": "Jazz",
"description": "",
"downloads": [
{
"magnetlink": "magnet:?xt=urn:sha1:ZSDFASFASDFAFWNAE52SSSSCZO5C",
"album-art": "no album art",
"direct-link": ""
}
]
}
],
"servers_count": 1,
"servers": [
{
"url": "www.performous.org",
"description": "main server"
}
]
}
so far I added a function downloadCatalog() usingboost::asiowhich downloads the catalog.json from a url and save it somewhere (where is should be saved?). then we can use libjson-cpp to parse catalog.json contents and play with it. I need to test 2 more things:
- first test parsing
- second test downloder class
Then the back-end is ready for GUI.
I have a problem! I cannot have both libtorrent-rasterbar and boost::asio in performous. when I compile with torrent option OFF everything works. but when I enable torrent option I get error about boost asio:
CMakeFiles/performous.dir/dlc.cc.o: In function `boost::asio::ip::resolver_service<boost::asio::ip::tcp>::destroy(std::shared_ptr<void>&)':
/usr/include/boost/asio/ip/resolver_service.hpp:89: undefined reference to `boost::asio::detail::resolver_service_base::destroy(std::shared_ptr<void>&)'
CMakeFiles/performous.dir/dlc.cc.o: In function `boost::asio::ip::resolver_service<boost::asio::ip::tcp>::construct(std::shared_ptr<void>&)':
/usr/include/boost/asio/ip/resolver_service.hpp:83: undefined reference to `boost::asio::detail::resolver_service_base::construct(std::shared_ptr<void>&)'
CMakeFiles/performous.dir/dlc.cc.o: In function `boost::asio::ip::resolver_service<boost::asio::ip::tcp>::destroy(std::shared_ptr<void>&)':
/usr/include/boost/asio/ip/resolver_service.hpp:89: undefined reference to `boost::asio::detail::resolver_service_base::destroy(std::shared_ptr<void>&)'
/usr/include/boost/asio/ip/resolver_service.hpp:89: undefined reference to `boost::asio::detail::resolver_service_base::destroy(std::shared_ptr<void>&)'
I tested rasterbar version 0.16.7 from ubuntu repository and compile&installed version 1.0.0 (latest version) from svn. and I tested both of rasterbar versions with boost version 1.49 and 1.53 but none of the combination worked!
I checked libtorrent-rasterbar.pc and saw lots of options there...but after removing some of them the error remained. interesting thing is that rasterbar depends on boost::filesystem1.49 and doesn't work with other version. I asked a question on stackoverflow about this issue but got no answer. asked on IRC got some answer. tested but nothing solved. I just give up with rasterbar :-/
but I want to propose to add other protocols for downloading for now and add torrent later. getting files using boost::asio using http and ftp is easy. and I checked LibArchive for extracting and it seems to be good and have support for many packages such as 7z, tar.gz, tar.bz2, etc.
I checked the screen_downloader class and looks like it uses downloader as a torrent downloader. some abstraction here may come useful. by the way screen_downloader is not working right now.
Pros
- easier to implement
- works for those who cannot download torrent.
- maybe faster
Cons
- need a server to hold packages (about starting with 20GB)
- copyright issue?
I need your comments, votes and corrections.
Those errors don't seem to have anything to do with libtorrent-rasterbar, as they originate from asio. To me it seems you haven't added asio library to CMake so it is not linked with Performous - try adding asio to our required boost components in game/CMakeLists.txt.
@tapio There is no need to add asio to boost component list in game/CMakeLists.cmake . adding asio to this file gives error and also asio works without adding it to game/CMakeLists.cmake file. I tested it in performous by turn OFF libtorrent and even in a separate project with asio and cmake. In both situation asio works without need to modify cmake file.
I also googled and saw lots of asio+cmake project and checked their cmake files. none of them has anything for asio. they just add thread, regex and system which are asio dependency.
What I have plan to do is to create a base class downloader.hh and two child torrentDownloader.hh and httpDownloder.hh in this case we can have multiple protocol supported.
I too a look at the torrent branch (locally merged it with master, did not commit the result) but I keep getting errors that DLCcontent is not declared anywhere. furthermore you should change
#include <json/json.h>
to
#include <json-cpp/json/json.h>
or it will use the c version of json lib (which is the wrong library)
If i have some spare time next weekend i might look into this, but starting from scratch since the current state of the branch is probably not updateable...
If i have some spare time next weekend i might look into this, but starting from scratch since the current state of the branch is probably not updateable...
Hello, don't forget to look at past discussions here (especially choosing a library with DHT ([http://www.bittorrent.org/beps/bep_0005.html BEP 05]) and Web Seeding ([http://www.bittorrent.org/beps/bep_0019.html BEP 19]). libtorrent (Rasterbar) was a good match a few years back.
Web Seeding is nice because it enables easy testing (just need a webserver) and reliability (no need to always have a torrent running somewhere.
A vote for libtorrent-rasterbar, still after all these years. QBittorrent and other popular clients are based on it.