steam workshop collection downloader

[shadow-absorber]

as requested by @dgibbs64 i am posting the script i made to download mod collections from steam workshop.

#!/bin/bash
curl https://steamcommunity.com/sharedfiles/filedetails/?id=1101449362 > source.html
echo "./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \\" > moddownload.sh
cat source.html \
| grep -E "<div class=\"workshopItemPreviewHolder " \
| sed 's/"><div class=.*//' \
| sed 's/.*id=//' \
| sed -e 's/^/+workshop_download_item 211820 /' \
| sed -e 's/$/ \\/' >> moddownload.sh
echo "+quit" >> moddownload.sh
chmod +x moddownload.sh
./moddownload.sh
rm source.html
rm moddownload.sh

[shadow-absorber]

this thing should help solve the issue #960

[shadow-absorber]

so what the script does in detail is: step 1. download the workshop collection's html page step 2. start creating a bash script to download the mods using steamcmd with the login info step 3. output the html text in a readable way step 4. make grep search the html file for the mod ids step 5. format the ids in a list way for steamcmd in the bash script step 6. add a line to quit the steamcmd download when its done step 7. make the scrip executable step 8. run the steamcmd script to download all the mods and validate their files step 9. delete the files create to reset the script to before it was run to not cause conflicts when it gets run again

[shadow-absorber]

the thing the script needs is a way to replace the id part in this line: curl https://steamcommunity.com/sharedfiles/filedetails/?id=1101449362 and also hook into the login for steam thats used when downloading the game wich is specified currently in this line echo "./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \\" > moddownload.sh

[phit]

no reason to parse the html, you can get all collection objects with the webapi specifically ISteamRemoteStorage_GetCollectionDetails

example using curl:

curl --data "collectioncount=1&publishedfileids[0]=1101449362" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/

output example: https://gist.github.com/phit/71cea6c43c7071a40589881966aa9fa4

[shadow-absorber]

thanks that is a huge improvement then probably. the thing is it still needs to be able to split the mods up into the list for the downloads and not include the collection number as that could cause heavy issues

so plz if you can improve the script post the improved version and we might be able to get this implemented @phit

[phit]

something like this should do, I just cut the first publishedfileid using inverted tail, since the first result will always be the collection, obviously this means you can always only have one collection

you could probably do this way nicer using jq to parse the json, but I don't think adding a dependency for this is really wanted

CMD="./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles "
DOWNLOADITEMS=`curl -s --data "collectioncount=1&publishedfileids[0]=1101449362" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/ \
    | sed -n 's/.*"publishedfileid": "\(.*\)",/\1/p' \
    | tail -n +2 \
    | sed 's/^/+workshop_download_item 211820 /' \
    | tr '\n' ' '`
    
echo $CMD $DOWNLOADITEMS

[shadow-absorber]

yeah that could work. tough i would prefer adding \n at the end of all of the +workshop downloads to make it more readable in output form

[phit]

the output will never be read by anyone in the end product, so I didn't think it was worthwhile

[shadow-absorber]

for example the output for how i wrote mine by myself before i made the first script.

#!/bin/bash
cd /home/sbserver/steamcmd
./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \
+workshop_download_item 211820 850109963 \
+workshop_download_item 211820 821455287 \
+workshop_download_item 211820 731220462 \
+workshop_download_item 211820 882900100 \
+workshop_download_item 211820 945707062 \
+workshop_download_item 211820 958539829 \
+workshop_download_item 211820 729426722 \
+workshop_download_item 211820 729558042 \
+workshop_download_item 211820 729429063 \
+workshop_download_item 211820 729438381 \
+workshop_download_item 211820 764887546 \
+workshop_download_item 211820 764888606 \
+workshop_download_item 211820 796877993 \
+workshop_download_item 211820 730544933 \
+workshop_download_item 211820 734170655 \
+workshop_download_item 211820 730745660 \
+workshop_download_item 211820 729427436 \
+workshop_download_item 211820 947429656 \
+workshop_download_item 211820 947922190 \
+workshop_download_item 211820 774083065 \
+quit

[shadow-absorber]

which is helpful if you have to debug it for some stupid reason

[shadow-absorber]

but then again i guess it would work your way also without problems

[shadow-absorber]

so now with the new api thing the scrip would look like this:

#!/bin/bash
echo "./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \\" > moddownload.sh
curl -s --data "collectioncount=1&publishedfileids[0]=1101449362" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/ \
| sed -n 's/.*"publishedfileid": "\(.*\)",/\1/p' \
| tail -n +2 \
| sed -e 's/^/+workshop_download_item 211820 /' \
| sed -e 's/$/ \\/' >> moddownload.sh
echo "+quit" >> moddownload.sh
chmod +x moddownload.sh
./moddownload.sh
rm moddownload.sh

that would use the api avoiding future problems because of html and should probably work forever it also makes the resulting temporary file for steam cmd a lot easier to read if something messes up.

[shadow-absorber]

for those wondering this is being worked on to get implemented.....

[Fyb3roptik]

Any updates? How can I download a map? I am unable to do this :(

[shinji257]

Does this script work if you set +login to anonymous?

[LoadingSAa]

Has there been anymore work put into this? I wanted to try adapt this for use with an ARMA3 server and was wondering if this has gotten any further?

[shadow-absorber]

@shinji257 the script does not work with anonymous logins @LoadingSAa there has been some work on it but it seemingly stopped when another person helping me quited the project and for other games then starbound you would change the game id(in starbounds case 211820) to whatever it is for that game. i also recall that the github wiki has guides for most of the games lgsm supports

[shadow-absorber]

for a link on the project info so far: WIP

[dgibbs64]

Decent effort so far. I have sadly been too busy to take a proper look :(. This is something that we should look at integrating in to LinuxGSM at some point.

[dgibbs64]

Decent effort so far. I have sadly been too busy to take a proper look :(. This is something that we should look at integrating in to LinuxGSM at some point.

[LoadingSAa]

This is what I have so far for arma3 there is an issue with steam CMD timing out constantly though. I'm trying to find out why.

version1: https://ghostbin.com/paste/uvxf8

this is what I'm trying to make work now:

https://gist.github.com/marceldev89/12da69b95d010c8a810fd384cca8d02a

The above script works for me the only thing I can't get to work is collections.

[phit]

steam started stripping whitespace at some point which broke my script, switched to jq as its more reliable

curl -s --data "collectioncount=1&publishedfileids[0]=IDHERE" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/ \
| jq '.response.collectiondetails[] | .children[] | .publishedfileid' \
| sed 's/^/+workshop_download_item 211820 /' | tr '\n' ' '

[rafaeljardim1660]

this script is for python?

[linonetwo]

And after that, you should edit the serverfiles/linux/sbinit.config

{
  "assetDirectories": [
    "../assets/",
    "../mods/",
    "../../Steam/steamapps/workshop/content/211820/939132577",
    "../../Steam/steamapps/workshop/content/211820/1507130642"
  ],
  "storageDirectory": "../storage/"
}

or steam/steamapps/common/Starbound/linux/sbinit.config if you are not using LinuxGSM.

{
  "assetDirectories" : [
    "../assets/",
    "../mods/",
    "/path/to/Steam/SteamApps/workshop/content/211820/733063633/",
    "/path/to/Steam/SteamApps/workshop/content/211820/743283539/",
    "/path/to/Steam/SteamApps/workshop/content/211820/729769049/",
    "/path/to/Steam/SteamApps/workshop/content/211820/736472581/",
    "/path/to/Steam/SteamApps/workshop/content/211820/732860513/"
  ],

  "storageDirectory" : "../storage/"
}

Don't forget to check whether this file is a good JSON.

[1285done]

Is there any update on this? I'm interested in using a script like this to run my server and if there's already a working script or implementation in gsm, that would be awesome

[dgibbs64]

@1285done there is task to review workshop support but cant give any eta

[ahoys]

For those looking for a fast solution for this, I created a small standalone script that downloads mods based on a workshop collection or by a custom mod file. https://github.com/ahoys/useful-sh/tree/main/steam-workshop-collection-downloader

I use it with a LinuxGSM arma3server instance.