steam-for-linux
steam-for-linux copied to clipboard
ValveSoftware
SteamCMD workshop timeout
Your system information
- Steam client version (build number or date): latest version (no idea how to get steamcmd to tell me)
- Distribution (e.g. Ubuntu): Ubuntu 19.04
- Opted into Steam client beta?: No
- Have you checked for system updates?: Yes
Please describe your issue in as much detail as possible:
I've seen it reported in a bunch of places, but I don't see an issue here. It appears that steamcmd struggles with downloading larger workshop items, with no specific solution.
steam@test:~$ /usr/games/steamcmd +login user +workshop_download_item 107410 583496184 validate +quit
Redirecting stderr to '/home/steam/.steam/logs/stderr.txt'
[ 0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...../steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (117) : Assertion Failed: SteamDatagramTransportLock held for 176.7ms!
OK.
Logging in user 'user' to Steam Public ...
Logged in OK
Waiting for user info...OK
Downloading item 583496184 ...ERROR! Timeout downloading item 583496184../common/pipes.cpp (774) : Assertion Failed: stalled cross-thread pipe
steamclient.cpp (779) : Assertion Failed: bufRet.TellPut() == sizeof(uint8)
CWorkThreadPool::~CWorkThreadPool: work processing queue not empty: 2 items discarded.
Steps for reproducing this issue:
- execute
/usr/games/steamcmd +login user +workshop_download_item 107410 583496184 validate +quit - wait
- should receive an error.
Not sure if these will help but these are some of the links I came across.
- alec-hs/Flaxs-Arma-Server-Tool#12
- https://forums.alliedmods.net/showthread.php?t=288061
- https://steamcommunity.com/discussions/forum/1/215439774859993377/
- https://www.reddit.com/r/Steam/comments/7hudin/steamcmd_bug/
- http://arkservermanager.freeforums.net/thread/2755/error-mod-download-failed
- https://www.reddit.com/r/armadev/comments/7qzgqt/workshop_downloads_timeout/
Ubuntu 19.04, was whatever the latest version was available at the time of creation.
root@ubuntu-test:~# uname -a
Linux ubuntu-test 5.0.0-17-generic #18-Ubuntu SMP Tue Jun 4 15:34:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Is there any solution or workaround? I don't think #6326 is really related to it.
./steamcmd.sh +login $user "$pass" +workshop_download_item 107410 450814997 +quit
Redirecting stderr to '/home/steam/Steam/logs/stderr.txt'
Looks like steam didn't shutdown cleanly, scheduling immediate update check
[ 0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...../steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (117) : Assertion Failed: SteamDatagramTransportLock held for 150.4ms!
OK.
Logging in user 'myuser' to Steam Public ...
Logged in OK
Waiting for user info...FAILED. Timed out.
OK
This is on a centos7 - 3.10.0-957.21.3.el7.x86_64
This kind of breaks my update script for arma server :-(
@michaelsstuff If you monitor your download folder, are you getting some of the item downloaded each time? Often SteamCMD times out on large downloads as you're aware, but if you keep retrying, it'll finally get to the end as long as you're using +validate.
It does however from your most recent log, look like it's timing out even with login details? Have you tried just typing your 2fa code into the blank space and hitting return when it's doing its "time out" part? I've had that be successful at times.
I run an ARMA server too and honestly, SteamCMD is borderline awful for us. We have to devise workarounds, scripts and more to keep re-trying SteamCMD for larger downloads. We've even considered having to remove security features on our own accounts because each time SteamCMD has to re-run, we're forced to put in our 2fa credentials each time. Plus, when you've done this 8 or 9 times, you'll get rate-restricted and have to wait another 30+ minutes until you can try the rest of your mods. I'm surprised this day and age that Valve have no way of setting up an access key for SteamCMD or something that allow us to restrict what it can do.
The fact that SteamCMD still cant take a list of mods and simply stay logged in and download them from the steam workshop in one go is very frustrating and hasn't changed in years.
@wakeuphate I can just wholeheartedly agree with everything. The mod i tried to download was CBA3 i think, so very small. And it is a timeout for the userinfo it seems. This is happens randomly throughout my testing. Sometimes is happens, sometimes not. Not sure why, or whats causing it.
But I will always get
Loading Steam API...../steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (117) : Assertion Failed: SteamDatagramTransportLock held for 190.1ms
There is just no usable API for steam :-/ As a workaround, I am creating a scrpipt, that can manually be triggered to download workshop stuff and put it on a ftp server, for my actual arma server to download from there....
In the end, it looks if retry the same error happens, but it will download the stuff. Kind of pray and repeat?
I can confirm that this does not appear to be related #6326 and that running the same command over and over did resolve the issue. I was using the below script workshop.sh that would loop through each workshop item, and attempt to download it. if one failed I could run workshop.sh 615007497 --only. I could also start form any point in the list by providing the last workshop item to fail workshop.sh 615007497. This worked pretty well, but was slow. In my case I only needed to authenticate once, and use SteamGuard once. For all subsequent executions it appeared to use cached authentication based on the provided user name.
/home/steam/scripts/workshop.sh
#!/usr/bin/env bash
set -e
set -o pipefail
WORKSHOP_ITEMS=(
787892271
463939057
773131200
773125288
884966711
708250744
1104460924
713709341
615007497
639837898
730310357
1537973181
642457233
849435425
886857451
820924072
450814997
837729515
724064220
871504836
583496184
583544987
882231372
686802825
333310405
743099837
366425329
433566590
648172507
694603075
1224892496
1494127420
873999690
520618345
909547724
736829758
860158903
823636749
1208517358
1400574293
1400566170
950966660
735566597
843425103
843593391
843632231
843577117
1368857262
498740884
699630614
1493291920
946763963
894678801
1206332670
784218341
949701797
949696541
940841212
1103511215
)
START_FROM=0
ONLY=0
if [[ -n "$1" ]]; then
found=0
for workshop_item in ${WORKSHOP_ITEMS[@]}; do
if [[ $1 -eq $workshop_item ]]; then
found=1
break
fi
done
if [[ $found -eq 1 ]]; then
echo "Starting from $1"
START_FROM=$1
if [[ $2 -eq "--only" ]]; then
ONLY=1
fi
else
echo "Provided workshop id $1 is not in list"
fi
fi
found_start=0
for workshop_item in ${WORKSHOP_ITEMS[@]}; do
if [[ $START_FROM -ne 0 ]]; then
if [[ $found_start -eq 0 && $workshop_item -ne $START_FROM ]]; then
continue
elif [[ $workshop_item -eq $START_FROM ]]; then
found_start=1
fi
fi
/usr/games/steamcmd +login user +workshop_download_item 107410 ${workshop_item} validate +quit | egrep "^(Downloading item ${workshop_item} ...|Success. Downloaded item ${workshop_item})"
if [[ $ONLY -eq 1 ]]; then
break
fi
done
Thanks to @SteelPhase I made a little something to download workshop stuff.
short version:
for workshop_item in "${WS_IDS[@]}"; do
modname="$(curl -s https://steamcommunity.com/sharedfiles/filedetails/?id="${workshop_item}" | grep "<title>" | sed -e 's/<[^>]*>//g' | cut -d ' ' -f 4-)"
modname_clean=$(echo "$modname" | dos2unix)
counter=1
printf "Downloading %s \n" "$modname_clean"
until steamcmd +login "${STEAMUSER}" "${STEAMPASS_decrypted}" +workshop_download_item 107410 "${workshop_item}" validate +quit; do
printf "Error Downloading %s. Will try again \n" "$modname_clean"
counter++
if ((counter > 4)); then
exit 1
fi
done
if [ ! -L "${home}/mods/@${modname_clean}" ]; then
ln -s "${home}/.steam/steamapps/workshop/content/107410/${workshop_item}/" "${home}/mods/@${modname_clean}"
fi
done
It basically tries again and again... not nice but eh
A bit more and explanation here:
https://github.com/michaelsstuff/Arma3-stuff/tree/22fde3f14590d0d52a2fb198334d250af20ebd34/mod-sync
Next step will be to upload it via ftp for other reasons... Thanks everybody for the ideas and feedback.
You could probably make this script a little smarter by monitoring the locations where the files are downloaded, and the location where the completed files are stored. Monitor the file size, and retry until it completes without error, or stop when the file size decreases. I've yet to run in to an instance where just running it over and over doesn't work.
This has been an issue with SteamCMD across platforms for atleast 5 years.
Only thing we've found is to do delete the .steam and Steam folders from the users directory. It has to be done for every single download, and doesn't always work.
@SteelPhase That kernel version would have been affected by the bug. Anyone else still getting bit by this should update their kernel, the fixes have been in most distros for a while now.
I confirmed earlier that my issues weren't related to that bug. i can try and reproduce again if need be. but it can't happen tonight.
@h1z1 I can tell you, I'm having this bug on about 30 machines, none of them are running the kernel you said and even after updating kernel it doesn't work.
Only fix that has been found is deleting .steam and Steam/ before running steamcmd every single time it needs to run. Which would indicate it's broke in a steamcmd update.
Not denying the message is valid rather there was a serious bug in the kernel version reported. I've run the script above for about a week out of curiosity and not hit that but then again I'm likely in a different region (NA). Can you throw up your logs anywhere?
I will try and set some time aside to retest this, and get the logs. It may take me a few days.
It happens with Space Engineers blueprints as well.
It also happens with the Windows version of steamcmd, so this is not Linux specific.
Always works: 2306920249
Microsoft Windows [Version 10.0.19041.685]
(c) 2020 Microsoft Corporation. All rights reserved.
C:\SEServer\steamcmd>steamcmd +"login anonymous" +"workshop_download_item 244850 2306920249" +quit
Redirecting stderr to 'C:\SEServer\steamcmd\logs\stderr.txt'
[ 0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.
Connecting anonymously to Steam Public...Logged in OK
Waiting for user info...OK
Downloading item 2306920249 ...
Success. Downloaded item 2306920249 to "C:\SEServer\steamcmd\steamapps\workshop\content\244850\2306920249" (1056527 bytes)
Never works, even if retried many times: 358485226
C:\SEServer\steamcmd>steamcmd +"login anonymous" +"workshop_download_item 244850 358485226" +quit
Redirecting stderr to 'C:\SEServer\steamcmd\logs\stderr.txt'
[ 0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.
Connecting anonymously to Steam Public...Logged in OK
Waiting for user info...OK
Downloading item 358485226 ...
ERROR! Download item 358485226 failed (Timeout).
Same error on Debian 10.5 Linux server. I don't repeat the output here. All are the same, only the path differs.
Nothing is logged into C:\SEServer\steamcmd\logs\stderr.txt, the file is empty!
A detailed error message why the download does not work would be highly useful.
Space Engineers (the game itself) can happily load both blueprints.
The main difference between them is that the non-working one is from 2014, while the working one is from 2019.
You got two actual examples now, this issue is quite easy to reproduce.
Come on Steam, you can fix it!
Or just give us plain HTTP(S) download URLs for the workshop files without the extra layer of difficulty.
You can cache those as well on the edge or just use a CDN. It won't be that expensive that way.
Anybody else: Could you please test the above two commands? Same behavior for you or both/none works? Thanks!
@viktor-ferenczi I was running into the same issue setting up a Space Engineers server, but I believe I've found a solution. It seems that some mods require SteamCMD to authenticate with a Steam account. Anonymous login just doesn't work with some mods, though I'm not sure why. I setup a separate Steam account just for my dedicated servers and that seems to have worked. The timeout error is very misleading here and a better error message could probably save a lot of folks some debugging.
I don't think this solves everyone's problems though. Looking at some of the other error message, I see assert statements regarding data packets. The errors that you and I were seeing don't include any of those assert statements, so I think they're separate issues masked by the same generic error message. Hopefully this helps at least a few folks get their mods downloaded.
That's definitely a good option to try, but I can say that I still ran into issues even with authentication. Not sure why it only helps sometimes either, but you may be on to something.
One thing that I've noticed (and this aligns with Viktor's post) is that the mods that require a Steam account were last updated before or during 2018. Any mods that were updated as of 2019 or later work with anonymous login. I'm wondering if something changed in the workshop around mid to late 2018 that caused this change in behavior. Just a hunch.
Thank you for the hint @Raznic, I will give the secondary account a try as soon as I get back the Torch based servers running after today's game update...
This bug occurs only with slow internet speed. I figured out that after this timeout occured, it still download this item in background. User has to wait before call next command. If it will not wait then next command will break background download and remove that downloading data from disk and start the execution of new command.
Its easy to reproduce: download item 583544987 by steamcmd (using interactive mode, not by script) with speed reduction to 1Mbits/sec. After about 10 minutes you will see something like that:
Steam>workshop_download_item 107410 583544987 validate
Downloading item 583544987 ...ERROR! Timeout downloading item 583544987
## HERE I'am waiting to finish steamcmd by looking on `sudo nethogs eth0` ##
Steam>workshop_download_item 107410 549676314 validate
Downloading item 549676314 ...
I have been struggling with this problem for 4 years.
It also happens on hosted dedicated server on a 1Gbps direct connection to the backbone. With plenty of CPU and RAM available.
I run a background process downloading Space Engineers blueprints on demand for pasting into a running game and that has this exact same issue on such a server. It depends only on the blueprint downloaded (workshop item) and seem to happen only to the older ones (sometime before 2018).
If this is a timeout due to slow download speed, then it may be that Steam is hosting the older data on dead slow or crowded edge servers or otherwise throttle their download. It would be nice to figure whether the slow speed is consistent, which would mean throttling.
I guess it would be possible to "patch" steamcmd and increase this timeout somehow, if this is not configurable.
The timeout period is counted from the start of the download. This should be counted from the moment of network inactivity. I think that the time limit is too short for certain items to download.
Maybe there is a way to disable the timeout entirely. It could be wrapped over it from the outside (equally badly) if really needed for hosted/backend scenarios. If not configurable, then may be there is a way to patch the binary to disable it.