ryanblenis
Scripttask causing MC remote connections to freeze every minute.
Hi @ryanblenis or anyone else who might have an idea of how to fix this. I noticed that my remote sessions were "freezing" or becoming unresponsive for a few seconds every minute. This coincided with the following journal error (also every minute):
Aug 22 11:37:05 control node[1136]: PLUGIN: ScriptTask: Could not dispatch job. TypeError: Cannot read property 'content' of undefined at /home/xyz/meshcentral-data/plugins/scripttask/scripttask.js:72:44
I was able to confirm this was the issue by stopping MC, renaming the scripttask folder, and restarting. Once I did this, the errors stopped being logged and my sessions stopped freezing.
I'm guessing that maybe this is pointing to a corrupt task or something? Does anyone have any idea (other than deleting all tasks) how I can work out which one is corrupt (if that's the problem) and fix it?
Thanks in advance.
OK. So the good news is I managed to get rid of the error be manually deleting the orphaned scripts in plugin-scripttask.db. However, the bad news is that wasn't the freezing problem... I'm guessing both were occurring every 60 seconds because that's when scripttask polls or something. So I'm back to square one.
OK. Another update. It seems that Scripttask is making my CPU hit 100 every minute:
vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 69888 443196 127024 1099592 0 0 9 124 0 7 9 2 89 0 0 0 0 69888 443060 127024 1099592 0 0 0 0 148 269 1 0 99 0 0 1 0 69888 443028 127024 1099628 0 0 0 0 136 243 2 1 97 0 0 1 0 69888 443028 127024 1099628 0 0 0 0 124 235 0 0 100 0 0 0 0 69888 443060 127024 1099628 0 0 0 0 159 294 6 1 93 0 0 1 0 69888 443548 127024 1099628 0 0 0 0 133 320 3 1 96 0 0 2 0 69888 432412 127032 1099776 0 0 0 24 369 2708 94 6 0 0 0 1 0 69888 422384 127032 1099776 0 0 0 0 351 645 97 3 0 0 0 1 0 69888 442968 127032 1099776 0 0 0 0 355 691 98 2 0 0 0 1 0 69888 442720 127032 1099776 0 0 0 0 351 696 100 0 0 0 0 0 0 69888 442256 127032 1099776 0 0 0 0 394 542 42 3 55 0 0 0 0 69888 442256 127036 1099772 0 0 0 28 160 295 2 1 97 0 0 0 0 69888 441824 127036 1099776 0 0 0 0 144 305 1 2 97 0 0 0 0 69888 441824 127036 1099776 0 0 0 0 122 232 1 0 99 0 0 0 0 69888 441824 127036 1099776 0 0 0 0 125 243 0 0 100 0 0
I haven't heard of anything like this before, but what happens when if you back up your scripttask db and remove / re-add with a fresh database? I say that because I don't know where you found or how you had "orphaned" scripts in the db (unless you just mean unused / unscheduled). Alternatively, removing scripts that you have scheduled to run once per minute until you find the culprit.
Actually, just looking for a script in the DB without a "content" field, or a content field that is null, given the initial error you listed as it's choking at the script content being nonexistent.
Hi @ryanblenis Thanks for the fast reply! Yup that is what I did. I basically searched in plugin-scripttask.db for ones missing content and deleted them. That fixed the error that I was getting in the journal. But the error wasn't actually related to the CPU spike (even though they were both occurring every 1 minute) even with the error gone, the CPU spike was still occurring.
I've deleted the scripttaks plugin. Now I'll delet the .db file and reinstall and see what happens.
OK. So I've reinstalled the plugin with a clean db and am not getting the issue. I'll re-upload my scripts. So you think what probably happened is I must have accidentally scheduled a script to run every minute and it was smashing the CPU for a few seconds every minute?
Well that's hard to say as I've never quite seen that issue and don't know how to repeat it (other than you sharing your scripttask db and taking a look, possibly). Yes, once per minute the script scheduler runs and checks for scripts that need to be scheduled out to endpoints. It certainly shouldn't spike CPU I wouldn't think unless you have an underpowered server and a large amount of scripts and schedules, in which case, I'd want to see what's going on and optimize the routine if possible.
The original error was trying to schedule a script with no content, which really shouldn't have happened but I also haven't tested creating a script, NOT adding content to it, and scheduling against endpoints, because that would be odd. Please let me know any additional info you can to help.