workflow_script
workflow_script copied to clipboard
nextcloud
Run script immediatly after trigger
Currently the scripts are run with the next cron job. This is not usable for some cases, e.g. I try to lock a file with an other app if it's opened or updated. I don't want to trigger cron any 10 secounds because there are other jobs like generating previews and file scan etc. Other APP like https://github.com/nextcloud/files_accesscontrol are grip immediatly. How can I configure this direct behavior with workflow_script?
files_accesscontrol addresses a very different use case. workflow_script always triggers the script per background job, currently.
Yes, it's a different use case. But the lock strategie in Nextcloud is not useable in corporate environment and I try to workaround this failure. Currently I use the workflow to trigger a script if a file is opened or updated. The script uses the app https://github.com/nextcloud/files_lock to lock the file. Automated unlock time is 5 min. This seems to be a good solution if workflow_script would be able to run the script immediatly.
I did not find a better solution. The currently lock strategie is unable to handle locks and migrations of multiple users on multiple access ways to the files (Web, WebDAV, Client).
files_lock could be extended to offer a flow actions itself and act accordingly.
Putting aside the specific use case above, it would be handy to run scripts immediately after the trigger. When I discovered this app I thought "great, I can do faster push processing instead of pull processing". But this app at the moment just queues itself on the pull-queue that is cron. It would be almost as easy to write my own cronjobs that just enforce their own quit-early event filtering and have the same performance.
In my case I am trying to provision some credentials when new users get created, and I think I can do it with this app, but I don't want my HR people to be waiting around for 5 minutes for cron to wake up and run the triggers when they're trying to onboard someone.
My workaround for the moment is replace this cronjob
*/5 * * * * php -f /var/www/nextcloud/cron.php
with
* * * * * php -f /var/www/nextcloud/cron.php
which I expect will have some negative consequences down the road.
Currently easiest thing is having a timer/cron/daemon running that looks for registered jobs and executes them. The latter can be achieved with something like:
php occ background-job:list --output json --class 'OCA\WorkflowScript\BackgroundJobs\Launcher' | jq -r .[].id | xargs -n1 php occ background-job:execute
What's the reason (the background) that the workflow job(s) are not executed immediately? Is this to prevent that that the Nexcloud engine gets blocked by a long running script?
Just an idea: Would it be possible that the workflow engine touches a file, when the engine determines that a workflow_script is to be executed? The file should be touched in a directory that is watched by inotify.
As soon as the file is touched inotify comes into action and execute the workflow scripts.
This script could figure out with the command mentioned in the previous comment the jobs that need to be run.
The directory that could be used as an example is, I think, /run/nextcloud/workflow-scripts (on linux).
Perhaps, multiple files must be created with each file containing the job-name-id that is to be executed. When that particular job has been executed the related file must be removed.
Would something like this be possible (this requires setting up inotify, but with a good procedure that may not be too difficult)?
As soon as the file is touched inotify comes into action and execute the workflow scripts. This script could figure out with the command mentioned in the previous comment the jobs that need to be run.
Hmmm, I think it should just run the "nextcloud-x.y.z/cron.php" job, that is now run from the cronjob.
Is this to prevent that that the Nexcloud engine gets blocked by a long running script?
Yes, and potential Exceptions that might be thrown.
The file should be touched in a directory that is watched by inotify.
You could watch the database file for oc_jobs.
You could watch the database file for oc_jobs.
How would that be done?
inotify (inode notification) is a buildin Linux kernel module that gets triggered when a file is created, modified, deleted, etc. There is no polling needed. One sets a watch on the directory and as soon as a file in the directory is added, modified or deleted, the watcher activates the specified/configured script.
The script can then start to look for Nextcloud jobs. And as soon as these appear in the Nextcoud table ( I assume oc_jobs), the jobs can be processed (occ cron background-job:list....)
So yes, use oc_jobs but after inotify is triggered by Nextcloud.
May depend on the database you use, but for instance MySQL/MariaDB stores each table in a single file, so you can watch this one for changes. Something like /var/lib/mysql/nextcloud/oc_jobs.ibd.
for instance MySQL/MariaDB stores each table in a single file, so you can watch this one for changes Nice one. I didn't realize that this was possible. I run only an small Nextcloud setup (with sqlite3), hence for me this does now work. I don't know about the creator of this ticket, whether he/she can watch the database files. But would this work with other database as well, like postgress?
Should each job in the oc_jobs.ibd table result in a trigger, or should there only be a trigger for certain tasks/jobs in the script workflow?
Would a DB engine independent solution, be possible and perhaps would that be better?
For sqlite you could only watch the database file itself which would be too noisy i guess. Not sure about postgres, i have not seen dedicated files there, but I did not spend too much time on it.
A different thing doable it to outsource this mechanism into another app. We could dispatch an Event on registering those jobs, and a dedicated app can touch a file or run some other action.
@blizzz thanks for the followup and idea. Another app sounds rather heavy to me, for what I think is something simple. What is against it to touch a file on the file system? Touching a file, can be started with besides dispatching an event. Isn't it?
A dedicated app can be created to process the dispatched event later, in the meantime people can use the touched file using inotify.
In case of dispatching an event, must the event be removed? Or will that be taken care of automatically?
Perhaps for both cases (dispatching an event, and creating a file) a configuration option can be created where the admin can enable the desired mode?
And there it is the other app: https://github.com/icewind1991/files_inotify but i don't know if it is suitable for the case discussed above.
@radoeka did you find a solution or practical workaround for this issue?
@XueSheng-GIT no. I'm still checking every 5 minutes if there is something to update. It would be better with an inotify solution.
@radoeka I'm using postgresql as database and had a look into trigger functions (https://www.postgresql.org/docs/current/plpgsql-trigger.html).
As an initial proof of concept, I added a trigger and trigger function for the oc_jobs table on INSERT (class OCA\WorkflowScript\BackgroundJobs\Launcher). Job id is sent via pg_notify to a listener script (python) which executes the job via php occ background-job:execute.
First tests were successful and jobs are immediatly executed.
@radoeka since NC30 there's occ background-job:worker which also should solve your issue.
The following reference (related to Nextcloud AI) can be taken to create a systemd service for this worker. But you need to adapt the required job-classes.
https://docs.nextcloud.com/server/latest/admin_manual/ai/overview.html#ai-overview-improve-ai-task-pickup-speed
You can do a simple test on console first to see if your workflow scripts are triggered immediately. e.g.:
sudo -u www-data occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob' 'OCA\WorkflowScript\BackgroundJobs\Launcher'