server feat: Add support for webhook listeners

Summary

Add support for registering webhooks as listeners of events.

Checklist

Code is properly formatted
Sign-off message is added to all commits
[ ] Tests (unit, integration, api and/or acceptance) are included
[x] Screenshots before/after for front-end changes
[ ] Documentation (manuals or wiki) has been updated or is not required
[x] Backports requested where applicable (ex: critical bugfixes)

May 23 '24 09:05 come-nc

Code looks good! How will this be used?

Plan is to use that for integration of an external workflow engine, but architecture is not final yet. One problem is we’d prefer if the webhook gets called only on certains conditions, but then we end up with apps/workflowengine complexity.

Can we assume that apps know their webhook listeners when the app is just registered and the system isn't fully booted yet?

Hum, what is not accessible at that point? If I want to have an app which search in its database and user settings to know which webhooks to register, will that work?

May 23 '24 15:05 come-nc

Can we assume that apps know their webhook listeners when the app is just registered and the system isn't fully booted yet?

Hum, what is not accessible at that point? If I want to have an app which search in its database and user settings to know which webhooks to register, will that work?

That works. Anything that is provided by other apps might be missing. Say information from a user backend.

If you always want to wire a specific event to webhook listeners it's fine. But the lack of this information could be critical when you need to conditionally register webhooks. I haven't yet understood the full picture of the new automations so this might not be necessary.

https://docs.nextcloud.com/server/latest/developer_manual/app_development/bootstrap.html#nextcloud-20-and-later for my original ideas and thoughts about the two step register/boot process.

May 23 '24 15:05 ChristophWurst

Webhooks are moved to an application with registrations in DB instead. They will also need to support filtering of events triggerring the webhook.

[x] Skeleton app
[x] API to register webhooks
[x] occ command to list them
[x] Actually call the webhooks, through background jobs
[x] Serialize the event to the webhook
[x] Add a filtering system like https://github.com/Akkroo/PHPMongoQuery/blob/master/src/Akkroo/PHPMongoQuery.php
[x] Switch to SPDX headers
[x] Add auth methods
[x] Encrypt auth info
[x] Remove core webhook registration
[x] Validate data passed to the API for webhook registration

May 28 '24 10:05 come-nc

[x] Store app id from appapi header if available

May 28 '24 14:05 come-nc

To test current version: 1 - register a webhook, for instance:

<?php
function callNextcloudApi(string $endpoint, array $postfields): void {
		$uri = 'http://nextcloud.local/';

		$ch = curl_init();

		curl_setopt($ch, CURLOPT_URL, "$uri/$endpoint");
		curl_setopt($ch, CURLOPT_POST, 1);
		curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($postfields));
		curl_setopt($ch, CURLOPT_HTTPHEADER,
			[
				'Accept:application/json',
				'Content-Type:application/json',
				'OCS-APIRequest: true',
				'Authorization: Basic '. base64_encode('admin:admin')
			]);
		curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);

		// Receive server response ...
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

		$server_output = curl_exec($ch);
		curl_close($ch);

		var_dump($server_output);
}

callNextcloudApi(
	'/ocs/v2.php/apps/webhooks/api/v1/webhooks',
	[
        'httpMethod' => 'POST',
        'uri' => 'http://127.0.0.1/hook.php',
        'event' => 'OCP\\Files\\Events\\Node\\NodeWrittenEvent',
        'eventFilter' => [
          'user.uid' => 'admin',
        ],
        'headers'		=> ['Header1' => 'value1'],
        'authMethod'	=> 'header',
        'authData'		=> ['Header2' => 'value2'],
	]
);

2 - react to webhook calls in your hook.php:

<?php

$filename = '/tmp/record.txt';

function callNextcloudApi($fp, string $endpoint, array $postfields): void {
	try {
		$uri = 'http://nextcloud.local/index.php/';

		$ch = curl_init();

		curl_setopt($ch, CURLOPT_URL, "$uri/$endpoint");
		curl_setopt($ch, CURLOPT_POST, 1);
		curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($postfields));
		curl_setopt($ch, CURLOPT_HTTPHEADER, ['Authorization: Basic '. base64_encode('admin:admin')]);
		curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);

		// Receive server response ...
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

		$server_output = curl_exec($ch);
		curl_close($ch);

		var_dump($server_output);

		if (fwrite($fp, $server_output."\n") === false) {
			die("Cannot write in ($filename)");
		}
	} catch (\Throwable $t) {
		if (fwrite($fp, $t."\n") === false) {
			die("Cannot write in ($filename)");
		}
	}
}

if (is_writable($filename)) {
	if (!$fp = fopen($filename, 'a')) {
		die("Cannot open ($filename)");
	}

	$event = json_decode(file_get_contents('php://input'), associative:true);

	if (fwrite($fp, file_get_contents('php://input').' - '.json_encode($_POST)."\n") === false) {
		die("Cannot write in ($filename)");
	}

	if (isset($event['event']['node']['id']) && isset($event['event']['node']['path']) && isset($event['user']['uid'])) {
		callNextcloudApi(
			$fp,
			'/apps/tables/api/1/tables/2/rows',
			[
				'data' => json_encode(
					[
						5 => $event['event']['node']['id'],
						6 => $event['event']['node']['path'],
						7 => $event['user']['uid'].' - '.($_SERVER['HTTP_HEADER1'] ?? '').' - '.($_SERVER['HTTP_HEADER2'] ?? ''),
						8 => time(),
					]
				)
			]
		);
	}

	fclose($fp);
} else {
	die("Cannot open ($filename) - no permission");
}

(this one needs a /tmp/records.txt writable file, and tables application with a table with id 2 and columns with id 5/6/7/8, adapt to your needs) 3 - You can check registered webhooks with occ webhooks:list

$ occ webhooks:list
+----+-------+--------+------------+---------------------------+----------------------------------------+----------------------+----------------------+------------+----------+
| id | appId | userId | httpMethod | uri                       | event                                  | eventFilter          | headers              | authMethod | authData |
+----+-------+--------+------------+---------------------------+----------------------------------------+----------------------+----------------------+------------+----------+
| 1  |       | admin  | POST       | http://127.0.0.1/hook.php | OCP\Files\Events\Node\NodeWrittenEvent | {"user.uid":"admin"} |                      | none       |          |
| 2  |       | admin  | POST       | http://127.0.0.1/hook.php | OCP\Files\Events\Node\NodeWrittenEvent | {"user.uid":"admin"} | {"Header1":"value1"} | header     |          |
+----+-------+--------+------------+---------------------------+----------------------------------------+----------------------+----------------------+------------+----------+

4 - You can check scheduled jobs with occ background-job:list --class="OCA\Webhooks\BackgroundJobs\WebhookCall"

$ occ background-job:list --class="OCA\Webhooks\BackgroundJobs\WebhookCall"
+-----+-----------------------------------------+---------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id  | class                                   | last_run                  | argument                                                                                                                                                                                        |
+-----+-----------------------------------------+---------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 104 | OCA\Webhooks\BackgroundJobs\WebhookCall | 1970-01-01T00:00:00+00:00 | [{"event":{"class":"OCP\\Files\\Events\\Node\\NodeWrittenEvent","node":{"id":185,"path":"\/admin\/files\/New text file.md"}},"user":{"uid":"admin","displayName":"admin"},"time":1717425671},1] |
+-----+-----------------------------------------+---------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Jun 03 '24 15:06 come-nc

Follow-up:

[ ] Extract user filter from eventFilter so that the listener is not even registered depending on logged in user
[ ] Same for group membership

Jun 04 '24 12:06 come-nc

I'm unsubscribing from this PR for now, please ping me if you need a review of the (Open)API.

Jun 06 '24 09:06 provokateurin

It seems the node events that you added the interface to are not actually emitted at the moment. I can only see the old \\OCP\\Files::preWrite ones

Jun 10 '24 09:06 marcelklehr

It seems the node events that you added the interface to are not actually emitted at the moment. I can only see the old \\OCP\\Files::preWrite ones

They are, I tested with a NodeWrittenEvent and the webhook was called. Please detail what scenario you tested and which event firing was missing and I’ll look into it.

Jun 10 '24 12:06 come-nc

[x] webhooks app id is already used on appstore, we should rename.

Jun 10 '24 12:06 come-nc

They are, I tested with a NodeWrittenEvent and the webhook was called. Please detail what scenario you tested and which event firing was missing and I’ll look into it.

Nevermind, as discussed, it was the leading backslash that caused this

Jun 11 '24 08:06 marcelklehr