Batch add recipes

[christianlupus]

Is your feature request related to a problem? Please describe.

From time to time I find some recipes I want to add to the app's storage. Typically this happens in batches as if I start to dig into some recipe databases, I do find quite some recipes I find interesting. I now have to add all the recipes URL by URL waiting for the cloud to fetch and store the relevant data until I can enter the next URL.

Describe the solution you'd like I would like to have an option to enter a list of URLs (textarea). This should then be parsed line by line and imported as best as it can do now. To avoid issues with timeouts, I suggest to simply do this on the client side. So a small JS script reads the textarea line by line and calls AJAX requests just as in the current implementation for the single-URL import. From the server side no change is needed. It is merely a UX issue with a small script to automate it.

Describe alternatives you've considered Except for manually adding the recipes, I have not had a good idea. Maybe, one could use a local file + a bash script calling a series of curl invocations to simulate the REST calls but that's all I could think of at the moment.

[mrzapp]

@sam-19 @mcorteel should we replace the input field + download button with a modal for this one?

[charles-997]

My coding knowledge is nearly non-existent, most of my Github activity is reading and learning but....

Does this cookbook app have any type of CLI that supports the import recipe feature?

If so wouldn't a very simple shell script be able to achieve this? It may not be the best solution, but for an app like this with a small community, it may be a workable intermediary solution until a more polished & permanent method is found.

[christianlupus]

Unfortunately no, there is no cli. I checked the docs and found no way to add some commands to occ.

This would also require shell access which is not given for quite some installations.

[mcorteel]

@christianlupus You're right, this is not documented, but it's pretty straightforward. For an example, you can see the nextcloud/news repository.

1. Create your command in lib/Command that implements Symfony\Component\Console\Command\Command (see https://github.com/nextcloud/news/blob/master/lib/Command/ExploreGenerator.php)
2. Register your command in the info.xml <commands> section (see https://github.com/nextcloud/news/blob/master/appinfo/info.xml).

[christianlupus]

@mcorteel Thanks for clarifying. I was not aware of that. Nice to know that.

About the main issue: Another (yet completely independent from the app) approach would be to use curl in a local (bash) script to iterate over all to be imported URLs and push them to the /import route. The only thing to be considered there is the authentication issue that needs to be covered. But that might be managible.

[geeseven]

use curl in a local (bash) script to iterate over all to be imported URLs and push them to the /import route. The only thing to be considered there is the authentication issue that needs to be covered.

I looked into this, but it looks like /import is not publicly available, for lack of a better term. In this example AUTH is echo -ne "user:password" |base64 --wrap 0.

$ curl \
--request POST "$HOST"/index.php/apps/cookbook/import \
--header "Authorization: Basic $AUTH" \
--header "Content-Type: application/x-www-form-urlencoded" \
--data "url=https://www.foodnetwork.com/recipes/food-network-kitchen/chicken-fettuccine-alfredo-3364118"
"This request is not allowed to access the filesystem"

It also returned a 500 status code. I am able to add this recipe via the web interface.

[christianlupus]

@geeseven Apart from the fact that I was not able to fetch any recipe from the URL in the command, I was able to import via command line interface.

First, I need a valid app token. Youi can create it manually or programmatically. The programmatic approach is described in the NC docs. Fo me this is

$ curl -X POST http://localhost:8000/index.php/login/v2 | jq
{
  "poll": {
    "token": "Jbqn30URQmZ7mOL08yG2LDXlIHUobSLVmjYY0uUBFCYTCTG30TampqCNLdTEbYrgSYzjzb8XtSK1PWsHO3i18l4FBcotKv7r1cXeDDnE4Cwjp5U5dKieje92RsUaHsKw",
    "endpoint": "http://localhost:8000/login/v2/poll"
  },
  "login": "http://localhost:8000/login/v2/flow/WxcbvJqUng1xXwfhWGUzuXnb2EyBVkKmmMErRkwFFuzINUAIQJgvrOiunoOIT0USy0wYVxGAhAQX9nfVZbhYZs6UpbwyNtrU4O6BHohJhiHOTC0lSdz7aRal8UkH7Alf"
}

I open the required URL in a browser (or instruct the user of my script to do so). Meanwile, I can try to fetch some data from the endpoint provided (see docs). Once the user has logged in, I get

$ curl -X POST 'http://localhost:8000/login/v2/poll' -d 'token=Jbqn30URQmZ7mOL08yG2LDXlIHUobSLVmjYY0uUBFCYTCTG30TampqCNLdTEbYrgSYzjzb8XtSK1PWsHO3i18l4FBcotKv7r1cXeDDnE4Cwjp5U5dKieje92RsUaHsKw' | jq
{
  "server": "http://localhost:8000",
  "loginName": "admin",
  "appPassword": "cJeyq7OyZu7uDftdKtsnuROVeAGt3CueTWDTV7SxLXkIg5hg10EIaXPNjTLqBkQAEuY1W5nj"
}

I need to save/store the app password somewhere.

Now, I can use bearer authentication to import a single recipe.

curl -X POST \
-H 'Authorization: Bearer cJeyq7OyZu7uDftdKtsnuROVeAGt3CueTWDTV7SxLXkIg5hg10EIaXPNjTLqBkQAEuY1W5nj' \
-d 'url=https://www.chefkoch.de/rezepte/1807401292413281/Zwiebelsuppe-nach-Art-der-legendaeren-Pariser-Marktfrauen.html' \
http://localhost:8000/apps/cookbook/import
{...}

As a result, I get the recipe as JSON+LD.

Of course the app password I do not need to recreate each time but only once.

[geeseven]

Apart from the fact that I was not able to fetch any recipe from the URL in the command

I should have not included a foodnetwork.com recipe in my example. We learned in #115, that site is very different in different regions.

Are you using Nextcloud's api instead of the cookbook app api?

[geeseven]

"This request is not allowed to access the filesystem"

I released that I was using an 'app password' where I had removed 'Allow filesystem access'. Once I enabled that, it works!

$ curl -s \
> --user user:password \
> --request POST https://example.com/index.php/apps/cookbook/import \
> --header "Content-Type: application/x-www-form-urlencoded" \
> --data "url=https://www.chefkoch.de/rezepte/1807401292413281/Zwiebelsuppe-nach-Art-der-legendaeren-Pariser-Marktfrauen.html" \
> |jq .name
"Zwiebelsuppe nach Art der legendären Pariser Marktfrauen"

[geeseven]

@christianlupus

$ ./import-cookbook.sh temp 
https://www.chefkoch.de/rezepte/1807401292413281/Zwiebelsuppe-nach-Art-der-legendaeren-Pariser-Marktfrauen.html added
https://www.chefkoch.de/rezepte/1807401292413281/Zwiebelsuppe-nach-Art-der-legendaeren-Pariser-Marktfrauen.html "Another recipe with that name already exists"
https://example.com "Could not find recipe element"

$ cat temp 
https://www.chefkoch.de/rezepte/1807401292413281/Zwiebelsuppe-nach-Art-der-legendaeren-Pariser-Marktfrauen.html
https://www.chefkoch.de/rezepte/1807401292413281/Zwiebelsuppe-nach-Art-der-legendaeren-Pariser-Marktfrauen.html
https://example.com

$ cat import-cookbook.sh
#!/usr/bin/env bash
HOST="https://nextcloud.example.com"
USER="user"
PASSWORD="password"
# If using an 'app password' ensure 'Allow filesystem access' is enabled.

IFS=""

while read -r url
do
	result=$(curl -s \
		--user "$USER:$PASSWORD" \
		--request POST "$HOST"/index.php/apps/cookbook/import \
		--header "Content-Type: application/x-www-form-urlencoded" \
		--data "url=$url")
	if [[ ${result} = \{\"\@context\"* ]] ; then
		echo "$url added"
	else
		echo "$url ${result}"
	fi
done < "$1"

$ shellcheck import-cookbook.sh
$

@mrzapp, What do you think about adding this script into the repository?

[mrzapp]

@geeseven my personal opinion is that scripts should be included in the standard fashion, runnable with occ. But that script is really useful for now, so maybe we could add it to a wiki page and then link to that page from the FAQ section of the README?

[christianlupus]

I recently asked at the NC server team. They have a (valid) remark on that: occ is not reachable always. First, for non-shell installations (aka pure webspace) you cannot call commands on the server. Second, you need to grant the user access on shell/ftp level. That includes access to all files of all users, PHP,... A huge security issue. The provided script runs on the client and thus and security is intact.

Either we put it on a wiki page or directly in the documentation folder and link there.

[mrzapp]

@christianlupus good point. Let's put it in the documentation folder, I think that makes more sense than the wiki

[geeseven]

@mrzapp, I agree that this script should only be used as a stop gap till a proper solution is developed.

Where should the documentation folder be located?

[mrzapp]

@geeseven a /documentation folder seems appropriate, I'd say, unless @christianlupus had something else in mind

[christianlupus]

I am perfectly fine with a /documentation folder. Could you open a PR @geeseven writing the required steps down and/or a bash script?

[geeseven]

I am starting to have mixed thoughts on a /documentation folder, if it is only going to contain a stop gap script. Kinda seems misleading to me.

What about something like /scripts, /tools or /utilities? Maybe append 'client-' to those names? This folder would contain a README with usage information, my import script, and a script version of my export one-liner from #264.

[christianlupus]

I am more thinking towards #294, #365. Also a general user usage documentation is missing at the moment. So I consider this issue here just one part of many. Therefore I suggested to keep it in the /documentation folder.

[geeseven]

@christianlupus, I agree that proper documentation is needed, but I am just not sure this is the best starting point. If you got something in mind, feel free to get the ball rolling. I do not care about credit for this script. It was something I whipped up for you while working on a qutebrowser userscript. I can add a recipe with a single keystroke. :grin: