Shiori extension never creates archive

[crawfordlong]

trafficstars

Data

• Shiori version: latest
• Database Engine: n/a
• Operating system:

server: ubuntu 22.10 client: fedora silverblue 37 + firefox

• CLI/Web interface/Web Extension: firefox extension

Describe the bug / actual behavior

Server is set up to create archives of saved bookmarks. Using the web interface, entering a new bookmark correctly creates an archive, pulls a banner image, etc. Using the referenced extension, however, the bookmark creates a plaintext bookmark (url and no banner image). It does not seem to be a matter of waiting for a job to execute, as I left a link for a few hours and it didn't pull the archive. Weirdly, this seems to conflict with another bug report.

Expected behavior

Clicking the shiori browser extension creates a bookmark and content archive.

To Reproduce

Steps to reproduce the behavior:

1. Described above.

Notes

Add any other context about the problem here.

[rmayergfx]

Try this one: https://github.com/go-shiori/shiori-web-ext This is the latest Version 0.8.6 and working fine for me Firefox Addon ist still not up to date. I hate users just uploading files to extension store and never updates them.

[Alistair1231]

The version I use has the same version number (0.8.6) and I still have this issue. When I noticed this yesterday and manually tried to create archives/retrieve thumbnails of the bookmarks from the last days/weeks where they were missing, it wouldn't work either.
After I restarted the container, with a docker pull for good measure, I could do manual archives again where they were missing, 5 at a time.

I just added a bookmark using the firefox addon and there is again only the tags and the link. This time the manual archive worked though.

docker log isn't very helpful, but here it is:

docker logs output

time="2023-05-21T10:33:25Z" level=info msg="GET /?search=office" proto=HTTP/1.1 remote="[IP ADDRESS]:36238" reqlen=0 size=5659 status=200
time="2023-05-21T10:33:25Z" level=info msg="GET /js/page/base.js" proto=HTTP/1.1 remote="[IP ADDRESS]:36238" reqlen=0 size=2627 status=200
time="2023-05-21T10:33:25Z" level=info msg="GET /api/bookmarks?keyword=office&tags=&exclude=&page=1" proto=HTTP/1.1 remote="[IP ADDRESS]:36294" reqlen=0 size=486 status=200
time="2023-05-21T10:33:25Z" level=info msg="GET /res/favicon-16x16.png" proto=HTTP/1.1 remote="[IP ADDRESS]:36294" reqlen=0 size=587 status=200
time="2023-05-21T10:33:25Z" level=info msg="GET /api/tags" proto=HTTP/1.1 remote="[IP ADDRESS]:36294" reqlen=0 size=3392 status=200
time="2023-05-21T10:33:50Z" level=info msg="POST /api/logout" proto=HTTP/1.1 remote="[IP ADDRESS]:51424" reqlen=0 size=1 status=200
time="2023-05-21T10:33:50Z" level=info msg="POST /api/login" proto=HTTP/1.1 remote="[IP ADDRESS]:51424" reqlen=53 size=141 status=200
time="2023-05-21T10:34:03Z" level=info msg="POST /api/bookmarks/ext" proto=HTTP/1.1 remote="[IP ADDRESS]:49552" reqlen=305773 size=353289 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /?search=" proto=HTTP/1.1 remote="[IP ADDRESS]:33432" reqlen=0 size=5659 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /api/bookmarks?keyword=&tags=&exclude=&page=1" proto=HTTP/1.1 remote="[IP ADDRESS]:33432" reqlen=0 size=15844 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /api/tags" proto=HTTP/1.1 remote="[IP ADDRESS]:33432" reqlen=0 size=3519 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /bookmark/139/thumb" proto=HTTP/1.1 remote="[IP ADDRESS]:33446" reqlen=0 size=11976 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /bookmark/138/thumb" proto=HTTP/1.1 remote="[IP ADDRESS]:33466" reqlen=0 size=6789 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /bookmark/137/thumb" proto=HTTP/1.1 remote="[IP ADDRESS]:33458" reqlen=0 size=50565 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /bookmark/136/thumb" proto=HTTP/1.1 remote="[IP ADDRESS]:33474" reqlen=0 size=9434 status=200
time="2023-05-21T10:34:21Z" level=info msg="GET /bookmark/140/thumb" proto=HTTP/1.1 remote="[IP ADDRESS]:33432" reqlen=0 size=133096 status=200
time="2023-05-21T10:34:32Z" level=info msg="POST /api/logout" proto=HTTP/1.1 remote="[IP ADDRESS]:42374" reqlen=0 size=1 status=200
time="2023-05-21T10:34:32Z" level=info msg="POST /api/login" proto=HTTP/1.1 remote="[IP ADDRESS]:42374" reqlen=53 size=141 status=200
time="2023-05-21T10:34:41Z" level=info msg="PUT /api/cache" proto=HTTP/1.1 remote="[IP ADDRESS]:42374" reqlen=55 size=9438 status=200
time="2023-05-21T10:34:41Z" level=info msg="GET /bookmark/165/thumb" proto=HTTP/1.1 remote="[IP ADDRESS]:42374" reqlen=0 size=37714 status=200

I can't find a way to increase the logging level. If you think that is helpful and can tell me how, I can post the output.

The network tab of the addon shows this request when I add something: https://pastebin.com/9QqiAV2w (pastebin becuase very long) The console of the addon doesn't show any errors

[rmayergfx]

No Problem here with synology docker, FF latest release with Extension 0.8.6. Be sure no Adblock or script is blocking. Also set "Create archive by default" on shiori Settings (Webpage) and allow all settings in Extensions. No problem. Do you have an example URL from an webpage not creating archive?

[Alistair1231]

No Problem here with synology docker, FF latest release with Extension 0.8.6. Be sure no Adblock or script is blocking. Also set "Create archive by default" on shiori Settings (Webpage) and allow all settings in Extensions. No problem. Do you have an example URL from an webpage not creating archive?

If it were an issue with content blocking, shouldn't it then not work manually either? Just to make sure I tried it again without adblock, same result. Here is the request from the addon including response again: https://pastebin.com/6DcFzR7h

Also, in the response of the HTTP request from the addon I see this at the bottom \"createArchive\":false}\n".

\"imageURL\":\"\",\"hasContent\":false,\"hasArchive\":false,\"tags\":[{\"id\":40,\"name\":\"wiki\"},{\"id\":87,\"name\":\"info\"}],\"createArchive\":false}\n"

It seems not to try to make an archive. The reponse also contains the entire HTML of https://en.wikipedia.org/wiki/Main_Page (what I tested it with)

These are my settings

I use this shiori.service file on Ubuntu 22.04 Server:

$  sudo cat /etc/systemd/system/shiori.service
# /etc/systemd/system/shiori.service
[Unit]
Description=Shiori container
After=docker.service

[Service]
Restart=always
ExecStartPre=-/usr/bin/docker rm shiori-1
ExecStart=/usr/bin/docker run \
  --rm \
  --name shiori-1 \
  -p 8080:8080 \
  -v /srv/shiori/shiori-data:/shiori \
   ghcr.io/go-shiori/shiori
ExecStop=/usr/bin/docker stop -t 2 shiori-1

[Install]
WantedBy=multi-user.target

Oh and here is the reverse proxy config (i use caddy)

$ sudo cat /srv/caddy/Caddyfile
{
        acme_dns cloudflare XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        log {
                output file /srv/caddy/logs/access.log
        }
}

example.com {
        reverse_proxy localhost:8080
}

[rmayergfx]

Sorry, no idea what is going wrong on your system. tested the given url from wikipedia an no trouble, all went fine and i have a working archive also:

[Alistair1231]

Is there a log I can provide somewhere? Also, is this correct? The server seems to think everytrhing goes fine, but for some reason it thinks that it shouldn't create an archive, even though that setting is active. At least that's my interpretation.

Here is the request from the addon including response again: https://pastebin.com/6DcFzR7h Also, in the response of the HTTP request from the addon I see this at the bottom \"createArchive\":false}\n".

\"imageURL\":\"\",\"hasContent\":false,\"hasArchive\":false,\"tags\":[{\"id\":40,\"name\":\"wiki\"},{\"id\":87,\"name\":\"info\"}],\"createArchive\":false}\n"

It seems not to try to make an archive.

[laalaguer]

Report the same issue here.

Server: v1.5.4 - binary

Client: Extension 0.8.6 in Firefox and tested with unpacked zip in Chrome (plain environment, no other extensions).

Url: https://book.douban.com/subject/35876121/

Error: Using ./shiori add on the server creates a correct cache page and archive page on the server. But using an extension just creates an entry + URL. No cache and no archive page were created.

Possible help: The database is upgraded as instructed in README from prior <1.5.0

[laalaguer]

Update:

I tried to down grade the server, using binary version 1.5.1 + 0.8.6 ext is correctly creating the bookmarks (although very slow).

[vensko]

It's an API issue, /api/bookmarks/ext is broken. /api/bookmarks works as intended. They use the same payload, so even simple update to use /api/bookmarks with added createArchive: true fixes the issue https://github.com/go-shiori/shiori-web-ext/blob/master/js/background-script.js#L195

[edsu]

It seems like the API call to create the bookmark is not specifying whether to archive it or not?

https://github.com/go-shiori/shiori-web-ext/blob/f043c1364b22b8fc9514ed4b2b5d07b12a102630/js/background-script.js#L137-L148

Maybe the server side API should use the default when the archive option is unspecified? It seems like it is defaulting to false even when it is set to true on the server?

[crabvk]

Using the latest shiori server and the latest version of extension 1.7.4.0 on Chromium 139.0.7258.66 and it always creates archives. I've set "Create archive by default" to unchecked on the server, but it does not make any difference.