504 Gateway Time-out on pocket import

[McMac]

Hello,

I get a 504 Gateway Time-out when trying to import from pocket:

1. launched wallabag docker: docker run -p 80:80 wallabag/wallabag
2. logged in using wallabag:wallabag
3. Config -> Consumer key for Pocket to import contents
4. Import -> Pocket: Import Contents
5. Dies during http://localhost/import/pocket/callback with error 504
6. After that the container does not respond anymore
7. Restart container: 380 entries added

::ffff:172.17.0.1 - - [19/Mar/2017:15:18:58 +0000] "POST /import/pocket/auth HTTP/1.1" 301 289 "http://localhost/import/pocket" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0"
2017/03/19 15:20:01 [error] 322#322: *58 upstream timed out (110: Operation timed out) while reading response header from upstream, client: ::ffff:172.17.0.1, server: _, request: "GET /import/pocket/callback HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "localhost"
::ffff:172.17.0.1 - - [19/Mar/2017:15:20:01 +0000] "GET /import/pocket/callback HTTP/1.1" 504 176 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0"

I have used pocket more as a bookmarks + content archive than actual read it (very much) later so over several years I added > 13k entries in pocket. I would like to extract them into wallabag and then from there sift through what's still valid or needs proper tagging.

Also tried but wallabag won't properly launch (obviously)

docker run -p 6379:6379 --name redis redis:alpine
docker run -p 80:80 --link redis:redis wallabag/wallabag import pocket

• Recreated the wallabag docker image without "import pocket"
• Enabled Redis async import within Internal Settings (pocket import configure in 3 different submenus?)
• again 504 error
• actually got no clue if and where I should provide the "import pocket" command.

The import never works/worked properly always dying of timeouts. The integration with redis also doesn't appear very stable enough.

Has anybody gotten the pocket import to work with more then 5 entries? The instructions on how to import using Redis seem to be broken as instructions state:

Import worker

To run the async redis import worker use the following command: $ docker run --name wallabag --link wallabag-db:wallabag-db --link redis:redis -e <... your config variables here ...> wallabag/wallabag import

As clearly at this point the container is already running or using that command only, the container won't launch. During first time setup nothing is set up for pocket import obviously.

Can anybody please enlighten me where I am going wrong?

[xsteadfastx]

i never used the importer. mmhhh.. maybe we should set up the timout higher for nginx to connect to php-fpm.

[Suncatcher]

I experienced the same error but all in all wallabag imported all articles in the end.

[DmitrySandalov]

Seems like fastcgi_read_timeout should be increased.

[bermudi]

Increasing fastcgi timout creates this new error.

500: Internal Server Error Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 10236384 bytes)

[j0k3r]

Increase memory limit then.

[bermudi]

gives same error even with memory limit -1

[xsteadfastx]

doesnt this look like a docker host problem?

[McMac]

The point being, this feature was probably tested with a few test records which for neither you would run into a timeout (lengthy response) or memory (too much data during response) issue.

The "easiest" fix would be to support a recovery (skip what was already imported - URL match? deduplication should be built-in IMHO - or simply define the db to make URL unique, then handle the respective db response in php) plus adding some sort of batching if the response is expected to be voluminous (not sure if pocket only streams the results or provides means to access the count). (get count, wrap current code into a for loop).

Currently, I do data migrations for a living but unfortunately not proficient enough in php to be of much help. :-S Maybe I need to jump into the deep end...

[viktormadarasz]

i have the same issue right now i have around 8000 entries in my Pocket. thing is pocket does offer an export into HTML but this as is can not be feeded into Wallabag, i guess if someone would know how to parse it and convert to a format like google s html exported bookmarks ( wallabag accepts those html formatting) it could be done in mass amount offline way ....

[JENkt4k]

Did this issue ever get fixed? still seeing it on 2.2.3?

[maltokyo]

I have a timeout on import too, not sure what to change

[gmolveau]

I'm still having this issue too.

[timespaced]

Same issue here. Importing 2200 Pocket articles results 504. I tried doing this over hours of refreshing and reimporting - it kept increasing the article count so I assumed it was picking up where it left off. Nope, TONS of duplicates.

[herrxyz]

Same issue here. Importing 2200 Pocket articles results 504. I tried doing this over hours of refreshing and reimporting - it kept increasing the article count so I assumed it was picking up where it left off. Nope, TONS of duplicates.

same problem here. prefer solution with loop as I installed walalbag on shared hosting without possibility to increase memory or timeout

[gmolveau]

@herrxyz did you find a solution with your "solution with loop" ? Thank's !

[herrxyz]

sorry, haven't tried further. As my php-skills are really low I'm just hoping for a wallabag update (instead of trying it itself)

[ghost]

Anyone has a workaround for the import? Or am I having a different problem?

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 131072 bytes) in /var/www/wallabag/vendor/j0k3r/php-readability/src/Readability.php on line 1434 [Symfony\Component\Debug\Exception\OutOfMemoryException]
Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate
131072 bytes) Exception trace: () at /var/www/wallabag/vendor/j0k3r/php-readability/src/Readability.php:1434 wallabag:import:redis-worker [--maxIterations [MAXITERATIONS]] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-e|--env ENV] [--no-debug] [--] <serviceName>

[j0k3r]

The best way to handle timeout error is to use import queue (using RabbitMQ or Redis). https://github.com/wallabag/docker#import-worker

[gmolveau]

Hello everyone, I created a python script to export your pocket items to wallabag : https://github.com/gmolveau/pocket_to_wallabag

Feel free to use it :) (also did a feedly script : https://github.com/gmolveau/feedly_to_wallabag)

[dotWee]

Hello everyone, I created a python script to export your pocket items to wallabag : https://github.com/gmolveau/pocket_to_wallabag

Feel free to use it :) (also did a feedly script : https://github.com/gmolveau/feedly_to_wallabag)

Works flawlessly, thanks for sharing!