hyextract
hyextract copied to clipboard
floogulinc
Attempting to connect from a remote computer but recieving error 400
I am currently running hydrus from a docker. When attempting to remotely connect with hyextract on a mac and on a windows PC, I receive an error 400. I have confirmed that my API URL and apikey are correct and the API port is open (as I am using hydrus-web and it is currently working, using a different apikey).
Generally hyextract is not supported on remote machines. It relies on running on the same filesystem as the Hydrus client itself. More specifically, it imports files by requesting the client import the path it sees for each file in the temp folder so those paths need to be the same and accessible by both hyextract and the client.
I suggest running hyextract on the same host as the Hydrus client which in your case would be in the docker container. Or if you can get the paths to be the same you can probably bring hyextract in it's own docker container just fine.
Additionally I have been considering making hyextract directly get zips and POST files back to the client which would allow it to be more portable.
I have installed npm and hyextract into my current hydrus docker container but even locally I am getting a connection error 400 response back. Edit I should be more specific, here is the entire output: API Permissions (hyextract): add tags to files, add urls for processing, import files, manage cookies, manage database, manage pages, search for files: Can search: all tags Error: Request failed with status code 400
Since it seems to read the API permissions correctly I am assuming that it is connecting, but since it doesn't show what its trying to do I'm not sure what is failing.
I turned on logging on hydrus and these are the messages it shows when running hyextract: 2021/12/08 12:58:46: 45869 GET /verify_access_key 200 in 2.00 milliseconds 2021/12/08 12:58:46: 45869 GET /get_files/file_metadata 400 in 3.00 milliseconds
It appears the above error was due to me exporting tags at some point on a file, does not look like hyextract can handle other files in the directory, (which makes sense).
I got further after install 7z into the docker but it looks like when its attempting to import I get a 400 error: moving unimported file /tmp/hyextract/7e91fe764f1b589f9791316f2f054fa78236c319dd3aa13b6535b6e6ded035c1/rinko_un/rinko8.jpg to /opt/hydrus/db/hyextract/unimported/7e91fe764f1b589f9791316f2f054fa78236c319dd3aa13b6535b6e6ded035c1/rinko_un/rinko8.jpg adding /tmp/hyextract/7e91fe764f1b589f9791316f2f054fa78236c319dd3aa13b6535b6e6ded035c1/rinko_un/rinko9.jpg to hydrus Error: Request failed with status code 400
I believe I found the issue, I needed to move my tmp directory, also I had a typo in the name of the tag portion of the config. Now it appears to be working as intended.
Unfortunately, after a server issue, I had to recreate my Hydrus instance. I am back to having the error 400, and I'm not sure how to fix it. is there any way to get more error messages from hyextract?
the errors looks like this in the hydrus.log:
POST /add_files/add_file 400
This likely means Hydrus can't access the temp directory hyextract has extracted the file to. Can you post your hyextract config here (censoring any sensitive info) or DM it to me on Discord?
This is what the config looks like, /hyextract is mapped via docker: { "hydrusApiUrl": "http://localhost:45869", "hydrusApiKey": "api_key_with_full_rights", "archivesDirectory": "/hyextract/archives", "tempDirectory": "/tmp/hyextract", "copyTags": true, "copyUrls": true, "tagServices": [ "my tags", "downloader tags" ], "passwordNamespace": "password", "passwordHexNamespace": "password hex", "tagBlacklist": [], "namespaceBlacklist": [ "filename", "password", "password hex" ], "tagFilenames": true, "filenameTagService": "my tags", "deleteOriginalArchiveFromDirectory": true, "deleteOriginalArchiveFromHydrus": true, "deleteTempFiles": true, "customServicesToTags": { "my tags": [ "hyextract" ] }, "moveUnimportedFiles": true, "unimportedFilesDirectory": "/hyextract/unimported" }
Are you running hyextract in the same docker container as Hydrus? Eg does Hydrus have access to /tmp/hyextract?
Yes i install it directly within then docker
Apr 14, 2022 7:35:13 PM Paul Friederichsen @.***>:
Are you running hyextract in the same docker container as Hydrus? Eg does Hydrus have access to /tmp/hyextract?
— Reply to this email directly, view it on GitHub[https://github.com/floogulinc/hyextract/issues/2#issuecomment-1099719948], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AASR7CDRW4APMXOF2MB7ORDVFCTS5ANCNFSM5JS2CONQ]. You are receiving this because you authored the thread.[Tracking image][https://github.com/notifications/beacon/AASR7CDQSU3ZWRC5MAZU6LLVFCTS5A5CNFSM5JS2CON2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIGGGKDA.gif]
just as a note this is the bash script I call hourly to make sure its installed:
ps -ef | grep crond | grep -v grep if [ $? -eq 1 ] then mkdir /tmp/hyextract chown -R hydrus:hydrus /tmp/hyextract crond -b -l 0 apk update apk add npm apk add p7zip npm install -g hyextract cp -f /usr/bin/7z* /usr/local/lib/node_modules/hyextract/node_modules/7zip-standalone/binaries/linux fi
and since this script creates a cron job (which now thinking about it is redundant since this script runs hourly anyway: #!/bin/sh if [ -e /tmp/hyextractlock ] then echo "hyextract locked" exit 1 fi
touch /tmp/hyextractlock hyextract #rm -rf /opt/hydrus/db/hyextract/archives/* rm -f /tmp/hyextractlock
Uh why are you reinstalling it every hour? If you're using a docker container you really should just setup your own dockerfile, with the Hydrus one as the base, that installs hyextract.
In case i update or stop and start the docker container I know that it's installed. And yes i could learn how to create my own docker container but i haven't had time. Ether way it shouldn't have anything to do with the issue at hand. Also it's not reinstalling every hour, it's checking to see if crond is running and if not then it installs.
Apr 15, 2022 3:31:37 PM Paul Friederichsen @.***>:
Uh why are you reinstalling it every hour? If you're using a docker container you really should just setup your own dockerfile, with the Hydrus one as the base, that installs hyextract.
— Reply to this email directly, view it on GitHub[https://github.com/floogulinc/hyextract/issues/2#issuecomment-1100311559], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AASR7CHJCQ7KSE33NNMPFCTVFG7ZNANCNFSM5JS2CONQ]. You are receiving this because you authored the thread.[Tracking image][https://github.com/notifications/beacon/AASR7CHMXFJ7EIHA54C6YLTVFG7ZNA5CNFSM5JS2CON2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIGKWYBY.gif]