gallery-dl icon indicating copy to clipboard operation
gallery-dl copied to clipboard

Stored refresh token is not used when a config file is specified

Open ermnick opened this issue 7 months ago • 5 comments

When using a custom config file, the stored refresh token isn't used for sites like DeviantArt and Tumblr. Here's the messages I'm getting in the console:

[deviantart][warning] Private or mature deviations detected! Run 'gallery-dl oauth:deviantart' and follow the instructions to be able to access them.
[tumblr][info] Run 'gallery-dl oauth:tumblr' to access dashboard-only blogs
[tumblr][error] AuthorizationError: This Tumblr is only viewable within the Tumblr dashboard

Running the oauth commands doesn't change anything.

I have set up API access for both sites in the config file, so not sure why I'm getting these errors.

ermnick avatar Jan 18 '24 22:01 ermnick

Refresh token is a correct string type in your config file?

Hrxn avatar Jan 18 '24 22:01 Hrxn

I'm not manually entering the refresh token into the config file. I'm not sure where the gallery-dl oauth: commands store the refresh tokens though. Trying to manually enter the same refresh token from the command into the config file gives me this error, even though downloading the same URL without a config file specified gives me no error: [deviantart][error] AuthenticationError: "The refresh_token is invalid." (invalid_request)

Edit: Sorry, forgot to answer your question. Yes, I stored the refresh token as as string like this: "refresh-token":"{refres_token}",

ermnick avatar Jan 18 '24 22:01 ermnick

Okay, I found where the oauth stuff is stored: ~/.cache/gallery-dl/cache.sqlite3 I think gallery-dl should still use the tokens from here even though you're specifying a config file with -c especially when the download fails

ermnick avatar Jan 25 '24 22:01 ermnick

Cached OAuth tokens get used even with a config file, unless you specifically set a refresh-token etc value that is not "auto" or null.

[deviantart][warning] Private or mature deviations detected … is only logged when gallery-dl does not have a refresh token, by the way. DeviantArt in particular is also a bit special in that its refresh token changes each time it is used and gallery-dl therefore needs to store the actual value in your cache file (~/.cache/gallery-dl/cache.sqlite3). When this file gets deleted/moved/etc, your refresh token becomes invalid.

mikf avatar Jan 26 '24 19:01 mikf

I've not moved or deleted the cache.sqlite3 file. However, I have used --clear-cache {extractor} on a different extractor because it was being very buggy and required manual update of the cookies to the db. I had to edit the Python file for that extractor because it didn't support 2fa and didn't properly fetch all metadata keys from the page. It shouldn't have affected the deviantart and tumblr extractors, but it is maybe possible? But I believe the issue has persisted since before I modified the Python code.

I tried adding this to the config to force it to use the cache, but I still get the errors/warnings from above:

"cache": {
        	"file": "~/.cache/gallery-dl/cache.sqlite3"
    		}

Here are the snippets for my deviantart and tumblr dictionary keys in my extractor config. I've created API keys for both sites:

"deviantart":{
         "client-id":"[REDACTED]",
         "client-secret":"[REDACTED]",
         "comments":false,
         "extra":true,
         "flat":true,
         "folders":false,
         "include":"gallery,scraps",
         "journals":"html",
         "mature":true,
         "metadata":false,
         "original":true,
         "wait-min":0,
         "mtime":true,
	 "cache": {
        	"file": "~/.cache/gallery-dl/cache.sqlite3"
    		},
         "postprocessors":[
            {
               "name":"mtime",
               "key":"published_time"
            }
         ]
      }
"tumblr":{
         "avatar":false,
         "external":true,
         "inline":true,
         "posts":"all",
         "reblogs":true,
         "mtime":true,
         "api-key":"[REDACTED]",
         "api-secret":"[REDACTED]",
         "access-token":"[REDACTED]",
         "access-token-secret":"[REDACTED]",
	 "cache": {
        	"file": "~/.cache/gallery-dl/cache.sqlite3"
    		},
         "postprocessors":[
            {
               "directory":"captions",
               "event":"post",
               "name":"metadata",
               "mode":"custom",
               "filename":"tumblr_{blog[name]}_{id}.txt",
               "format":"{date}\n{caption}\n{tags}\n{body}"
            }
         ]
      }

ermnick avatar Jan 29 '24 23:01 ermnick