Cannot Import Photos or Drive when using reverse proxy

[mike-lloyd03]

Problem

I followed the instructions and have authenticated my Google account with the integration app. The app can see how many documents and photos I have but when I try to import, nothing happens. The Google API dashboard shows requests being made with 200 response codes but no data is transferred.

Current Setup

• Nextcloud 20.0.1
• Raspbian 10 (Buster)
• Apache2 running behind nginx reverse proxy

Apache Configuration

<VirtualHost 127.0.0.1:8080>
    Alias "/" "/var/www/nextcloud/"

    <Directory /var/www/nextcloud/>
        Options +FollowSymlinks
        AllowOverride All

        <IfModule mod_dav.c>
            Dav off
        </IfModule>

        SetEnv HOME /var/www/nextcloud
        SetEnv HTTP_HOME /var/www/nextcloud
    </Directory>

    ErrorLog /var/log/apache2/nextcloud-error_log
    CustomLog /var/log/apache2/nextcloud-access_log common
</VirtualHost>

nginx Configuration

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name <external-site>;

    ssl_certificate /etc/letsencrypt/live/<external-site>/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/<external-site>/privkey.pem;

    client_max_body_size 0;
    underscores_in_headers on;

    location ~ {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header Front-End-Https on;

        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 64;

        proxy_buffering off;
        proxy_redirect off;
        proxy_max_temp_file_size 0;
        proxy_pass http://127.0.0.1:8080;
    }
}

Nextcloud Configuration

<?php
$CONFIG = array (
    'instanceid' => <instanceid>,
    'passwordsalt' => <passwordsalt>,
    'secret' => <secret>,
    'trusted_domains' =>
    array (
        0 => <external_url>,
    ),
    'datadirectory' => '/media/data/nextcloud-data',
    'dbtype' => 'mysql',
    'version' => '20.0.1.1',
    'overwrite.cli.url' => <external_url>,
    'dbname' => 'nextcloud',
    'dbhost' => 'localhost:3306',
    'dbport' => '',
    'dbtableprefix' => 'oc_',
    'mysql.utf8mb4' => true,
    'dbuser' => 'ncuser',
    'dbpassword' => <dbpassword>,
    'installed' => true,
    'memcache.local' => '\OC\Memcache\APCu',
    'overwriteprotocol' => 'https',
);

Nextcloud Logs

[integration_google] Warning: Google API error getting album list, no "albums" key in []

GET /index.php/apps/integration_google/photo-number
from 127.0.0.1 by Mike at 2020-11-16T17:04:34+00:00

[PHP] Error: Error: count(): Parameter must be an array or an object that implements Countable at /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php#56 at <<closure>>

0. <<closure>>
   OC\Log\ErrorHandler::onError(2, "count(): Parame ... e", "/var/www/nextcl ... p", 56, {accessToken: "y ... ]})
1. /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php line 56
   count(null)
2. /var/www/nextcloud/apps/integration_google/lib/Controller/GoogleAPIController.php line 136
   OCA\Google\Service\GoogleContactsAPIService->getContactNumber("ya29.A0AfH6SMAm ... s", "Mike")
3. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 169
   OCA\Google\Controller\GoogleAPIController->getContactNumber()
4. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 100
   OC\AppFramework\Http\Dispatcher->executeController(OCA\Google\Contr ... "}, "getContactNumber")
5. /var/www/nextcloud/lib/private/AppFramework/App.php line 152
   OC\AppFramework\Http\Dispatcher->dispatch(OCA\Google\Contr ... "}, "getContactNumber")
6. /var/www/nextcloud/lib/private/Route/Router.php line 308
   OC\AppFramework\App::main("OCA\\Google\\Co ... r", "getContactNumber", OC\AppFramework\ ... {}, {action: null,_r ... "})
7. /var/www/nextcloud/lib/base.php line 1009
   OC\Route\Router->match("/apps/integration_google/contact-number")
8. /var/www/nextcloud/index.php line 37
   OC::handleRequest()

GET /index.php/apps/integration_google/contact-number
from 127.0.0.1 by Mike at 2020-11-16T17:04:32+00:00

[PHP] Error: Error: Undefined index: connections at /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php#56 at <<closure>>

0. /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php line 56
   OC\Log\ErrorHandler::onError(8, "Undefined index: connections", "/var/www/nextcl ... p", 56, {accessToken: "y ... ]})
1. /var/www/nextcloud/apps/integration_google/lib/Controller/GoogleAPIController.php line 136
   OCA\Google\Service\GoogleContactsAPIService->getContactNumber("ya29.A0AfH6SMAm ... s", "Mike")
2. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 169
   OCA\Google\Controller\GoogleAPIController->getContactNumber()
3. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 100
   OC\AppFramework\Http\Dispatcher->executeController(OCA\Google\Contr ... "}, "getContactNumber")
4. /var/www/nextcloud/lib/private/AppFramework/App.php line 152
   OC\AppFramework\Http\Dispatcher->dispatch(OCA\Google\Contr ... "}, "getContactNumber")
5. /var/www/nextcloud/lib/private/Route/Router.php line 308
   OC\AppFramework\App::main("OCA\\Google\\Co ... r", "getContactNumber", OC\AppFramework\ ... {}, {action: null,_r ... "})
6. /var/www/nextcloud/lib/base.php line 1009
   OC\Route\Router->match("/apps/integration_google/contact-number")
7. /var/www/nextcloud/index.php line 37
   OC::handleRequest()

GET /index.php/apps/integration_google/contact-number
from 127.0.0.1 by Mike at 2020-11-16T17:04:32+00:00

[Bergum]

I have the same pronlem

[julien-nc]

Thanks for the precise bug report. The error about contacts is fixed in v0.0.22. It would be nice if you could try it.

The only thing related to photos in your logs is [integration_google] Warning: Google API error getting album list, no "albums" key in []. This means Google API is returning an empty array when asking for album list.

Could you try the "live API" there https://developers.google.com/photos/library/reference/rest/v1/albums/list?hl=en_US ? Just click "execute" in the right sidebar. I'm interested to see what's the response (appearing right under the "execute" button).

[mike-lloyd03]

Thanks. The response I get is a json object with the following format:

{
  "albums": [
    {
      "id": "<id>",
      "title": "<title>",
      "productUrl": "<productUrl>",
      "mediaItemsCount": "<mediaItemsCount>",
      "coverPhotoBaseUrl": "<coverPhotoBaseUrl>",
      "coverPhotoMediaItemId": "<coverPhotoMediaItemId>"
    }
  ],
  "nextPageToken": "<token>"
}

[julien-nc]

Well apparently the response the integration_google app is receiving is something like [] or {} :grin:. I don't know why...and there's not much I can do without further information. The only difference between the live API and the Nextcloud app is the "OAuth app" that is used. With the Nextcloud app, it's the one that you configured.

It's disturbing that the app can tell you the number of photos and then not get the album list because the same https://photoslibrary.googleapis.com/v1/albums request is done in both cases.

[mike-lloyd03]

What information can I give you to help?

[mike-lloyd03]

Also interesting, it knows which albums are shared because checking/unchecking the 'Ignore Shared Albums' checkbox updates the photo count.

[mike-lloyd03]

Okay I just checked my Google API console. Under 'OAuth Consent Screen', there's a warning saying:

We are currently experiencing service disruptions and are unable to provide quota information at this time.

That could be it.

[julien-nc]

What information can I give you to help?

I don't know :grin:.

It would help if you could add this:

if (preg_match('/v1\/albums$/i', $url)) {
    file_put_contents('/tmp/test', $body);
}

between lines 113 and 114 (just before return json_decode($body, true);) of lib/Service/GoogleAPIService.php.

Then launch the photo import and when the [integration_google] Warning: Google API error getting album list, no "albums" key in [] warning appears in the logs, get the content of /tmp/test file. It is the raw response of the API.

That could be it.

I doubt it as you can see get the photo number...you can get this album list when loading the migration settings page but not when importing. Weird stuff, really :grin:.

[julien-nc]

If you prefer, I can produce a nightly build of the app and you would only have to install it manually. Would you prefer that?

[mike-lloyd03]

Let me give this a try real quick.

[mike-lloyd03]

Now I'm getting this error banner five times when I load the page:

This application requires JavaScript for correct operation. Please enable JavaScript and reload the page.

[julien-nc]

Yeah there is a syntax error in the snippet. I fixed it. Sorry.

[mike-lloyd03]

I should have caught that.

Okay the empty array error is still popping up in the log but nothing is being written to /tmp/test

[mike-lloyd03]

For what it's worth, I get the same problem when trying to import files from Drive including the empty Album array error.

[julien-nc]

Damn, long day, replace $url by $endPoint in the snippet. It should now write something in /tmp/test.

[julien-nc]

Thanks for your patience.

[BenjaminAlpert]

Same problem here

[Bergum]

Photo import at noe working for my part. Files does still not work.

[mike-lloyd03]

Okay still nothing is being written to /tmp/test. This is what I have in 'GoogleAPIService.php`, lines 111-118:

if ($respCode >= 400) {
    return ['error' => $this->l10n->t('Bad credentials')];
} else {
    if (preg_match('/v1\/albums$/i', $endPoint)) {
        file_put_contents('/tmp/test', $body);
    }
    return json_decode($body, true);
}

Am I missing something?

[julien-nc]

@mike-lloyd03 Is the file created? If it's created but stays empty it means the response you get from Google API is an empty string.

Let's try something else: Could you authorize this URL : https://localhost/dev/server/index.php/apps/integration_google/oauth-redirect in developer console -> OAuth 2.0 Client IDs -> your web app -> Authorized redirect URIs ?

I would need the app's Client ID and secret (you can send them by email).

It will let me try to authenticate to my google account through your OAuth app. I will hopefully be able to reproduce the bug on my side and then...we'll see.

[mike-lloyd03]

@eneiluj

Sorry it has taken me so long to get back to you. No file is being created at /tmp/test.

I just sent you an email with the details you needed and I added the URL to my Google web app.

Thanks for your help with this.

[julien-nc]

@mike-lloyd03 Don't be sorry, you're helping a lot!

It's working for me with your OAuth app so let's find out what happens exactly. I wonder if it can be because you're using a reverse proxy like @Bergum. @BenjaminAlpert Are you using a reverse proxy too?

One thing is for sure, things are working fine when your Nextcloud does the requests as an authenticated user because you get the photo count in the settings page. The background job used to import photos/drive is launched in a different context. This might be the source of the problem.

I suspect the authentication is not correctly provided to the requests in the background job. Let's find out.

So I made a nightly release with a lot of level 1 logs. Could you install it (just replace the .../nextcloud/app/integration_google directory) and change .../nextcloud/config/config.php to set log level to 1:

'loglevel' => 1,

There will be much more information in your logs. I would need to see what happens there when you import your google photos. We're gonna nail this bug...hopefully.

[BenjaminAlpert]

Yes, I am using Nginx Reverse Proxy. I am also using cron for background jobs.

[Bergum]

You have a new log file.

[julien-nc]

@BenjaminAlpert ok good to know, thanks.

@Bergum Thanks. I guess you were using latest release and not the nightly build. Anyway, fixed 2 more things thanks to your logs:

• missing album name in API response (not missing for me with untitled albums, but seems to happen for some of you... :grin:)
• it's sometimes impossible to create the downloaded file in Nextcloud. I don't know why. Those files are skipped now.

v0.0.24 is out.

Something interesting in @Bergum log file:

{"Exception":"Error","Message":"fread(): read of 8192 bytes failed with errno=9 Bad file descriptor at /var/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php#91...

So either:

• your Nextcloud server is having trouble downloading the file (I don't see why, could reverse proxy be messing with the request?)
• or google does not answer to some requests

If someone has time to setup a test instance without reverse proxy on the same machine and try importing photos and drive, it would be interesting to know if the issue we're facing is related to reverse proxies.

Thanks again to all of you for you perseverance.

[Bergum]

I was using the latest nightly. Or, I think i was. I dowloaded i and extrackted

but, im a noob on ubuntu.

Edit: Nope. I failed. Extrackted to the wrong place. New log is coming... .

[Bergum]

New logfile uploaded.

[julien-nc]

@Bergum There is nothing on 2020-11-18 related to drive or photo import in your logfile. What did you do before getting it? I need a full run of those import to see something in the log file. Thanks anyway :grin:

[mike-lloyd03]

I should be able to just forward traffic to my Apache server directly and forego nginx. I'll try tonight.

[Bergum]

I installed the nightly, set log level, stopped all of the imports that was running, started a new import of all possible imports, and watched until new lines stopped coming into the log app. Then i sendt you the log.