Data Migration shows 2250 photos downloaded and stops.

[seanodea]

Is there some limit? Is it working on some giant video? What queries can I run to check what it's doing? I do not receive a 429 from google in data/nextcloud.log. The word google doesn't appear there so I can't determine the status of what the app is doing.

[julien-nc]

Did you try to wait to see it at some point the import process continues?

Each photo import job stops after having downloaded 500 MB and schedules the next job. 500 / 2250 = 0.222 If the average photo size is 222 KB, it just seems that the first import job finished and you have to wait for the second one to start.

Are you using the "Cron" background job method? (Admin Basic settings section) If yes, how often is the cron.php script executed?

[Malspherus]

I have the same problem, download hasn't been moving for a few days now. In my case I am well over the 500MB limit (showing 2.7GB in the folder), my cron is also running properly.

[seanodea]

@eneiluj I'm running the cron via cronjob in linux. It runs successfully and every 5 minutes. I just reauthed with google it's sitting at 279 photos. How does it handle large photos, videos or gifs?

[seanodea]

Next day 279 still. I think it's downloading something larger than 500MB and doesn't complete.

[seanodea]

279 still, a gagillion successful cron runs later. At this point, I'm considering it broken.

The only log msgs are: {"reqId":"xxxxxxxxxxxxxx","level":0,"time":"2021-03-28T01:05:29+00:00","remoteAddr":"98.156.188.159","user":"MyUser","app":"extract","method":"GET","url":"/apps/integration_google/import-photos-info","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","version":"21.0.0.18"}

[seanodea]

I raised the 500MB limit to 99 Terrabytes, set my cron pod's spec.startingDeadlineSeconds to ten, canceled and reinitiated import, seems to be working now.

I recommend that you schedule in terms of the number of files and not size, preventing concurrent crons in code. That might end up being slower, but slower and working is better than faster and not working.

Really great software by the way, looking through the code I'm astonished the care put into it.

[julien-nc]

@seanodea Hi, thanks for the interest you put in this issue and this project.

I had a quick look at the job management code and it's true that having concurrent jobs running is the issue here. I published a nightly build with a fix attempt (v1.0.1-1-nightly). Could you try it?

In this nightly, jobs just quit if there is another one currently running for the same user. This way we can keep the 500MB limit.

[ProgramComputer]

I have the same issue but different number of photos initially around 250 when it became stuck. I tried deleting the photos and followed the steps again and now its stuck at 0 photos uploaded. Edit- Followed docs and redid installation and used redis. Migration stopped when I restarted docker container.

[git529]

Can anyone here confirm if this app will import the photos with the metadata included in the jpg files? As you know Google Takeaout strips the metadata and puts into a separate json file.

I am looking for an easy way of backing up my Google Photos including the metadata.

[lkthomas]

Can anyone here confirm if this app will import the photos with the metadata included in the jpg files? As you know Google Takeaout strips the metadata and puts into a separate json file.

I am looking for an easy way of backing up my Google Photos including the metadata.

this is not a forum, if you have another question unrelated to the topic, open another issue to discuss there.

[shalak]

I have a brand new installation, using nextcloud-aio, 25.0.3.

I'm having similar issues as @seanodea had. I was stuck at 3.3G (3439104 according to du . on ext4) with 3138 out of >4370 photos.

I didn't see any errors in google developers console, nor did I get any notifications from them.

The https://console.cloud.google.com/apis/dashboard shows 0 errors and just a sudden drop of traffic around the same time which the Google Integration mentioned in "Last photo import job at...".

After around 20h I cancelled the import and started it again.

It went through 152 photos, all appears to be unique (the current filecount in /Google Photos dir is 3290, and I also did a quick find . -type f -exec stat --format "%20s %n" {} \; | sort | uniq -D -w 20 and saw no duplicates).

Now it's again stuck since this morning. I see no errors in logs, cronjobs are running every few minutes.

How can I diagnose what's going on?

[marcelklehr]

How can I diagnose what's going on?

Could you try to find out the internal state of the app? Try the following occ command: php occ user:setting YourUsernameHere integration_google. Make sure to avoid pasting sensitive information here.

[shalak]

Huh. I re-authorized the app and triggered the process again. Everything went smoothly this time. I guess somewhere between 25.0.3 and 25.0.6 my issue was resolved.

[github-actions[bot]]

Hello :wave: This issue appears to have had no activity for 3 months. We cannot keep track of whether individual issues have resolved themselves or still require attention without user interaction. We're thus adding the stale label to this issue to schedule it for getting closed in 5 days time. If you believe this issue is still valid and should be fixed, you can add a comment or remove the label to avoid it getting closed.

Cheers :blue_heart:

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.