Export stopped working

[ph00lt0]

I noticed that newly created albums in the last weeks have not been exported, unlike the others. Continuous sync is still enabled, and last export time is today, yet quite some pictures are missing.

I tried to resolve it by setting the export folder again, but it now shows, the total number of my pictures under Pending items for quite some hours already, nothing much seems to be happening. How should I go about fixing this? It seems quite mad to export it all from scratch, if that would help at all.

[vishnukvmd]

Hey, could you please share logs (Help > View logs) with [email protected]? We'll take a look and get back to you.

[ph00lt0]

@vishnukvmd don't have time to go through all of this and remove all metadata, but just looking at it, I think this could be the problem:

[2023-10-06T18:10:23+02:00] [info]  export not in progress, starting export
[2023-10-06T18:10:25+02:00] [info]  error: Error export record json parsing failed Error: export record json parsing failed
    at E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241686)
    at async E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241826)
    at async E.preExport (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:232641)
    at async E.scheduleExport (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:244153)
    at async S (ente://app/_next/static/chunks/412-ec80d14e57a8889b.js:1:232574) msg: export Record JSON parsing failed 
[2023-10-06T18:10:26+02:00] [info]  error: Error export record json parsing failed Error: export record json parsing failed
    at E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241686)
    at async E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241826)
    at async E.updateExportRecordHelper (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241081)
    at async i.processQueue (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:259880) msg: export Record JSON parsing failed 
[2023-10-06T18:10:26+02:00] [info]  error: Error export record json parsing failed Error: export record json parsing failed
    at E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241686)
    at async E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241826)
    at async E.updateExportRecordHelper (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241081)
    at async i.processQueue (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:259880) msg: error updating Export Record 
[2023-10-06T18:10:26+02:00] [info]  error: Error update file exported record failed Error: update file exported record failed
    at E.updateExportRecordHelper (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241339)
    at async i.processQueue (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:259880) msg: postExport failed 
[2023-10-06T18:10:26+02:00] [info]  resetting export in progress after completion
[2023-10-06T18:10:26+02:00] [info]  error: Error export record json parsing failed Error: export record json parsing failed
    at E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241686)
    at async E.getExportRecord (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:241826)
    at async E.preExport (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:232641)
    at async E.scheduleExport (ente://app/_next/static/chunks/pages/_app-4d9886a058f36139.js:931:244153)
    at async S (ente://app/_next/static/chunks/412-ec80d14e57a8889b.js:1:232574) msg: scheduleExport failed 
[2023-10-06T18:10:32+02:00] [info]  Syncing location entities localEntitiesCount: 0
[2023-10-06T18:10:32+02:00] [info]  Syncing location entities syncTime: 0

I see similar errors many times in the logs

[abhinavkgrd]

Hey @ph00lt0, Can you confirm, that you are on the latest version v1.6.46?

[ph00lt0]

yes I am on Version 1.6.46 (1.6.46)

[abhinavkgrd]

So, these errors can be caused by intermittent file read issues, we retry these calls if they fail, so these might root cause of the issue. Do you see any export failed for a file logs? Those would be a better idea of what might be happening

[ph00lt0]

Hi @abhinavkgrd I do fin those back but only from yesterday, and they are followed by successful exports:

...
[2023-09-15T10:59:35+02:00] [info]  error: undefined undefined undefined msg: HTTP Service Error 
[2023-09-15T10:59:35+02:00] [info]  error: undefined undefined undefined msg: Failed to download file 
[2023-09-15T10:59:35+02:00] [info]  error: undefined undefined undefined msg: download and save failed 
[2023-09-15T10:59:35+02:00] [info]  error: undefined undefined undefined msg: export failed for a file 
[2023-09-15T10:59:47+02:00] [info]  exporting file REDACTED.jpg with id 26430034 from collection REDACTED successful
...

I looked through the logs, but there are no logs of actual “export failed for a file” from this month at all, only from previous month. So something probably broke there.

[abhinavkgrd]

@ph00lt0 could you please kill the Desktop app and let us know if export_status.json within the Export folder is a valid JSON or not?

You could use an offline JSON parser like https://jsonformatter.org/json-parser to confirm.

Thank you!!

[mnvr]

Possibly stale. Instead of migrating this to the new repo, I'm tentatively closing this. But if it is still an issue, please reopen!

[ph00lt0]

@mnvr this issue was never resolved. I am unable to find this json you talk about

[mnvr]

Apologies then, maybe I got a bit too trigger happy! I'll reopen this and move it to the new repo, and will also try to look at it soon.

[mnvr]

For my own reference, I'll also summarize the status here:

• The best guess we had was that the export_status.json was getting corrupted in some way (multiple clients running or some other threading issue)
• However, that file just doesn't seem to exist, so maybe the problem is something else (maybe we're not able to write the export_status for some reason).

[ph00lt0]

i did locate the file now, just wasn't clear to me where to look. But it was last updated January this year. I surely have taken many pictures since. This confirms the issue I have, the export does not continue to work after the first export. It just stops working after. The json seems valid.

[ph00lt0]

In the app I see this now.

Pressing the "resync" button does not seem to do anything.

[mnvr]

Is there is something in the logs (Help > View logs) when you press "Resync"?

[ph00lt0]

you guessed right

Error: EACCES: permission denied, open '/Users/*****/ente Photos/export_status.json'

The location is granted tho in the security panel of preferences. Wonder what I could change the permissions of the file itself to. Any suggestions?

Besides that would be good if the app can show useful errors / solution to resolve this.

[mnvr]

Ha, nice! At least now we know what's going wrong.

Besides that would be good if the app can show useful errors / solution to resolve this.

Agreed! This really isn't meant to be happening, but at the least the app should've given some generic error instead of silently doing nothing on resync.

The location is granted tho in the security panel of preferences. Wonder what I could change the permissions of the file itself to. Any suggestions?

Ente might not have access to the directory in which you've kept the "ente Photos" folder (I assume that's where you're trying to export to). That is, from the example path that you've shown, it seems that the "ente Photos" folder is in your home directory, so in this case it'll need access to the home directory in System settings > Privacy & Security > Files and Folders.

If you don't want to give ente access to the entire home directory, you can kill the desktop app, move the entire folder to another location (say Documents or Downloads), give access to that area, and then restart your export, selecting the new location.

[ph00lt0]

Ente might not have access to the directory in which you've kept the "ente Photos" folder (I assume that's where you're trying to export to). That is, from the example path that you've shown, it seems that the "ente Photos" folder is in your home directory, so in this case it'll need access to the home directory in System settings > Privacy & Security > Files and Folders.

What you describe here is exactly what I tried to explain in my previous response that I already assesed that ente does have granted access to this location.

Also note, it isn't my home dir as in home home but a subfolder so in theory /Users/***/***/***/

[mnvr]

What you describe here is exactly what I tried to explain in my previous response that I already assesed that ente does have granted access to this location.

It is not easy for me to debug this remotely. I understand that you've tried many of these steps already, but (a) my current belief is that it is some permissions issue on your local machine, and (b) I'm not sure what to suggest more. It is always possible there is a bug though, so I'll continue to think more, if something clicks.

[ph00lt0]

@mnvr can you tell me what read/write permissions should be on the file location for which users? I can try to modify that. But I need to know what it should be like.

Just for reference also. I had this issue multiple times. As I also tried exporting it to other directories before and had the same results. it just stops.

[mnvr]

can you tell me what read/write permissions should be on the file location for which users? I can try to modify that. But I need to know what it should be like.

-rw-r--r-- for the current user should be fine

My feeling is that it is not the file permissions per se, but some of the macOS Security and Privacy restrictions that are getting in the way. One scenario I can imagine is that the Ente app that you've authorized in your system settings is different from the Ente app that you're running (as in, both of them are Ente only, maybe even the same version, but they were in different places so maybe the OS is treating them differently) - to rule this out, you can right click on the entry for ente in the "Files and Folders" privacy setting, and use "Show in Finder).

But yes, let's try with file permissions too! 📿 Maybe making it read/write for all to give it a shot too.

[ph00lt0]

Strangely the file was read only for my own user.

It does work now when pressing thee resync button.

But I really wonder how this permission was flipped. The rest of the folder contents have R&W just fine.

I think the issue is solved. But I'll monitor and reopen if needed.

[mnvr]

Haha, nice! Just happy that this got solved :)

Thanks for debugging this!