Thumbnails not showing/Photos becoming unresponsive on folder with many files
Open
goddib
opened this issue 4 years ago
•
15 comments
Describe the bug
I have a picture folder with about 50k photos/about 108GB and when I open it in Nextcloud it does not show any thumbnails or react to search/filter. Instead, the system becomes completely unresponsive and I cannot even navigate away while the loading animation shows. Running Nextcloud 20.0.8
To Reproduce
Steps to reproduce the behavior:
Have 50k .jpg files in a folder with about 108GB total file size
Open exactly that folder in "Photos" app (through "My Folders")
Expected behavior
Photos opens with thumbnails of my photos, I am able to filter/search through them by using the search functionality (.i.e. reduce number of shown pictures)
Desktop (please complete the following information):
OS: Windows 10
Browser Vivaldi/Chrome
Browser log
```
cxWeb3.js:18 MEWCX Web3 provider injected
index.js:46 No OC found
t.getLanguage @ index.js:46
value @ gettext.js:45
(anonymous) @ l10n.js:3
(anonymous) @ main.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ main.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ main.js:1
(anonymous) @ main.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ main.js?v=dc0e9b2b-5:1
jquery-migrate.min.js:2 JQMIGRATE: Migrate is installed, version 1.4.1
globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
o @ jquery.js:9789
0 @ jquery.js:26
n @ bootstrap:19
671 @ files_client.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ files_client.js?v=dc0e9b2b-5:1
globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
o @ jquery.js:9792
0 @ jquery.js:26
n @ bootstrap:19
671 @ files_client.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ files_client.js?v=dc0e9b2b-5:1
globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
mounted @ App.vue:117
Ue @ vue.runtime.esm.js:1854
Kt @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
w @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
e._update @ vue.runtime.esm.js:3945
r @ vue.runtime.esm.js:4066
pn.get @ vue.runtime.esm.js:4479
pn @ vue.runtime.esm.js:4468
e @ vue.runtime.esm.js:4073
Ln.$mount @ vue.runtime.esm.js:8415
e._init @ vue.runtime.esm.js:5018
Ln @ vue.runtime.esm.js:5085
(anonymous) @ main.js:37
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:1
globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
mounted @ App.vue:121
Ue @ vue.runtime.esm.js:1854
Kt @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
w @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
e._update @ vue.runtime.esm.js:3945
r @ vue.runtime.esm.js:4066
pn.get @ vue.runtime.esm.js:4479
pn @ vue.runtime.esm.js:4468
e @ vue.runtime.esm.js:4073
Ln.$mount @ vue.runtime.esm.js:8415
e._init @ vue.runtime.esm.js:5018
Ln @ vue.runtime.esm.js:5085
(anonymous) @ main.js:37
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:1
globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
mounted @ App.vue:121
Ue @ vue.runtime.esm.js:1854
Kt @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
w @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
e._update @ vue.runtime.esm.js:3945
r @ vue.runtime.esm.js:4066
pn.get @ vue.runtime.esm.js:4479
pn @ vue.runtime.esm.js:4468
e @ vue.runtime.esm.js:4073
Ln.$mount @ vue.runtime.esm.js:8415
e._init @ vue.runtime.esm.js:5018
Ln @ vue.runtime.esm.js:5085
(anonymous) @ main.js:37
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:1
globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
s @ jquery.js:9789
(anonymous) @ jquery.js:26
n @ bootstrap:19
(anonymous) @ sidebar.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ sidebar.js?v=dc0e9b2b-5:1
globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
s @ jquery.js:9792
(anonymous) @ jquery.js:26
n @ bootstrap:19
(anonymous) @ sidebar.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ sidebar.js?v=dc0e9b2b-5:1
globals.js:60 Handlebars is deprecated: please ship your own, this will be removed in Nextcloud 20
ie @ globals.js:60
get @ globals.js:91
(anonymous) @ templates.js:2
n @ bootstrap:19
(anonymous) @ comments.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ comments.js?v=dc0e9b2b-5:1
commentstabview.js:21 Uncaught TypeError: Cannot read property 'extend' of undefined
at commentstabview.js:21
at Module. (commentstabview.js:16)
at n (bootstrap:19)
at bootstrap:83
at comments.js?v=dc0e9b2b-5:1
(anonymous) @ commentstabview.js:21
(anonymous) @ commentstabview.js:16
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ comments.js?v=dc0e9b2b-5:1
globals.js:60 Handlebars is deprecated: please ship your own, this will be removed in Nextcloud 20
ie @ globals.js:60
get @ globals.js:91
t.default @ no-conflict.js:8
(anonymous) @ handlebars.runtime.js:60
n @ bootstrap:19
(anonymous) @ runtime.js:3
n @ bootstrap:19
(anonymous) @ item.handlebars:1
n @ bootstrap:19
(anonymous) @ files_versions.js?v=dc0e9b2b-5:1
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ files_versions.js?v=dc0e9b2b-5:1
versionstabview.js:18 Uncaught TypeError: Cannot read property 'extend' of undefined
at Module. (versionstabview.js:18)
at n (bootstrap:19)
at bootstrap:83
at files_versions.js?v=dc0e9b2b-5:1
(anonymous) @ versionstabview.js:18
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ files_versions.js?v=dc0e9b2b-5:1
globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
mounted @ Notification.vue:209
Ue @ vue.runtime.esm.js:1854
Kt @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
w @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
e._update @ vue.runtime.esm.js:3948
r @ vue.runtime.esm.js:4066
pn.get @ vue.runtime.esm.js:4479
pn.run @ vue.runtime.esm.js:4554
cn @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
Ve @ vue.runtime.esm.js:1906
Promise.then (async)
Qe @ vue.runtime.esm.js:1933
tt @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
pn.update @ vue.runtime.esm.js:4544
ce.notify @ vue.runtime.esm.js:730
set @ vue.runtime.esm.js:1055
mn.set @ vue.runtime.esm.js:4631
(anonymous) @ App.vue:234
u @ runtime.js:63
(anonymous) @ runtime.js:293
(anonymous) @ runtime.js:118
O @ notifications-main.js?v=dc0e9b2b-5:303
i @ notifications-main.js?v=dc0e9b2b-5:303
Promise.then (async)
O @ notifications-main.js?v=dc0e9b2b-5:303
i @ notifications-main.js?v=dc0e9b2b-5:303
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:303
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:303
_fetch @ App.vue:224
mounted @ App.vue:126
Ue @ vue.runtime.esm.js:1854
Kt @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
w @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
e._update @ vue.runtime.esm.js:3945
r @ vue.runtime.esm.js:4066
pn.get @ vue.runtime.esm.js:4479
pn @ vue.runtime.esm.js:4468
e @ vue.runtime.esm.js:4073
Ln.$mount @ vue.runtime.esm.js:8415
e._init @ vue.runtime.esm.js:5018
Ln @ vue.runtime.esm.js:5085
(anonymous) @ main.js:37
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:1
globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
mounted @ Notification.vue:209
Ue @ vue.runtime.esm.js:1854
Kt @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
w @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
e._update @ vue.runtime.esm.js:3948
r @ vue.runtime.esm.js:4066
pn.get @ vue.runtime.esm.js:4479
pn.run @ vue.runtime.esm.js:4554
cn @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
Ve @ vue.runtime.esm.js:1906
Promise.then (async)
Qe @ vue.runtime.esm.js:1933
tt @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
pn.update @ vue.runtime.esm.js:4544
ce.notify @ vue.runtime.esm.js:730
set @ vue.runtime.esm.js:1055
mn.set @ vue.runtime.esm.js:4631
(anonymous) @ App.vue:234
u @ runtime.js:63
(anonymous) @ runtime.js:293
(anonymous) @ runtime.js:118
O @ notifications-main.js?v=dc0e9b2b-5:303
i @ notifications-main.js?v=dc0e9b2b-5:303
Promise.then (async)
O @ notifications-main.js?v=dc0e9b2b-5:303
i @ notifications-main.js?v=dc0e9b2b-5:303
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:303
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:303
_fetch @ App.vue:224
mounted @ App.vue:126
Ue @ vue.runtime.esm.js:1854
Kt @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
w @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
e._update @ vue.runtime.esm.js:3945
r @ vue.runtime.esm.js:4066
pn.get @ vue.runtime.esm.js:4479
pn @ vue.runtime.esm.js:4468
e @ vue.runtime.esm.js:4073
Ln.$mount @ vue.runtime.esm.js:8415
e._init @ vue.runtime.esm.js:5018
Ln @ vue.runtime.esm.js:5085
(anonymous) @ main.js:37
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ notifications-main.js?v=dc0e9b2b-5:1
capabilities.js:32 OC.getCapabilities is deprecated and will be removed in Nextcloud 21. See @nextcloud/capabilities
getCapabilities @ capabilities.js:32
(anonymous) @ types.js:24
n @ bootstrap:19
(anonymous) @ viewer.js?v=dc0e9b2b-5:23
n @ bootstrap:19
(anonymous) @ viewer.js?v=dc0e9b2b-5:324
n @ bootstrap:19
(anonymous) @ bootstrap:83
(anonymous) @ viewer.js?v=dc0e9b2b-5:1
globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
(anonymous) @ script.js?v=dc0e9b2b-5:492
globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.
ie @ globals.js:60
get @ globals.js:91
(anonymous) @ files.js?v=dc0e9b2b-5:122
Viewer.vue:217 Please do NOT wait for the DOMContentLoaded before registering your viewer handler
handlers @ Viewer.vue:217
fr.run @ vue.runtime.esm.js:4568
cr @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
Xe @ vue.runtime.esm.js:1906
Promise.then (async)
Ye @ vue.runtime.esm.js:1933
tt @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
fr.update @ vue.runtime.esm.js:4544
le.notify @ vue.runtime.esm.js:730
(anonymous) @ vue.runtime.esm.js:882
value @ Viewer.js:78
(anonymous) @ viewer.js:16
Viewer.vue:511 The following mime is already registered text/csv {__ob__: De}
(anonymous) @ Viewer.vue:511
registerHandler @ Viewer.vue:508
handlers @ Viewer.vue:220
fr.run @ vue.runtime.esm.js:4568
cr @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
Xe @ vue.runtime.esm.js:1906
Promise.then (async)
Ye @ vue.runtime.esm.js:1933
tt @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
fr.update @ vue.runtime.esm.js:4544
le.notify @ vue.runtime.esm.js:730
(anonymous) @ vue.runtime.esm.js:882
value @ Viewer.js:78
(anonymous) @ viewer.js:16
session-heartbeat.js:97 session heartbeat polling started
Viewer.vue:217 Please do NOT wait for the DOMContentLoaded before registering your viewer handler
handlers @ Viewer.vue:217
fr.run @ vue.runtime.esm.js:4568
cr @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
Xe @ vue.runtime.esm.js:1906
Promise.then (async)
Ye @ vue.runtime.esm.js:1933
tt @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
fr.update @ vue.runtime.esm.js:4544
le.notify @ vue.runtime.esm.js:730
(anonymous) @ vue.runtime.esm.js:882
value @ Viewer.js:78
(anonymous) @ main.js:26
DevTools failed to load SourceMap: Could not load content for chrome-extension://mmhlniccooihdimnnjhamobppdhaolme/lib/tp/browser-polyfill.min.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME
```
Additional context
This issue has been haunting me ever since I started using nextcloud. will loading large folders like that ever be supported?
Sometimes my nextcloud instance has this bug, especially with "new" pictures (nextcloud server hasn't created thumbnails yet).
But my server is a nanopi-m4 (4GB ram, Dual Cortex-A72 + Quad Cortex-A53, usb3 ssd, ubuntu server), so I can't complain about that behaviour...
thank you for your feedback. First off, my server is a V-Server with 6 cores @2,2 Ghz, 16 GB Ram and 400GB storage. I feel this should be plenty for a single digit user instane of Nextcloud, right?
As of now I am running Nextcloud 20.0.8 on PHP 7.4.15 with all apps up-to-date and a green check-mark in the system overview.
I do not seem to have ffmpeg enabled - that is a PHP module, correct? I understood that is only needed for videos?
I also ran an occ maintenance:repair but without any difference. I can open the "Photos" app without any problem, I get nice previews (although loading slowly) but when I navigate to said folder, Nextcloud becomes unresponsive to a point where I need to close the browser tab (does not react to "reload").
Just as an addition: I am able to open any other folder (with way less pictures) in the app "Photos" without any problem but some delay. Maybe the slow rendering is also what crashes the system in case of the big folder?
I'd like to second this and say that I am also experiencing this. Tried it last on Nextcloud 20.
It runs on a webserver from an external web hosting provider. I don't know the exact specs but Nextcloud system settings says:
In my case, I am trying to access ~10.000 photos and some videos, totalling 70gb (most of the size comes from the videos).
For a reatively small folder (400 photos, 1gb) it works.
Some suggestions on how this could be resolved (from an end-user perspective that is):
Maybe it would be good to have some option to have the generation of thumbnails run on the background, with some sort of CPU throttling enabled or max only one photo at a time (single core loaded), otherwise my host will become very unhappy and probably shut down the process. I would not mind manually running occ the first time to generate the thumbnails.
Viewing the folder with photos should also work if the thumbnails are not yet (all) generated and not be dependent on all the thumbnails being available (or e.g. not for folders containing more than XXX photos).
P.S. Thanks for all the development effort, I realize a community-developed solution like this is limited in its resources, I hope this issue can be picked up :).
I have the same problem with nextcloud 25.0.0.18. Opening the photos app and scrolling to images with low resolution thumbnails makes the server uneesponsive and I get a bad gateway error.
Same here, this app on nc25 is absolutely murdering my CPU. Just opening the app sends it into overdrive. REALLY wish I could easily limit where this thing looks for photos and other media. I think that would help tremendously. I think it's checking mounted volumes, movie archives and everything. The app is called photos, I just want it to display photos albums, i don't need it to scan my entire instance and import 100,000k files per account.
Having 600k images here and the app never worked for me, It just never loads. NC Photos on the other hand is working fine but it has to be on the phone.
Is there some reason that media can not be index into the database like Memories does? I'm not really clear on why either that app has not been integrated better or the features from it have not been carried over to Photos? Is it possibly the sidebar and album creation that slows down Photos? Visually I can not tell what it is doing differently then Memories to slow to a crawl what happens in Memories almost instantly.
Edit: I see that when loading favorites the bottom of the screen shows a spinning loading icon and on the server only a single thread is being used. Is there some reason for only a single thread to be used?
Edit 2: There is also something with Photos that causes me to have multiple of the same favorite photo showing in some cases quadruple, I'm sure contributing to why it's so slow. Memories is not doing this.
Is it possible to fix the preview request to some specific sizes ? (not max size)
I think the problem is that the preview automatic creation depend on the browser windows size. If the web page would ask for limited number of thumbnail, they could be generated only one time for each photos, like peviewgenerator do.
I did a script which get the size of the picture asked by the http request and use preview generator to generate thumbnail.
But the http request from the web app depend of the browser windows size so i have an infinite number of thumbnail.
I have a related issue, where scrolling through photos causes NC to try to generate too many thumbnails at once, leading to an out of memory error that crashes both Apache and MariaDB.
It would make sense to queue thumbnail generation to prevent too many from running at once.
Consider previewgenerator app if you haven't already
Configure the relatively new preview_concurrency_* parameters to match your CPU cores/capacity: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#preview-concurrency-all
Explore other options for customizing various preview related parameters (size, memory usage, etc.)
Explore using Imaginary (or NC AIO which has it built-in in and already tuned): https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#previews
Confirm the behavior reported here still exists with recent versions of NC Server + Photos
nextcloud
