previewgenerator icon indicating copy to clipboard operation
previewgenerator copied to clipboard
verified publisher icon nextcloud

./occ preview:generate-all" - Size issue - no precalcuation done, no information to the Admin

Open Githopp192 opened this issue 6 years ago • 1 comments

Steps to reproduce

Run --> ./occ preview:generate-all"

Expected behaviour

Important steps, which Need to be improved:

  • There needs to be a pre-calculation on existing storage - where a rough pre-space estimation will be calculated, upon storage usage (depending on existing Images).

Actual behaviour

No Storage Pre-Calculation will be done. Bigger Nextcloud instances are at the mercy of a huge increase of storage space and Database space.

I read some user comments related to previewgenerator, where some Nextcloud instances increased to approx. the double of an existing user space allocation. After i have read more then one issue about this topic and after exploring a high increase of my storage consumption of my users in the preview directory in a short of time, and an huge increase of the Database, i decided to stop the "occ preview:generate-all" Job.

My feeling is, that this app could have the power of breaking a Nextcloud instance due to short on Storage. (of course i want to say a big thank to the Developper, having great ideas to improve the functionality of Nextcloud).

About the - maybe - far-reaching consequences - such an App should do a Storage estimated pre-calculation upon existing Storage usage. Of course i read some / the recommendations, how the Preview-Image sizes could be reduced in size, but also this reduction first needs to be reflected into a pre-Calculation.

  1. I have disabled the App and want to get rid of all the Preview, generated from the App.

I saw a comment about here in the discussions:

To "reset" the preview cache

  1. Stop Webserver (apache / nginx)
  2. Moved the appdata_instaceName/preview folder outside the appdata folder. You can just remove it if you are confident.
  3. Run sudo -u www-data /usr/bin/php /var/www/nextcloud/occ files:scan-app-data to fix database
  4. Start Webserver and Test everything works
  5. Finish up (delete moved folder, etc).

Is this the right approach ?

Maybe i missed some details .. then i want to apologize about :-)

Server configuration detail

Operating system: Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64

Webserver: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.1.26 (apache2handler)

Database: mysql 5.5.60

PHP version:

7.1.26 Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, apache2handler, apcu, bcmath, bz2, calendar, ctype, curl, dba, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, imagick, imap, intl, json, ldap, exif, mcrypt, mysqli, PDO, pdo_mysql, pdo_sqlite, Phar, posix, redis, shmop, SimpleXML, soap, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, xml, wddx, xmlreader, xmlwriter, xsl, memcached, zip, Zend OPcache

Nextcloud version: 14.0.6 - 14.0.6.0

Updated from an older Nextcloud/ownCloud or fresh install: Upgrade from 14.0.4

Where did you install Nextcloud from: Origin Updater

Signing status
List of activated apps 
Enabled:
 - accessibility: 1.0.1
 - activity: 2.7.0
 - announcementcenter: 3.3.1
 - apporder: 0.6.0
 - audioplayer: 2.6.0
 - bookmarks: 0.16.2
 - bruteforcesettings: 1.3.0
 - calendar: 1.6.4
 - circles: 0.15.2
 - cloud_federation_api: 0.0.1
 - comments: 1.4.0
 - contacts: 2.1.8
 - data_request: 1.1.0
 - dav: 1.6.1
 - deck: 0.5.2
 - drawio: 0.9.2
 - drop_account: 0.1.0
 - federatedfilesharing: 1.4.0
 - federation: 1.4.0
 - files: 1.9.0
 - files_antivirus: 1.4.2
 - files_fulltextsearch: 1.1.3
 - files_mindmap: 0.0.11
 - files_pdfviewer: 1.3.2
 - files_rightclick: 0.11.0
 - files_sharing: 1.6.2
 - files_texteditor: 2.6.0
 - files_trashbin: 1.4.1
 - files_versions: 1.7.1
 - files_videoplayer: 1.3.0
 - firstrunwizard: 2.3.0
 - flowupload: 0.0.9
 - fulltextsearch: 1.1.1
 - gallery: 18.1.0
 - issuetemplate: 0.4.0
 - logreader: 2.0.0
 - lookup_server_connector: 1.2.0
 - mood: 0.4.1
 - nextcloud_announcements: 1.3.0
 - notes: 2.5.1
 - notifications: 2.2.1
 - oauth2: 1.2.1
 - onlyoffice: 2.1.2
 - ownbackup: 18.11.0
 - passman: 2.2.1
 - password_policy: 1.4.0
 - polls: 0.9.5
 - previewgenerator: 2.0.0
 - provisioning_api: 1.4.0
 - quicknotes: 0.1.5
 - quota_warning: 1.3.0
 - radio: 0.6.4
 - rainloop: 6.0.2
 - ransomware_protection: 1.2.0
 - serverinfo: 1.4.0
 - sharebymail: 1.4.0
 - sharerenamer: 2.3.0
 - socialsharing_diaspora: 1.0.4
 - socialsharing_email: 1.0.5
 - socialsharing_facebook: 1.0.4
 - socialsharing_googleplus: 1.0.4
 - socialsharing_twitter: 1.0.4
 - spreed: 4.0.3
 - support: 1.0.0
 - survey_client: 1.2.0
 - systemtags: 1.4.0
 - tasks: 0.9.8
 - theming: 1.5.0
 - twofactor_backupcodes: 1.3.1
 - twofactor_totp: 1.5.0
 - updatenotification: 1.4.1
 - workflowengine: 1.4.0
Disabled:
 - admin_audit
 - carnet
 - cms_pico
 - encryption
 - external
 - files_accesscontrol
 - files_external
 - files_markdown
 - impersonate
 - jsloader
 - mindmaps
 - phonetrack
 - richdocuments
 - user_external
 - user_ldap
 - weather
Configuration (config/config.php) 
{
    "memcache.local": "\\OC\\Memcache\\APCu",
    "filelocking.enabled": true,
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "dbindex": 0,
        "timeout": 1.5
    },
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        xxxx
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/xxxxxx",
    "htaccess.RewriteBase": "\/",
    "overwriteprotocol": "https",
    "dbtype": "mysql",
    "version": "14.0.6.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "maintenance": false,
    "theme": "",
    "loglevel": 1,
    "updater.release.channel": "production",
    "auth.bruteforce.protection.enabled": true,
    "check_for_working_htaccess": true,
    "data-fingerprint": "0xxxxx65",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpsecure": "xxx",
    "mail_smtpauth": 1,
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "xxx",
    "session_lifetime": 1200,
    "session_keepalive": false,
    "logtimezone": "Europe\/xxx",
    "logfile": "\/media\/log\/nextcloud.log",
    "knowledgebaseenabled": false,
    "log_rotate_size": 3145728,
    "onlyoffice": {
        "verify_peer_off": true
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: false

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134

Operating system: Windows 10

Logs

Browser log
Nextcloud log
Browser log

Githopp192 avatar Feb 08 '19 00:02 Githopp192

Nice input. Would prefer to have that information before having the initial preview generation process running for maaaaaaany hours (see https://github.com/rullzer/previewgenerator/issues/129#issuecomment-555098864).

bcutter avatar Nov 18 '19 16:11 bcutter