mail icon indicating copy to clipboard operation
mail copied to clipboard

Opening messages is slow

Open ChristophWurst opened this issue 1 year ago • 35 comments

Steps to reproduce

  1. Open the app
  2. Click on a message
  3. Click on another message

Expected behavior

Fast switching

Actual behavior

Mediocre to slow switching. HTML messages are worse than plain text messages.

Mail app version

4.0

Nextcloud version

No response

Mailserver or service

No response

Operating system

No response

PHP engine version

None

Nextcloud memory caching

No response

Web server

None

Database

None

Additional info

No response

ChristophWurst avatar Nov 21 '24 14:11 ChristophWurst

This is a huge issue for me. Mails need multiple seconds, sometimes up to 30 s to display. On every other client it's instantaneous.

9-FS avatar Jan 14 '25 20:01 9-FS

Thank you for the feedback, @9-FS. Could you please open the browser developer console, switch to the network tab, reload the page and try to find the specific requests which are slow, if any? That should help narrowing it down.

ChristophWurst avatar Jan 15 '25 07:01 ChristophWurst

FYI - had the same problem both with Nextcloud Mail AND Snappymail. The cause was a bad DNS configuration:

  • Nextcloud running in Docker
  • local DNS server (Technitium/Unbound/pihole) running in Docker on the same machine
  • Docker internal network: 172.16.0.0
  • LAN-IP: 192.168.1.10 --> the local router sets this (server IP) as DNS for all clients in LAN
  • Server host-OS and therefore Docker daemon gets it's own IP assigned as DNS server (via DHCP)
  • DNS lookup of mailserver-URL from Nextcloud via 192.168 IP fails (something about docker networking)

Solution: in the docker-compose file of the DNS server, make sure port 53 is bound to the correct host-IP:

ports:
      - 192.168.1.10:53:53/udp
      - 192.168.1.10:53:53/tcp

Could you please check if you run a similar setup? Easy test: set s/t like 1.1.1.1 as host-OS DNS and restart Docker daemon.

solderdev avatar Jan 15 '25 08:01 solderdev

@ChristophWurst Unfortunately, it's incosistent. The requests that take multiple seconds all have in common that they are all JSON data from Google Mail and are not displayed in a pure text fashion. In contrast, the mails I have imported from my university server seem to be text only and load instantaneously.

@solderdev This is a pretty similar setup to mine, I just have my Technitium DNS server in its own network br0 under 192.168.0.3 and my router advertises that IP to all devices in the network as DNS server. From the console of other containers like Nextcloud, that are in the default bridge network, I've tried some nslookup and ping 192.168.0.3 and they were all instantaneous. I post my docker composes for Technitium and Nextcloud down here just in case it might help.

services:
    technitium:
        container_name: "technitium"
        image: "technitium/dns-server:13.3.0"
        environment:
            HOST_OS: "Unraid"
            TZ: "UTC"
            TCP_PORT_5380: 5380
            UDP_PORT_53: 53
        volumes:
            - "/mnt/user/appdata/technitium/:/etc/dns/:rw"
        user: "99:100"
        networks:
            br0:
                ipv4_address: "192.168.0.3"
                ipv6_address: "fd00::9e6b:ff:fe2e:d542"
        sysctls:
            net.ipv4.ip_local_port_range: "1024 65000"
        restart: "always"
        deploy:
            resources:
                limits:
                    memory: "1G"

networks:
    br0:
        external: true
services:
    mariadb:
        container_name: "nextcloud_mariadb"
        image: "mariadb:11.4.4"
        environment:
            HOST_OS: "Unraid"
            TZ: "UTC"
            # MARIADB_DATABASE: ""
            # MARIADB_PASSWORD: ""
            MARIADB_RANDOM_ROOT_PASSWORD: yes
            MARIADB_USER: "felix"
        env_file: ".env"
        volumes:
            - "/mnt/user/appdata/nextcloud/mariadb/config/:/etc/mysql/conf.d:rw"
            - "/mnt/user/appdata/nextcloud/mariadb/data/:/var/lib/mysql/:rw"
        user: "99:100"
        network_mode: "bridge"
        ports:
            - "3306:3306"
        deploy:
            resources:
                limits:
                    memory: "1G"

    redis:
        container_name: "nextcloud_redis"
        image: "redis:7.4.1"
        environment:
            HOST_OS: "Unraid"
            TZ: "UTC"
        network_mode: "bridge"
        ports:
            - "6379:6379"
        deploy:
            resources:
                limits:
                    memory: "1G"

    nextcloud:
        container_name: "nextcloud"
        image: "nextcloud:30.0.4"
        depends_on:
            - "mariadb"
            - "redis"
        environment:
            HOST_OS: "Unraid"
            TZ: "UTC"
        volumes:
            - "/mnt/user/appdata/dlh_duty_plan_converter/calendar/:/var/www/html/data/files_external/dlh_duty_plan_converter/calendar/:rw"
            - "/mnt/user/appdata/nextcloud/nextcloud/:/var/www/html:rw"
            - "/mnt/user/nextcloud/backup/:/var/www/html/data/files_external/backup/:rw"
            - "/mnt/user/nextcloud/Felix/:/var/www/html/data/Felix/:rw"
            - "/mnt/user/nextcloud/media/:/var/www/html/data/files_external/media/:rw"
        user: "99:100"
        network_mode: "bridge"
        ports:
            - "8666:80"
        deploy:
            resources:
                limits:
                    memory: "1G"

    nextcloud_cronjob_runner:
        container_name: "nextcloud_cronjob_runner"
        image: "rcdailey/nextcloud-cronjob:latest"
        depends_on:
            - "nextcloud"
        environment:
            HOST_OS: "Unraid"
            TZ: "UTC"
            NEXTCLOUD_CONTAINER_NAME: "nextcloud"
            NEXTCLOUD_CRON_MINUTE_INTERVAL: 5
            NEXTCLOUD_EXEC_SHELL: "bash"
            NEXTCLOUD_EXEC_USER: 99
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock:ro"
        network_mode: "bridge"
        deploy:
            resources:
                limits:
                    memory: "1G"

I'm currently further looking into perhaps tweaking some Google Mail or IMAP settings...

9-FS avatar Jan 16 '25 11:01 9-FS

In my case, snappymail load mails faster like with in 3 seconds, but in Nextcloud Mail it takes more than 10 seconds

elhananjair avatar Mar 05 '25 17:03 elhananjair

i tried to optimize services on dedicated server but no matter how you tune it ... nextlcoud mail is unusable because rendering simple email content in 4-8 seconds is joke with 0.14 load avarage, very low CPU utilization. I open simple email than open another and go back to that first and it is still rendering as the first time ... the worst thing is that without webmail the application loses its business sense, because why have a cloud with the ability to manage a team, when the most important function, i.e. webmail, is weak? scaling server is not a solution. 2 seconds is much for dedicated solutions. This is definitelly not acceptable for business standards

I dont even mention poor search.

Mail 4.3.1, Apache, Redis cache, mariadb, Nextcloud 30.0.8

cenobitz avatar Mar 20 '25 21:03 cenobitz

Thank you for your feedback

ChristophWurst avatar Mar 27 '25 13:03 ChristophWurst

I am also having this issue on a private nextcloud installation over imap ssl/tls with self signed certs. cert size is 8kb.

listing emails is relatively quick but the body of any particular email takes quite awhile to load. somewhere around 8-15s probably on average.

ive been running my own private email server since 2005 and i have never seen email load this slow.

generally all my email clients regardless if its desktop or a mobile client load email pretty much instantaneously (<1s). i also have other close family and friends that i host email for and they report its instant.

but nextcloud mail is doing something out of band and not sure exactly what. imap in general is a decently fast protocol outside of search. container utilization is quite low as well. around 1-2% cpu

when inspecting the network calls it looks like nextcloud mail is retrieving different parts of the email separately but its not doing it asynchronously. each call/part is completed after the next waiting for the previous call to complete.

i think simply doing things in async would be a quick and good optimization. nonetheless there is something with the nextcloud imap implementation that appears to be very slow as you can see each call taking a few to several seconds to complete.

next cloud 31.0.4, mail 5.0.3, nginx, postgres.

dovecot on the email server.

edit: official signed certs make no difference, cert is 2kb

k3mist avatar May 02 '25 20:05 k3mist

Thank you for sharing your analysis results

when inspecting the network calls it looks like nextcloud mail is retrieving different parts of the email separately but its not doing it asynchronously. each call/part is completed after the next waiting for the previous call to complete.

Would you be able to share redacted logs?

Do you have any suggestions for how to make asynchronous calls with PHP?

ChristophWurst avatar May 05 '25 05:05 ChristophWurst

@ChristophWurst

Would you be able to share redacted logs?

ill look into that. when i did check them there was nothing in there that was of value. you just see the requests happening in order and nothing in particular stood out. just takes a long time to retrieve a single email. pretty normal logs. but i will see if i can pull them

Do you have any suggestions for how to make asynchronous calls with PHP?

its been a very long time since ive done php or ajax (15+ years since ive touched php and about 10+ years since ive touched javascript).

nonetheless i do know it shouldnt be done with php directly but instead php would simply facilitate the ability to do so.

meaning; there are several parts to the email that are retrieved separately (headers, metadata, body, attachments). so the php should provide code paths to retrieve the individual parts respectively.

i do not know the URI schemes that are used on nextcloud internally but this might look something like;

/email-message.php?messageId=X&part=headers
/email-message.php?messageId=X&part=metadata
/email-message.php?messageId=X&part=body
/email-message.php?messageId=X&part=attachments

then using async <> promises with javascript execute all those calls at once minus attachments and minus what is already displayed in the UX(most likely headers, but feasibility?).

and then simply populate the UX as they are completed. its how i would approach it anyway. generally speaking i think this is how most clients operate.

as an aside; i would definitely make the attachments call a secondary click by default that the user must perform after the email message is displayed. many clients provide an option to "automatically download attachments". personally i keep that setting off in any client i use as it just slows down email browsing, but it is subjective/preference.

k3mist avatar May 14 '25 16:05 k3mist

Thanks

ChristophWurst avatar May 14 '25 16:05 ChristophWurst

just another thing i thought of. if you want to take full advantage of browser/edge caching i think the URI scheme would need to be in this format and not use querystring. but i could be wrong, been awhile

/email/message/<messageId>/<part>

so basically

/email/message/id/headers
/email/message/id/metadata
etc..

k3mist avatar May 14 '25 16:05 k3mist

hey @ChristophWurst i wanted to report back that the "slowdown" or "slowness" may be related to nextcloud on the same machine as local dns. for instance if someone is running nextcloud in docker on a docker "nextcloud_network" and also using docker for dnsmasq or bind9 on a separate docker network such as "dns_network". meaning neither is on the default docker network but running on the same machine.

i recently made changes to my local network including moving my local dns and interestingly nextcloud mail is relatively speaking pretty fast.

this is after moving my dns off docker and onto raspberry's. so its not on the same machine anymore. maybe ~2s or so for an email to load which is basically very usable now!

just wanted to share that it may not entirely be a nextcloud issue. but maybe nextcloud mail + docker with local dns + docker on two separate docker networks on the same machine is creating some weird issue.

a temporary solution for people may be moving local dns onto a different machine if that is they have the same setup as i did.

anyway, just wanted to share this information as i found it very interesting. have a great day!

k3mist avatar Jul 07 '25 18:07 k3mist

Thanks for the input!

ChristophWurst avatar Jul 08 '25 07:07 ChristophWurst

hey @ChristophWurst i wanted to report back that the "slowdown" or "slowness" may be related to nextcloud on the same machine as local dns. ...

Hi @k3mist ! Please see my answer above: https://github.com/nextcloud/mail/issues/10384#issuecomment-2591970772 I fixed the DNS issues this way.

solderdev avatar Jul 08 '25 14:07 solderdev

I have a user reporting slow opening of email and with the PHP-FPM slow log set to log things that take over 10 seconds there are several of these instances logged each day:

[21-Jul-2025 10:14:39]  [pool cloud] pid 3041385
script_filename = /home/cloud/sites/nextcloud/index.php
[0x00007ffa26c146c0] curl_exec() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php:44
[0x00007ffa26c14620] __invoke() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php:142
[0x00007ffa26c14560] {closure:{closure:GuzzleHttp\Middleware::tap():137}:138}() /home/cloud/sites/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php:149
[0x00007ffa26c14400] {closure:{closure:OC\Http\Client\DnsPinMiddleware::addDnsPinning():103}:104}() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php:35
[0x00007ffa26c14320] __invoke() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php:31
[0x00007ffa26c14270] {closure:{closure:GuzzleHttp\Middleware::cookies():28}:29}() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php:71
[0x00007ffa26c141d0] __invoke() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php:66
[0x00007ffa26c14130] {closure:{closure:GuzzleHttp\Middleware::httpErrors():60}:61}() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php:75
[0x00007ffa26c140a0] __invoke() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php:333
[0x00007ffa26c13f50] transfer() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php:169
[0x00007ffa26c13e70] requestAsync() /home/cloud/sites/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php:189
[0x00007ffa26c13de0] request() /home/cloud/sites/nextcloud/lib/private/Http/Client/Client.php:206
[0x00007ffa26c13d30] get() /home/cloud/sites/nextcloud/apps/mail/lib/Service/Avatar/FaviconDataAccess.php:26
[0x00007ffa26c13c90] retrieveUrl() /home/cloud/sites/nextcloud/apps/mail/lib/Vendor/Favicon/Favicon.php:268
[0x00007ffa26c13ba0] getInPage() /home/cloud/sites/nextcloud/apps/mail/lib/Vendor/Favicon/Favicon.php:202
[0x00007ffa26c13ad0] getFavicon() /home/cloud/sites/nextcloud/apps/mail/lib/Vendor/Favicon/Favicon.php:160
[0x00007ffa26c13a20] get() /home/cloud/sites/nextcloud/apps/mail/lib/Service/Avatar/FaviconSource.php:63
[0x00007ffa26c13910] fetch() /home/cloud/sites/nextcloud/apps/mail/lib/Service/Avatar/CompositeAvatarSource.php:43
[0x00007ffa26c13850] fetch() /home/cloud/sites/nextcloud/apps/mail/lib/Service/AvatarService.php:104
[0x00007ffa26c137b0] getAvatar() /home/cloud/sites/nextcloud/apps/mail/lib/Controller/AvatarsController.php:54

Is this DNS related? I'm wondering about installing a local copy of Unbound on the server to cache DNS requests as I'd expect DNS look ups for avatar and favicon URLs would be quite cachable?

This server is not using Docker, it is using systemd-resolved for DNS lookups to two Unbound servers on the same subnet.

The PHP curl module is enabled.

chriscroome avatar Jul 24 '25 08:07 chriscroome

Favicon URLs are cached, so only the first request for a user's avatar would be slow if DNS is the bottleneck

ChristophWurst avatar Jul 25 '25 09:07 ChristophWurst

I'm suffering this too. It's just way slower than any other client I use, be that web based or Thunderbird or whatever. A very tedious long long wait any time I click on a mail item for it to render in the preview window. I can watch on the network tab in dev tools but not find a sensible easy way to copy a performance log short of a screen shot (which embarrasses me some):

Image

As far as I know I'm the latest of everything:

  • Nextcloud Hub 10 (31.0.9)
  • Mail Version 5.5.1, AGPL-licensed

I run Nextcloud on bare metal Ubuntu, not containerised, and behind a NAT firewall of course, but the performance of Nextcloud is define, and all mail clients behind same NAT work fine (I use Thunderbird primarily). This mail app stands out as, exceptional in every regard (very awesome IMHO) bar this one performance issue that makes it essentially unusable.

bernd-wechner avatar Sep 23 '25 01:09 bernd-wechner

There seems to be a consistent delay for various requests.

Would you be able to install and configure xdebug to capture some performance profiles?

ChristophWurst avatar Sep 23 '25 06:09 ChristophWurst

With https://github.com/nextcloud/server/pull/55265 we will see info/warning/error log entries when the DNS pinning middleware is the bottleneck.

ChristophWurst avatar Sep 23 '25 07:09 ChristophWurst

I can confirm this issue is present in Nextcloud 32. SMTP/IMAP is configured to use Yahoo and working. However when I click into a message, it takes 10 seconds for the message to display. In watching the server load, nothing is chewing CPU. What I seeing is:

  • Click email message
  • 5 second mark (the toolbar icons inside the message appear)
  • 10 second mark (the message renders)

So to me it would be interesting to know what the code is doing to prior to and after the toolbar icons are displayed. This is right in the middle of the process in terms of time.

I want to also repeat what others have said. This app is super cool and worth the time for Nextcloud to have an all-in-one solution. But this delay in message viewing makes it too slow to use for regular users.

daveatpinellas avatar Oct 06 '25 14:10 daveatpinellas

So to me it would be interesting to know what the code is doing to prior to and after the toolbar icons are displayed.

Would you be able to install and configure xdebug to capture some performance profiles?

ChristophWurst avatar Oct 06 '25 14:10 ChristophWurst

I don't want to blow up my regular Nextcloud server :), but do have a Nextcloud sandbox on Proxmox and just kicked off a clone to play with. If you get me general tips on how to focus xdebug to just the Mail app, I will do it. It looks straight forward to install and activate the php module.

daveatpinellas avatar Oct 06 '25 20:10 daveatpinellas

Cool! Set up the instance with the same account to see if you even experience the same slowness.

ChristophWurst avatar Oct 07 '25 05:10 ChristophWurst

@ChristophWurst Ok, I have installed php-xdebug on my sandbox and logged into Yahoo via IMAP/SMTP to the same account and it's fully replicated. Click message > 5 seconds later the icons appear in the message pane > 5 second later the message renders.

Let me know how I can help or in your view the best way to get good information to assist.

daveatpinellas avatar Oct 07 '25 13:10 daveatpinellas

Perfect! See https://xdebug.org/docs/profiler for the xdebug configuration. For your case it makes most sense if you just enable profiling for all requests. There might be a bunch of files to search through but it should be manageable.

Open the browser console and switch to the network tab to see the XHRs. There should be at least one slow request. Try to find its profile. You can use KCachegrind to view the files. It will show where most time is spent.

ChristophWurst avatar Oct 07 '25 14:10 ChristophWurst

@ChristophWurst @kesselb similar behavior here. It takes 4 seconds on my system to load an html mail. No other client takes that much time with my mailserver.

As a reference, this is the loading time with Titan email, a web based mail client. It takes a bit less than 500 ms to load a mail.

Image

The same email takes around 3900 ms with mail on my Nextcloud instance.

Image

Mail app version: 5.5.11 Nextcloud version: 32.0.0

I guess a good workaround would be to offer users an option to not only fetch headers + short preview initially (for speed), but to also provide a checkbox in mail settings to fetch the full message body before you click a message. WDYT?

badger12345 avatar Oct 28 '25 19:10 badger12345

Hi @ChristophWurst , thanks for your help with this! Did you ever get ahold of XHProf traces?

I can get you one of these if not.... would any others besides the slow XHR request be useful?

damienvancouver avatar Oct 31 '25 15:10 damienvancouver

Same issue.. not as bad as others but 2 to 4s to pull down an HTML email to the preview pane after clicking on it until its cached in the browser. Text only still takes 1 to 2s. I agree with badger12345, an option to fetch the preview along with the headers is probably the solution here.

kejar31 avatar Nov 03 '25 12:11 kejar31

Same issue on my side. Depending on the user and how many emails they have the search can be very slow. Some cases exceeding 10 000 MS.

Mail app version: 5.5.15 Nextcloud version: 31.0.7

Image

LostDovahkin avatar Nov 18 '25 10:11 LostDovahkin