server icon indicating copy to clipboard operation
server copied to clipboard

[Bug]: Tried to log in "user" but could not verify token

Open smart7324 opened this issue 2 years ago • 18 comments

⚠️ This issue respects the following points: ⚠️

  • [X] This is a bug, not a question or a configuration/webserver/proxy issue.
  • [X] This issue is not already reported on Github (I've searched it).
  • [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
  • [X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
  • [X] I agree to follow Nextcloud's Code of Conduct.

Bug description

As soon as I open Nextcloud in a new tab, I get redirected to login page and have to login again. Then always the first login fails/nothing happens, so I have to login twice. I am seeing lots of "Tried to log in "user" but could not verify token" errors in log.

It is only happening on Safari (macOS, iOS, iPadOS), tried several versions, also did a clean install of Nextcloud 26 and still the same. Also tried with another user account on a different Mac.

At first I thought it could be related to #33919, but it doesn't seem to be the case. I really spent many hours in trying to get this fixed, but I have no clue, why it is not working.

Steps to reproduce

  1. Login to Nextcloud in Safari
  2. Open another tab and open Nextcloud (alternatively close browser and open it again)
  3. You will be redirected to login page and the message "Tried to log in "user" but could not verify token" is in log file.

Expected behavior

The user should still be logged in and not be redirected to login page.

Installation method

Community Manual installation with Archive

Nextcloud Server version

26

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • [X] Default user-backend (database)
  • [ ] LDAP/ Active Directory
  • [ ] SSO - SAML
  • [ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "26.0.0.11",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "csrf.disabled": true,
        "integrity.check.disabled": true,
        "logfile": "\/var\/www\/cloud\/data\/nextcloud.log",
        "loglevel": 4,
        "enable_previews": true,
        "remember_login_cookie_lifetime": 31536000,
        "session_lifetime": 31536000,
        "session_relaxed_expiry": true,
        "session_keepalive": true,
        "simpleSignUpLink.shown": false,
        "htaccess.IgnoreFrontController": true,
        "default_phone_region": "DE",
        "default_language": "de",
        "force_language": "de",
        "theme": "***REMOVED SENSITIVE VALUE***",
        "defaultapp": "files",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "updater.release.channel": "stable"
    }
}

List of activated Apps

Enabled:
  - cloud_federation_api: 1.9.0
  - comments: 1.16.0
  - dav: 1.25.0
  - federatedfilesharing: 1.16.0
  - files: 1.21.1
  - files_pdfviewer: 2.7.0
  - files_rightclick: 1.5.0
  - files_sharing: 1.18.0
  - files_versions: 1.19.1
  - logreader: 2.11.0
  - lookup_server_connector: 1.14.0
  - notes: 4.7.2
  - notifications: 2.14.0
  - oauth2: 1.14.0
  - password_policy: 1.16.0
  - provisioning_api: 1.16.0
  - related_resources: 1.1.0-alpha1
  - settings: 1.8.0
  - sharebymail: 1.16.0
  - systemtags: 1.16.0
  - theming: 2.1.1
  - theming_customcss: 1.13.0
  - twofactor_backupcodes: 1.15.0
  - updatenotification: 1.16.0
  - viewer: 1.10.0
  - workflowengine: 2.8.0
Disabled:
  - activity: 2.18.0 (installed 2.14.3)
  - admin_audit: 1.16.0
  - bruteforcesettings: 2.6.0 (installed 2.3.0)
  - circles: 26.0.0 (installed 26.0.0)
  - contactsinteraction: 1.7.0 (installed 1.7.0)
  - dashboard: 7.6.0 (installed 7.1.0)
  - encryption: 2.14.0
  - extract: 1.3.5 (installed 1.3.5)
  - federation: 1.16.0 (installed 1.16.0)
  - files_external: 1.18.0
  - files_texteditor: 2.15.0 (installed 2.15.0)
  - files_trashbin: 1.16.0 (installed 1.11.0)
  - firstrunwizard: 2.15.0 (installed 2.15.0)
  - nextcloud_announcements: 1.15.0 (installed 1.15.0)
  - photos: 2.2.0 (installed 1.3.0)
  - privacy: 1.10.0 (installed 1.10.0)
  - recommendations: 1.5.0 (installed 1.0.0)
  - serverinfo: 1.16.0 (installed 1.12.0)
  - support: 1.9.0 (installed 1.9.0)
  - survey_client: 1.14.0 (installed 1.9.0)
  - suspicious_login: 4.4.0
  - text: 3.7.2 (installed 3.3.0)
  - twofactor_totp: 8.0.0-alpha.0
  - user_ldap: 1.16.0
  - user_status: 1.6.0 (installed 1.1.1)
  - weather_status: 1.6.0 (installed 1.1.0)

Nextcloud Signing status

No response

Nextcloud Logs

{"reqId":"***REMOVED SENSITIVE VALUE***","level":1,"time":"2023-03-30T11:50:23+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"--","app":"core","method":"GET","url":"/login","message":"Tried to log in user but could not verify token","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15","version":"26.0.0.11","data":{"app":"core"}}

Additional info

No response

smart7324 avatar Mar 30 '23 15:03 smart7324

I also had this issue today and I could only fix it with a database maintenance run (command below). my environment infos:

root@Nextcloud:# apache2 -v
Server version: Apache/2.4.41 (Ubuntu) Server built: 2023-03-08T17:32:54 root@Nextcloud:# php --version PHP 8.1.17 (cli) (built: Mar 16 2023 14:38:17) (NTS) Copyright (c) The PHP Group Zend Engine v4.1.17, Copyright (c) Zend Technologies with Zend OPcache v8.1.17, Copyright (c), by Zend Technologies root@Nextcloud:# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal root@Nextcloud:# cat /var/www/nextcloud/version.php $OC_Version = array(26,0,0,11); $OC_VersionString = '26.0.0'; $OC_Edition = ''; $OC_Channel = 'stable'; $OC_VersionCanBeUpgradedFrom = array ( 'nextcloud' => array ( '25.0' => true, '26.0' => true, ), 'owncloud' => array ( '10.11' => true, ), ); $OC_Build = '2023-03-21T09:23:03+00:00 62cfd3b4c9ff4d8cdbbe6dcc8b63a1085bb94e3d'; $vendor = 'nextcloud';

(PostgreSQL) 12.14 (Ubuntu 12.14-0ubuntu0.20.04.1)

How I fix the loop: alias FIX_LOOP='cd /var/www/nextcloud && sudo -u www-data php ./occ maintenance:repair' and then wait 30 minutes for the rate limiting to cool down.

iOS is the latest 16.04 (20E247)

here is an excerpt from my logs when I tried to log in with my admin account: Screenshot 2023-04-04 at 21 22 02

Please answer to this if I should provide more info

TheCrimsonLady avatar Apr 04 '23 19:04 TheCrimsonLady

I gave it a try, but this didn't work for me. Same issue. It also happened to me on a clean new install. So we definitely need help here. At this time NC is completely unusable on Safari no matter what apple device...

smart7324 avatar Apr 05 '23 19:04 smart7324

(moved from #33919)

This problem does not seem to have been solved in v26.0.0.11 - even though https://github.com/nextcloud/server/pull/35419 was merged - seeing how as I'm currently unable to login using Firefox/Android on a device which had a single tab open yesterday. Deleting site data does not change this, nor does running occ maintenance:repair.

I can login using a different browser but not with Firefox, all I get is an empty page showing the site logo and the footer - there is no error message but no login/password request either.

This does not work:

  • deleting all site data and cookies for the domain
  • force-stopping and restarting Firefox/Android
  • rebooting the device
  • trying different network connections - wifi, 4G, VPN
  • deleting browsing history for the affected domain
  • using another open session to delete all sessions for the affected device (in Settings->Security->Devices & Sessions)
  • updating Firefox/Android (to 111.0)
  • running occ maintenance:repair
  • burning black candles in a fairy ring in the forest while chanting obscure incantations (well, did not try but I don't think it would work)

This does work:

  • using a different browser
  • using private mode in Firefox

The error message in the log is the one which has been shown countless times already: Tried to log in "username" but could not verify token:

{"reqId":"aupvuif3Msicz86FxhbY","level":1,"time":"April 06, 2023 06:06:04","remoteAddr":"192.168.9.2","user":"--","app":"core","method":"GET","url":"/login","message":"Tried to log in frank but could not verify token","userAgent":"Mozilla/5.0 (Android 9; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0","version":"26.0.0.11","data":{"app":"core"}}
{"reqId":"q8JEudtB0oT3gfNqLYye","level":1,"time":"April 06, 2023 06:06:04","remoteAddr":"192.168.9.2","user":"--","app":"core","method":"GET","url":"/apps/theming/image/background?v=27","message":"Tried to log in frank but could not verify token","userAgent":"Mozilla/5.0 (Android 9; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0","version":"26.0.0.11","data":{"app":"core"}}

The really annoying thing is that I do not get a chance to login at all since the login/password request does not show up - only the site logo and the footer on an otherwise empty page.

Yetangitu avatar Apr 06 '23 06:04 Yetangitu

Another thing which does work:

  1. enable USB debugging in Firefox/Android
  2. connect it to another machine though USB
  3. open the debugger on the Nextcloud tab
  4. go to the Network section
  5. make sure that 'Disable cache' is checked
  6. reload the tab

This way I do get a login/password request. It seems that Firefox' Clear cookies and site data is not enough to actually clear everything related to the page.

Yetangitu avatar Apr 06 '23 12:04 Yetangitu

Update: This now happens multiple times per day, which is a lot worse than it was before updating to NC 26

TheCrimsonLady avatar Apr 07 '23 06:04 TheCrimsonLady

This is really a serious issue. Right now, I can't use NC with Safari... I am getting logged out every page refresh, so it's completely unusable. Are there any updates? :)

smart7324 avatar Apr 30 '23 12:04 smart7324

Some of the new issues could be related to a safari bug: https://bugs.webkit.org/show_bug.cgi?id=255524

mafjensengithub avatar May 13 '23 10:05 mafjensengithub

Maybe iOS 17 brings a change or the root cause is found somewhere else, either way I hope this will soon be solved because sometimes I can’t log into my NC for days

TheCrimsonLady avatar May 21 '23 10:05 TheCrimsonLady

Seems to be fixed for me with iOS 16.5 and macOS 13.4.

smart7324 avatar Jun 05 '23 06:06 smart7324

Updated a few days ago and for me it seems to be just as bad as before. Haven’t replied earlier because I wanted to gather some data.

TheCrimsonLady avatar Jun 05 '23 06:06 TheCrimsonLady

I'm no longer experiencing any issues, also on NC 27.0.0. We can close here.

smart7324 avatar Jun 14 '23 17:06 smart7324

I updated ~12h ago and just had this issue reappear. Setup is NC in a Ubuntu 20.04 LXC run on Proxmox 7.4-3.

Kernel: 5.15.107-2-pve Ubuntu: Ubuntu 20.04.6 LTS PHP: PHP 8.1.17 (cli) (built: Mar 16 2023 14:38:17) (NTS) Copyright (c) The PHP Group Zend Engine v4.1.17, Copyright (c) Zend Technologies with Zend OPcache v8.1.17, Copyright (c), by Zend Technologies Apache Server version: Apache/2.4.41 (Ubuntu) NC version: 27.0.0.8

Reverse Proxy: Nginx-Proxy-Manager RP version: 2.10.3

Client: iOS 16.5 - Safari Screenshot 2023-06-16 at 07 51 26 Screenshot 2023-06-16 at 07 52 19 Screenshot 2023-06-16 at 07 50 11

Did you do anything else than simply updating NC to fix this? It is getting more and more frustrating to use NC since I can't access it ~50% of the time I need to

TheCrimsonLady avatar Jun 16 '23 05:06 TheCrimsonLady

Okay hm, I also didn’t experience the bug on NC 26 since iOS 16.5… I did not change anything, but it’s just working.

So I reopen this issue for you.

smart7324 avatar Jun 16 '23 07:06 smart7324

Thanks a lot

that’s weird… Do you think my or any reverse proxy could be an issue since my TLS connection is terminated there? I can’t really think of anything else that could cause this in my setup

TheCrimsonLady avatar Jun 16 '23 07:06 TheCrimsonLady

Honestly I don't think so, as I also had this issue and don't have a reverse proxy. I also did some debugging, but I haven't found anything... Is it working with other browsers for you?

smart7324 avatar Jun 16 '23 07:06 smart7324

I rarely use other devices to access my NC, but I had a few situations where this error occurred with my employer provided laptop. On my Debian laptop with Firefox, I had a kinda similar error where i was kinda logged in, but was repeatedly kicked out of NC with the error message in the browser „you are not logged in“. Even when I logged out and back in, this error would persist. I blamed a weird cookie issue and just let it be.

Another possibility that just came to mind: I’m basically always connected to my VPN server at home, which gives my phone, my Mac and NC the same public IP address. Could this be an issue?

(Just for clarification: the issue for me is almost exclusively in iOS, macOS only caused this error once since NC 24 plus the rare occurrences on windows or Linux with Firefox)

TheCrimsonLady avatar Jun 16 '23 07:06 TheCrimsonLady

Hm very interesting… Sorry, but I don’t know if your ip can be a source of the issue. Maybe someone else can help?

smart7324 avatar Jun 16 '23 10:06 smart7324

Yeah me neither, I’m just throwing guesses at the wall here to see what sticks haha

To anyone reading this: all suggestions are welcome

Btw, I played around on my work phone (also iPhone and safari) and was able to provoke the error relatively quickly with two open tabs and some reloads/NC-App switching The error occurred but I was not logged out however, that also happens a lot

TheCrimsonLady avatar Jun 16 '23 11:06 TheCrimsonLady

Hi, We have also been struggling with this problem for about two months. Even an update to version 27 has not brought any improvement. On the contrary, we have the feeling that the bug has increased significantly in recent weeks. In the meantime, our power users can no longer use Nextcloud on certain days.

Even deleting the cookies only helps to a limited extent. After deleting them, they are simply set again and the problem is back.

Our Nextcloud is connected to a very large LDAP directory of our institution. We have about 70 active users (once a week) and about 20 power users (every day, several hours). We are thinking that a connection to the LDAP could be increasing the problem, but probably the trigger is somewhere else.

Access is via a reverse proxy (nginx). There, too, we have already changed some settings for header modification, but without any noticeable effect. In addition, the token errors are occurring more and more frequently with reports of a brute force attack. For this reason, we have to deactivate the brute force detection in the meantime in order not to be locked out all the time. Apparently, Nextcloud counts every expired cookie as a failed login.

It is frustrating. The error pattern is so varied that it is difficult for us to identify the origin of the error.

MrRies avatar Jul 20 '23 13:07 MrRies

Yes, that’s also my experience And that’s on a very small instance with only me as a user. what client devices do your users use? Maybe we have an overlap and can help narrow down the scope for the devs

TheCrimsonLady avatar Jul 20 '23 13:07 TheCrimsonLady

Yes, great idea. We have tested our way through various browsers: Chrome, Edge, Firefox and Opera. The problem is the same everywhere. Most users use Windows machines. However, the problem also occurs with our iOS, iPadOS and Android users. Also with Safari, Brave, Opera, Chrome... We haven't had a chance to test it on MacOS yet.

Sometimes our users are even logged out of the Nextcloud apps (iOS+Android). Talk in particular (which we use a lot).

We initially thought there was a connection with the use of Nextcloud calendars via CardDAV or in connection with app passwords, which a handful of our users are using. However, we could not find any further evidence for this.

MrRies avatar Jul 20 '23 13:07 MrRies

yeah, that's pretty much my device variety, just at a way smaller scale. I have an iPhone with iOS 16.5.1(c), an iMac with macOS 13.2.1 (both most recent Safari), Laptop with Debian 11 + Firefox and a Windows laptop from my employer with Windows 10, now windows 11. All devices had the issue with always the same symptoms that have been described here multiple times.

Just now I updated my NC instance to 27.0.1 and I could not immediately provoke the error. I'll report back in a few days if the issue surfaces again.

Edit: just finished reading through the changelog and at the very bottom, one point stood out "Send CSRF token in rawStat": https://github.com/nextcloud/viewer/pull/1798 @MrRies MAYBE, just maybe, our issue could be solved with this. If possible in your large setup, try updating to 27.0.1 and see if it behaves differently.

TheCrimsonLady avatar Jul 21 '23 23:07 TheCrimsonLady

update after a few days:

the message "tried to log in $USER but could not verify token" still appears when tabs get reactivated (browser opened after some Time on iOS e.g.) or occasionally when you have multiple tabs open, but no issues as in kicked out, rate limited and unable to log in.

So from my user+admin perspective, the symptoms are mended but the cause still persists in some form. Since it's on multiple OSs and different browsers on these and NC changed behaviour after an update, I assume the issue is still somewhere in NC.

I'll post another update in like 2 weeks or when I am facing the initial issues again.

anything new in your setup? @MrRies

TheCrimsonLady avatar Jul 24 '23 21:07 TheCrimsonLady

Screenshot 2023-08-09 at 22 29 00 Update from my setup: It's happening rarely, but still relatively predictable. For example, a tab has been open for a few days but not active for most of this time, then another gets opened. This, in my case, triggers the spew of "tried to log in $USER but could not verify token" messages (see screenshot).

Even after the brute force cooldown and a successful login, I get these log messages but without a kick.

Would be great if anyone from the team took a look in here to tell us what other infos to provide. My setup did not change from this message.

TheCrimsonLady avatar Aug 09 '23 20:08 TheCrimsonLady

I finally lost my patience and tried to assign NC a "unique" domain. I have quite a few subdomains and cnames under the domain that NC runs on, which lead me to the hail mary to move it to its own domain.

I'll report back with my findings in about a week or earlier if it's the same as before.

TheCrimsonLady avatar Aug 10 '23 12:08 TheCrimsonLady

Never mind, my loose theory was proven wrong unsurprisingly. As soon as I have one „older“ tab and open another, I get kicked out immediately.

It seems NC is messing up the cookies with different tabs, like it tries to verify tab1 with cookie 2 and thus throws errors.

I just don’t know how to troubleshoot this or what information to provide from where

TheCrimsonLady avatar Aug 13 '23 11:08 TheCrimsonLady

Sorry for the long absence. We also updated to 27.0.1, but this did not lead to any change.

Our access is via a subdomain using a reverse proxy (nginx). Out of sheer desperation, we tried deactivating the settings "Block Common Exploits" and "Cache assets" in the nginx proxy manager for this subdomain. This resulted in the error message appearing just as often, but it no longer has any consequences for the users. We can currently work in several tabs again without any problems. However, the problem is so random that we are not sure whether this was really the cause of the problem or pure coincidence.

MrRies avatar Aug 16 '23 11:08 MrRies

Sorry for the long absence. We also updated to 27.0.1, but this did not lead to any change.

Our access is via a subdomain using a reverse proxy (nginx). Out of sheer desperation, we tried deactivating the settings "Block Common Exploits" and "Cache assets" in the nginx proxy manager for this subdomain. This resulted in the error message appearing just as often, but it no longer has any consequences for the users. We can currently work in several tabs again without any problems. However, the problem is so random that we are not sure whether this was really the cause of the problem or pure coincidence.

that's a valuable tip, I'll try to deactivate that as well

TheCrimsonLady avatar Aug 16 '23 11:08 TheCrimsonLady

We are seeing the same problem with NC 27.0.1 on PHP 8.1

Log gets flooded every some seconds with "Tried to log in $USER but could not verify token". Only very few users are affected by it.

dafi87 avatar Aug 24 '23 13:08 dafi87

I applied the "trick" @MrRies suggested (deactivate caching, web sockets and 'block common exploits' in Nginx proxy manager) and it seems to remedy the symptoms i.e. being kicked out. However I still have the messages in the log, but for some reason NC doesn't lock me out anymore.

So, it could be a (hopefully) temporary fix until the actual root issue is fixed.

TheCrimsonLady avatar Aug 26 '23 20:08 TheCrimsonLady