gitea icon indicating copy to clipboard operation
gitea copied to clipboard

Published 20 hours ago • verified publisher icon go-gitea

Incomplete and Inconsistent SSH Public Key Synchronization with LDAP

Open johnstonjs opened this issue 3 years ago • 8 comments

Gitea Version

1.15.6

Git Version

2.25.1

Operating System

Linux 5.11.0-1020-aws #21~20.04.2-Ubuntu

How are you running Gitea?

Gitea is running from the compiled linux-amd64 binary released via GitHub on an AWS EC2 instance. It is updated regularly via this script I wrote

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/johnstonjs/8ce3ccf2f3cdd1e38053fa9e487acb50

Description

Gitea does not appropriately synchronize SSH Public Keys from the LDAP External User Data source. It is very apparent when a user has 4+ SSH Public Keys in their LDAP entry. When Gitea executes Synchronize External User Data it will not result in all 4+ of the SSH Public Keys being available to Gitea.

Reviewing the logs, it appears that Gitea is marking 1-2 of the keys for deletion every time, and they seem to alternate every time the synchronization is run.

I typically use 4+ different SSH keys, all stored in LDAP. To synchronize git repositories managed by my Gitea server I often have to execute Synchronize External User Data multiple times until the correct keys are loaded in Gitea.

My LDAP configuration is described here.

The log file contained in the Gist above only includes keys for a test user, and shows the behavior of alternately removing keys between manual synchronizations. The LDAP entry was not modified between synchronizations.

I'm happy to provide further details from the logs if needed.

Screenshots

Three keys synchronized, five in the LDAP database. Screen Shot 2021-11-24 at 8 21 34 AM

Now four keys have been synchronized, but Gitea deleted one of the previous keys. Screen Shot 2021-11-24 at 8 22 44 AM

Back to three keys synchronized, same as the first three but in a different order. Screen Shot 2021-11-24 at 8 24 36 AM

johnstonjs avatar Nov 24 '21 13:11 johnstonjs

Hello, we see the exact same issue on our self hosted gitea instance as well.

DB: postgres 11 OS where gitea runs: Debian 10.12 with kernel 4.19.0-17

With the small difference, that it even impacts accounts with less ssh keys stored in ldap. We have seen it for users which have only one ssh keys which fully disappears and also for some who have two keys both or one disappears.

I have had a look and compared the backup files of the authorized_keys file with the db and found that the missing key id (used in the command section) was not present any more in the table public_key, so it got fully removed from the db as it seems.

Had also a look into ldap and did not found any update on the ssh key attributes of the affected users, so it was available from ldap perspective.

After that we performed a resync of the external user data + an update of the authorized_keys file we got them back and the keys which were missing got readded with new ids in the db.

suchAdemon avatar Apr 13 '22 13:04 suchAdemon

I'm seeing this as well. On /admin I run "Synchronize external user data" over and over and see the # of public keys in the summary change each time.

gitea 1.16.7 git: 2.35.1 db: postgres 14 os: gentoo 5.15.44

mintyhippoxyz avatar Jul 25 '22 19:07 mintyhippoxyz

Unfortunately you're just not giving us helpful information to solve this.

We need logs. Add tracelogger to your configuration:

[log]
MODE= ..., traceldap ;
...

[log.traceldap]
MODE=file
FILE_NAME=traceldap.txt
LEVEL=trace
EXPRESSION= (services/auth/source/ldap)|(models/asymkey)

Then run sync_external_users.

Next I need some more information.

  • Is it that users aren't being updated who should be updated?
  • Is it that keys aren't being removed/added that should being added?
  • Is it always the same user or keys?

Is Gitea getting different results from the LDAP each time?

But honestly we need more information about what you are seeing and what is going on because you just haven't told us anything that would allow us to debug further.

zeripath avatar Jul 25 '22 19:07 zeripath

I appreciate the quick response and info on logging! I did notice we had a couple users with old/invalid keys I thought may have been the problem but still happening with those cleaned up.

Here's 5 runs, grepping out just my user. I have 5 keys in ldap. Every other run consistently gives me 3/5 then 4/5. It seems to rotate through all 5 but never gives me all 5 at once.

2022/07/25 15:06:17 ...dap/source_search.go:152:checkAdmin() [T] Checking admin with filter (host=@adm) and base uid=matta,ou=People,dc=companyname,dc=com
2022/07/25 15:06:18 ...s/asymkey/ssh_key.go:404:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Handling Public SSH Key synchronization for user matta
2022/07/25 15:06:18 ...s/asymkey/ssh_key.go:434:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Public Key needs update for user matta (Source:5/DB:3)
2022/07/25 15:06:18 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:v5DrJA2fvHTzqTVf98vndP9AF7O0Hw+f6oedzMM0890]: Added Public SSH Key for user matta
2022/07/25 15:06:18 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:cROZlwWj5md1lv20C1vXSF+ucJliiem4fIaTwXvkz3U]: Added Public SSH Key for user matta
2022/07/25 15:06:18 ...s/asymkey/ssh_key.go:384:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:5iqhLgyKYTE69M920ivQLvqys+YNmfgzYTuftdlMq+Q]: Public SSH Key matta already exists for user
2022/07/25 15:06:18 ...s/asymkey/ssh_key.go:451:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Marking Public SSH Key for deletion for user matta: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILD2mh8nOQnCVqxhYQJ+psCnoPNFIm3B+3fWZiqrg5bZ
2022/07/25 15:06:41 ...dap/source_search.go:152:checkAdmin() [T] Checking admin with filter (host=@adm) and base uid=matta,ou=People,dc=companyname,dc=com
2022/07/25 15:06:42 ...s/asymkey/ssh_key.go:404:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Handling Public SSH Key synchronization for user matta
2022/07/25 15:06:42 ...s/asymkey/ssh_key.go:434:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Public Key needs update for user matta (Source:5/DB:4)
2022/07/25 15:06:42 ...s/asymkey/ssh_key.go:384:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:v5DrJA2fvHTzqTVf98vndP9AF7O0Hw+f6oedzMM0890]: Public SSH Key matta already exists for user
2022/07/25 15:06:42 ...s/asymkey/ssh_key.go:384:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:5iqhLgyKYTE69M920ivQLvqys+YNmfgzYTuftdlMq+Q]: Public SSH Key matta already exists for user
2022/07/25 15:06:42 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:aXb92RtQ4r+RWjlA6zNKaZQkxCl5y87Wl6WwUNEJpp0]: Added Public SSH Key for user matta
2022/07/25 15:06:42 ...s/asymkey/ssh_key.go:451:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Marking Public SSH Key for deletion for user matta: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEe65Hp6EdmA7Vp6M+pasYj0PUHvbyumiv7ddJXjaofB
2022/07/25 15:06:42 ...s/asymkey/ssh_key.go:451:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Marking Public SSH Key for deletion for user matta: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEDUovNUkRMY/VNmUXnVxMrGSQ76GktU2TqJp1YR6Lmv
2022/07/25 15:06:51 ...dap/source_search.go:152:checkAdmin() [T] Checking admin with filter (host=@adm) and base uid=matta,ou=People,dc=companyname,dc=com
2022/07/25 15:06:52 ...s/asymkey/ssh_key.go:404:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Handling Public SSH Key synchronization for user matta
2022/07/25 15:06:52 ...s/asymkey/ssh_key.go:434:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Public Key needs update for user matta (Source:5/DB:3)
2022/07/25 15:06:52 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:v5DrJA2fvHTzqTVf98vndP9AF7O0Hw+f6oedzMM0890]: Added Public SSH Key for user matta
2022/07/25 15:06:52 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:cROZlwWj5md1lv20C1vXSF+ucJliiem4fIaTwXvkz3U]: Added Public SSH Key for user matta
2022/07/25 15:06:52 ...s/asymkey/ssh_key.go:384:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:5iqhLgyKYTE69M920ivQLvqys+YNmfgzYTuftdlMq+Q]: Public SSH Key matta already exists for user
2022/07/25 15:06:52 ...s/asymkey/ssh_key.go:451:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Marking Public SSH Key for deletion for user matta: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILD2mh8nOQnCVqxhYQJ+psCnoPNFIm3B+3fWZiqrg5bZ
2022/07/25 15:07:03 ...dap/source_search.go:152:checkAdmin() [T] Checking admin with filter (host=@adm) and base uid=matta,ou=People,dc=companyname,dc=com
2022/07/25 15:07:04 ...s/asymkey/ssh_key.go:404:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Handling Public SSH Key synchronization for user matta
2022/07/25 15:07:04 ...s/asymkey/ssh_key.go:434:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Public Key needs update for user matta (Source:5/DB:4)
2022/07/25 15:07:04 ...s/asymkey/ssh_key.go:384:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:v5DrJA2fvHTzqTVf98vndP9AF7O0Hw+f6oedzMM0890]: Public SSH Key matta already exists for user
2022/07/25 15:07:04 ...s/asymkey/ssh_key.go:384:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:5iqhLgyKYTE69M920ivQLvqys+YNmfgzYTuftdlMq+Q]: Public SSH Key matta already exists for user
2022/07/25 15:07:04 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:aXb92RtQ4r+RWjlA6zNKaZQkxCl5y87Wl6WwUNEJpp0]: Added Public SSH Key for user matta
2022/07/25 15:07:04 ...s/asymkey/ssh_key.go:451:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Marking Public SSH Key for deletion for user matta: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEe65Hp6EdmA7Vp6M+pasYj0PUHvbyumiv7ddJXjaofB
2022/07/25 15:07:04 ...s/asymkey/ssh_key.go:451:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Marking Public SSH Key for deletion for user matta: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEDUovNUkRMY/VNmUXnVxMrGSQ76GktU2TqJp1YR6Lmv
2022/07/25 15:07:13 ...dap/source_search.go:152:checkAdmin() [T] Checking admin with filter (host=@adm) and base uid=matta,ou=People,dc=companyname,dc=com
2022/07/25 15:07:14 ...s/asymkey/ssh_key.go:404:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Handling Public SSH Key synchronization for user matta
2022/07/25 15:07:14 ...s/asymkey/ssh_key.go:434:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Public Key needs update for user matta (Source:5/DB:3)
2022/07/25 15:07:14 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:v5DrJA2fvHTzqTVf98vndP9AF7O0Hw+f6oedzMM0890]: Added Public SSH Key for user matta
2022/07/25 15:07:14 ...s/asymkey/ssh_key.go:389:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:cROZlwWj5md1lv20C1vXSF+ucJliiem4fIaTwXvkz3U]: Added Public SSH Key for user matta
2022/07/25 15:07:14 ...s/asymkey/ssh_key.go:384:AddPublicKeysBySource() [T] AddPublicKeysBySource[COMPANYNAME LDAP-SHA256:5iqhLgyKYTE69M920ivQLvqys+YNmfgzYTuftdlMq+Q]: Public SSH Key matta already exists for user
2022/07/25 15:07:14 ...s/asymkey/ssh_key.go:451:SynchronizePublicKeys() [T] synchronizePublicKeys[COMPANYNAME LDAP]: Marking Public SSH Key for deletion for user matta: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILD2mh8nOQnCVqxhYQJ+psCnoPNFIm3B+3fWZiqrg5bZ

I can say I'm getting consistent results with ldapsearch. Is there a way I can log the raw ldap results from gitea to better answer if it does or not or is this enough to answer that?

mintyhippoxyz avatar Jul 25 '22 20:07 mintyhippoxyz

Sorry I take that back, 1 of the 5 keys it never pulls in looking at those results closer.

mintyhippoxyz avatar Jul 25 '22 20:07 mintyhippoxyz

The logs I provided in my initial post are similar to those provided by @mattin4d .

It appears that when an SSH key is identified as already existing for a user, it is then deleted.

johnstonjs avatar Jul 28 '22 11:07 johnstonjs

Following up, are there additional details from logs that we can provide that would help resolve this issue?

johnstonjs avatar Sep 18 '22 18:09 johnstonjs

Clicked the wrong button... sorry

Are there additional details from the logs that we can provide that would help resolve this issue?

johnstonjs avatar Sep 18 '22 18:09 johnstonjs

This week I re-installed gitea (1.18.0-rc1), initialized a new MariaDB database, and configured an LDAP authentication source using command line arguments. After doing so, all of my LDAP keys are synchronized all the time.

johnstonjs avatar Dec 24 '22 21:12 johnstonjs