SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)

[calestyo]

trafficstars

Hey.

In 6.0.9, the SRM log gets filled with:

Feb 25 03:09:15 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:09:15+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc14@webdav_lcg-lrz-dc14:AAWfXPl/nTg webdav.tls_lcg-lrz-dc14 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:09:25 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:09:25+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc13@webdav_lcg-lrz-dc13:AAWfXPoV3hA webdav.tls_lcg-lrz-dc13 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:09:31 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:09:31+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc13@webdav_lcg-lrz-dc13:AAWfXPp6Mag webdav.tls_lcg-lrz-dc13 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:09:32 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:09:32+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc14@webdav_lcg-lrz-dc14:AAWfXPqBC5A webdav.tls_lcg-lrz-dc14 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:09:49 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:09:49+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc13@webdav_lcg-lrz-dc13:AAWfXPuJzOg webdav.tls_lcg-lrz-dc13 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:09:53 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:09:53+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc13@webdav_lcg-lrz-dc13:AAWfXPvAA9A webdav.tls_lcg-lrz-dc13 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:09:58 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:09:58+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc14@webdav_lcg-lrz-dc14:AAWfXPwX2DA webdav.tls_lcg-lrz-dc14 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:10:04 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:10:04+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc14@webdav_lcg-lrz-dc14:AAWfXPxpNFA webdav.tls_lcg-lrz-dc14 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:10:08 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:10:08+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc14@webdav_lcg-lrz-dc14:AAWfXPyrhOA webdav.tls_lcg-lrz-dc14 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:10:18 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:10:18+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc14@webdav_lcg-lrz-dc14:AAWfXP1J92A webdav.tls_lcg-lrz-dc14 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:10:19 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:10:19+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc13@webdav_lcg-lrz-dc13:AAWfXP1UHQg webdav.tls_lcg-lrz-dc13 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:10:22 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:10:22+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc14@webdav_lcg-lrz-dc14:AAWfXP18r8A webdav.tls_lcg-lrz-dc14 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)
Feb 25 03:10:28 lcg-lrz-srm dcache@srm_lcg-lrz-srm[20490]: 2020-02-25 03:10:28+01:00 (SrmManager) [door:webdav.tls_lcg-lrz-dc13@webdav_lcg-lrz-dc13:AAWfXP3h8aA webdav.tls_lcg-lrz-dc13 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2082708136 (No such file or directory)

Any idea why this happens?

Cheers, Chris.

[ageorget]

Hi, I'm also flooded by these error messages "error reading the credentials from database" and I may have an explanation. dCache stores webdav credentials in the directory /var/lib/dcache/credentials/, like /var/lib/dcache/credentials/-2147452629 This information can be found in the srmrequestcredentials table of the SRM DB :

select * from srmrequestcredentials;
     id      | creationtime  |                                         credentialname                                          |  role   | numberofusers |          delegatedcredentials  
         | credentialexpiration 
-------------+---------------+-------------------------------------------------------------------------------------------------+---------+---------------+--------------------------------
---------+----------------------
 -2147452629 | 1623852614402 | 1234 /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ewp2c01/CN=817926/CN=Robot: ESCAPE WP2 CERN 01 | /escape |             0 | /var/lib/dcache/credentials/-21
47452629 |        1623859034000

When a dCache instance runs multiple srmmanager service on different host for HA, this credential is store on one of the host. If another srmmanager on a different host is looking for the credential file in the /var/lib/dcache/credentials directory as mentioned in the srmrequestcredentials table, the service is not able no find it.

ie :

ccnessie01[~] tail -f /var/spool/dcache/log/srmDomain.log
16 Jun 2021 16:23:06 (srmmanager) [door:webdav-ccdcalitest10@webdav-ccdcalitest10Domain:AAXE4tRMQzA webdav-ccdcalitest10 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2147452629 (No such file or directory)
16 Jun 2021 16:23:19 (srmmanager) [door:webdav-ccdcalitest10@webdav-ccdcalitest10Domain:AAXE4tUREgA webdav-ccdcalitest10 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2147452629 (No such file or directory)
16 Jun 2021 16:25:52 (srmmanager) [door:webdav-ccdcalitest10@webdav-ccdcalitest10Domain:AAXE4t4nOhg webdav-ccdcalitest10 SrmRequestCredential] error reading the credentials from database: java.io.FileNotFoundException: /var/lib/dcache/credentials/-2147452629 (No such file or directory)

ccnessie01[~] sudo ls /var/lib/dcache/credentials
//nothing
ccnessie02[~] sudo ls /var/lib/dcache/credentials
//nothing
ccnessie03[~] sudo ls /var/lib/dcache/credentials
-2147452629

The credentials should be stored on every host running srmmanager or maybe the host information should be stored in the srmrequestcredentials table.

Adrien

[lemora]

Hi. The SrmManager needs a distinct, separate database per SrmManager as documented in its configuration file in the description of the srmmanager.cell.replicable property. Am I right in assuming that both of you share one database between all your SrmManagers?

Lea

[ageorget]

Oh yes, I missed this part when I configured SrmManagers replication! Is there any other reason to have a separate database per SrmManager than managing these credentials? Thanks for your answer!

[lemora]

When sharing a database, all SrmManagers can restore and try to process the same requests. It has been observed to lead to pinning errors that are logged with exceptions along the lines of "could not serialize access due to concurrent update".

[ageorget]

I see, thanks for the explanation. I'm going to fix that. Cheers