DMS Error: Server error while serving listDirectory

[IgorPelevanyuk]

Hello, We faced with strange issue related to a particular files in FileCatalog directory.

:x: - It is impossbile to list, remove or replace a particular corrupted file. :x: - It is also impossible to perform ls of the directory with corrupted file. :white_check_mark: - It is possible to perform dirac-dms-find-lfns of the directory with corrupted file. And the corrupted files will be listed correctly

If we try to perfrom dls on this particular corrupted file - we see the following error in CLI:

dls /dirac_group/...long_path.../file.data
/dirac_group/...long_path.../file.data:
Traceback (most recent call last):
  File "/home/ipelevan/dirac8/diracos/bin/dls", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ipelevan/dirac8/diracos/lib/python3.11/site-packages/DIRAC/Core/Base/Script.py", line 74, in call
    return entrypointFunc._func()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ipelevan/dirac8/diracos/lib/python3.11/site-packages/DIRAC/Interfaces/scripts/dls.py", line 401, in main
    fccli.do_ls(optstr + p)
  File "/home/ipelevan/dirac8/diracos/lib/python3.11/site-packages/DIRAC/DataManagementSystem/Client/FileCatalogClientCLI.py", line 1204, in do_ls
    dList.addFile(os.path.basename(path), fileDict, {}, numericid)
  File "/home/ipelevan/dirac8/diracos/lib/python3.11/site-packages/DIRAC/DataManagementSystem/Client/DirectoryListing.py", line 20, in addFile
    perm = fileDict["Mode"]
           ~~~~~~~~^^^^^^^^
KeyError: 'Mode'

I modified /DIRAC/DataManagementSystem/Client/DirectoryListing.py to see fileDict. And for corrupted file it is the following:

{'Size': 288, 
  'UID': 17, 
  'Owner': 'username', 
  'GID': 3, 
  'OwnerGroup': 'dirac_group', 
  'Status': 'AprioriGood'}

For correctly working files in the same directory it is the following:
{'Size': 1316683643, 
  'UID': 17, 
  'Owner': 'username', 
  'GID': 7, 
  'OwnerGroup': 'dirac_group', 
  'Status': 'AprioriGood', 
  'FileID': 9030062, 
  'Mode': 509,
  'ModificationDate': datetime.datetime(2025, 3, 19, 1, 38, 2), 
  'CreationDate': datetime.datetime(2025, 3, 19, 1, 38, 2), 
  'GUID': 'BDF52615-4840-4D88-FC29-175BEE376E68', 
  'ChecksumType': 'Adler32', 
  'Checksum': 'ec394835'}

@atsareg gave me a hint that possible reason of this failure is inconsintency in FileCatalogDB. There are two tables related to files: FC_Files and FC_FilesInfo. And for corrupted file there are no data in FC_FilesInfo for a file records in FC_Files. The following query allows checking of inconsistent DB file records with directory path:

select f.FileID, f.Size, dlt.DirName, f.FileName from FC_DirectoryLevelTree dlt right join (select * from FC_Files where FileID in (select FileID from FC_Files where FileID not in (select FileID from FC_FileInfo))) f on f.DirID=dlt.DirID;

For some files there were file replicas on SE, for another there were no replicas. It is yet unclear what led to this kind of issue. Luckly it is not critical for now and easily fixable manually with SQL request (the following SQL request is potentially dangerous, it worth creating DB backup before running it):

delete from FC_Files where FileID in (select FileID from FC_Files where FileID not in (select FileID from FC_FileInfo));

Additional text for search: Server error while serving listDirectory: File entry for is corrupt, please contact the catalog administrator