dcache
dcache copied to clipboard
dCache
Attribute is not defined: STORAGECLASS
Hello
Alarms on doors dcdoor13 and dcdoor15 on high space and cpu usage
@vgaronne collected the following information
04 May 2023 06:17:07 (WebDAV2-dcdoor13-externalipv6) [] Failed to fetch information for progress marker: failed to query pool dcdoor20_1: Failed to deliver String message <5940928842082369868:-9083128157705912248> to [>dcdoor20_1@dcdoor20oneDomain]: l-AAX295SbNWA-<unknown>-AAX295Ssrgg@core-chimera03Domain is busy (its estimated response time of 2265 ms is longer than the message TTL of 2250 ms).
04 May 2023 06:17:07 (WebDAV2-dcdoor13-externalipv6) [] Failed to fetch information for progress marker: failed to query pool dcdoor20_1: Failed to deliver String message <-5468401769476308638:-4857535659056344874> to [>dcdoor20_1@dcdoor20oneDomain]: l-AAX295SbNWA-<unknown>-AAX295Ssrgg@core-chimera03Domain is busy (its estimated response time of 2290 ms is longer than the message TTL of 2250 ms).
04 May 2023 03:30:48 (WebDAV2-dcdoor13-internal) [door:WebDAV2-dcdoor13-internal@webdav2-dcdoor13_httpsDomain:AAX62SYlyPg] Internal server error: org.dcache.webdav.WebDavException: Request to [>SpaceManager@local:dc214_6@dc214sixDomain] timed out.
04 May 2023 07:51:45 (WebDAV2-dcdoor14-externalipv6) [] Failed to fetch information for progress marker: failed to query pool dcdoor11_1: Request to [>dcdoor11_1@dcdoor11oneDomain] timed out.
04 May 2023 07:51:47 (WebDAV2-dcdoor14-external) [] Failed to fetch information for progress marker: failed to query pool dcdoor11_1: Request to [>dcdoor11_1@dcdoor11oneDomain] timed out.
04 May 2023 07:51:47 (WebDAV2-dcdoor14-externalipv6) [] Failed to fetch information for progress marker: failed to query pool dcdoor11_1: Request to [>dcdoor11_1@dcdoor11oneDomain] timed out.
04 May 2023 07:51:49 (WebDAV2-dcdoor14-external) [] Failed to fetch information for progress marker: failed to query pool dcdoor11_1: Request to [>dcdoor11_1@dcdoor11oneDomain] timed out.
04 May 2023 07:51:49 (WebDAV2-dcdoor14-externalipv6) [] Failed to fetch information for progress marker: failed to query pool dcdoor11_1: Request to [>dcdoor11_1@dcdoor11oneDomain] timed out.
04 May 2023 07:51:51 (WebDAV2-dcdoor14-external) [] Failed to fetch information for progress marker: failed to query pool dcdoor11_1: Request to [>dcdoor11_1@dcdoor11oneDomain] timed out.
04 May 2023 07:51:52 (WebDAV2-dcdoor14-externalipv6) [] Failed to fetch information for progress marker: failed to query pool dcdoor11_1: Request to [>dcdoor11_1@dcdoor11oneDomain] timed out.
On the DMZ pool
04 May 2023 05:22:00 (dcdoor20_1) [door:WebDAV2-dcdoor06-externalipv6@webdav2-dcdoor06_httpsDomain:AAX62rGH5Og RemoteTransferManager PoolDeliverFile 00005D30B210EA9142DEBE2394520310A007] Unexpected failure during state change notification
java.lang.IllegalStateException: Attribute is not defined: STORAGECLASS
at org.dcache.vehicles.FileAttributes.guard(FileAttributes.java:314)
at org.dcache.vehicles.FileAttributes.getStorageClass(FileAttributes.java:627)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:19)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:9)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at org.dcache.pool.migration.Job.accept(Job.java:606)
at org.dcache.pool.migration.Job.stateChanged(Job.java:655)
at org.dcache.pool.repository.v5.StateChangeListeners.lambda$stateChanged$0(StateChangeListeners.java:60)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[root@dcdoor01 ~]# grep 'Attribute is not defined: STORAGECLASS' /var/log/dcache/dcdoor01oneDomain.log |wc
191 1146 13752
```
Thank you.
Carlos
Let's concentrate on exception. Do they occur in the course of normal running or during migration?
Hello,
On the DMZ pools migration is part of the normal operations, they are triggered every time there is a new file written to the system
All the best, Carlos
On Jun 20, 2023, at 10:30 AM, Dmitry Litvintsev @.***> wrote:
Let's concentrate on exception. Do they occur in the course of normal running or during migration?
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/7146#issuecomment-1598907301, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO2BSXU2NTLNLBRCEI3XMGXY5ANCNFSM6AAAAAAXVZL5HI. You are receiving this because you authored the thread.
Hi Carlos. Do you still observe these exceptions regularly? Which dCache version are you running currently?
Hi Lea,
We are on 8.2.30 and still observed
01 Oct 2023 04:54:02 (dcdoor26_1) [door:WebDAV2-dcdoor24-externalipv6@webdav2-dcdoor24_httpsDomain:AAYGo75Kjmg dc212_5 PoolDeliverFile 00000C8739EAEE4C4E7D882624EF8CE85990] Unexpected failure during state change notification
java.lang.IllegalStateException: Attribute is not defined: STORAGECLASS
Hi Carlos. Could you please provide your migration job configuration? Also, are there different storage classes configured in source and target?
Hello Lea,
migration move -storage=cgamboa:USERS -permanent -eager -replicas=1 -target=pgroup PRIMEDISKONLY
Each storage class has its own migration job.
All the best, Carlos
On Oct 13, 2023, at 11:18 AM, Lea @.***> wrote:
Hi Carlos. Could you please provide your migration job configuration? Also, are there different storage classes configured in source and target?
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/7146#issuecomment-1761692059, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMOZBEXOVRW2IEWC33KDX7FLVHAVCNFSM6AAAAAAXVZL5HKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRRGY4TEMBVHE. You are receiving this because you authored the thread.
Hello,
On 9.2.6 I am able to see this issue again
Door log
08 Feb 2024 09:15:11 (dcdoor30_1) [door:WebDAV2-dcdoor33-externalipv6@webdav2-dcdoor33_httpsDomain:AAYQ32WnYuA dc246_7 PoolDeliverFile 0000A41D2EA94A984F59B816EB05D8F7FA85] Unexpected failure during state change notification
java.lang.IllegalStateException: Attribute is not defined: STORAGECLASS
at org.dcache.vehicles.FileAttributes.guard(FileAttributes.java:335)
at org.dcache.vehicles.FileAttributes.getStorageClass(FileAttributes.java:648)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:19)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:9)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at org.dcache.pool.migration.Job.accept(Job.java:606)
at org.dcache.pool.migration.Job.stateChanged(Job.java:655)
at org.dcache.pool.repository.v5.StateChangeListeners.lambda$stateChanged$0(StateChangeListeners.java:60)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
08 Feb 2024 09:15:11 (dcdoor30_1) [door:WebDAV2-dcdoor33-externalipv6@webdav2-dcdoor33_httpsDomain:AAYQ32WnYuA dc246_7 PoolDeliverFile 0000A41D2EA94A984F59B816EB05D8F7FA85] Unexpected failure during state change notification
java.lang.IllegalStateException: Attribute is not defined: STORAGECLASS
at org.dcache.vehicles.FileAttributes.guard(FileAttributes.java:335)
at org.dcache.vehicles.FileAttributes.getStorageClass(FileAttributes.java:648)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:19)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:9)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at org.dcache.pool.migration.Job.accept(Job.java:606)
at org.dcache.pool.migration.Job.stateChanged(Job.java:655)
at org.dcache.pool.repository.v5.StateChangeListeners.lambda$stateChanged$0(StateChangeListeners.java:60)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
08 Feb 2024 09:15:11 (dcdoor30_1) [door:WebDAV2-dcdoor33-externalipv6@webdav2-dcdoor33_httpsDomain:AAYQ32WnYuA dc246_7 PoolDeliverFile 0000A41D2EA94A984F59B816EB05D8F7FA85] Unexpected failure during state change notification
java.lang.IllegalStateException: Attribute is not defined: STORAGECLASS
at org.dcache.vehicles.FileAttributes.guard(FileAttributes.java:335)
at org.dcache.vehicles.FileAttributes.getStorageClass(FileAttributes.java:648)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:19)
at org.dcache.pool.migration.StorageClassFilter.test(StorageClassFilter.java:9)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
at org.dcache.pool.migration.Job.accept(Job.java:606)
at org.dcache.pool.migration.Job.stateChanged(Job.java:655)
at org.dcache.pool.repository.v5.StateChangeListeners.lambda$stateChanged$0(StateChangeListeners.java:60)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Billing log entry
02.08 09:15:10 [pool:dcdoor30_1:transfer] [0000A41D2EA94A984F59B816EB05D8F7FA85,0] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/12/27/AOD.37004481._001305.pool.root.1] bnlt0d1:BNLT0D1@osm 0 0 true {RemoteHttpsDataTransfer-1.1:https://lcgdpmse.dnp.fmph.uniba.sk:443/dpm/dnp.fmph.uniba.sk/home/atlas/atlasdatadisk/rucio/mc16_13TeV/12/27/AOD.37004481._001305.pool.root.1?copy_mode=pull} [door:RemoteTransferManager@dccore03Domain:1707401529498-56842] {10027:"rejected GET: 500 Server Error"}
02.08 09:15:10 [pool:dcdoor30_1@dcdoor30oneDomain:remove] [0000A41D2EA94A984F59B816EB05D8F7FA85,0] [Unknown] bnlt0d1:BNLT0D1@osm {0:"cleaner-disk@dccore03Domain [PoolRemoveFiles]"}