dcache icon indicating copy to clipboard operation
dcache copied to clipboard

XROOTD door :io.netty.channel.StacklessClosedChannelException: null

Open cfgamboa opened this issue 1 year ago • 1 comments

Hello,

Some transfer TPC pull involving xrootd door failed looking at the logs the following trace is observed. The dcache version is 8.2.30 Here the source is the local BNL site, the destination is a US tier 2.

03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] Outbound write on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195]: {}
io.netty.channel.StacklessClosedChannelException: null
	at io.netty.channel.AbstractChannel.close(ChannelPromise)(Unknown Source)
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, connection unexpectedly closed on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195], cause io.netty.channel.StacklessClosedChannelException.
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, connection unexpectedly closed on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195], cause io.netty.channel.StacklessClosedChannelException.
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] Outbound write on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195]: {}
io.netty.channel.StacklessClosedChannelException: null
	at io.netty.channel.AbstractChannel.close(ChannelPromise)(Unknown Source)
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, connection unexpectedly closed on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195], cause io.netty.channel.StacklessClosedChannelException.
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, connection unexpectedly closed on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195], cause io.netty.channel.StacklessClosedChannelException.
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] Outbound write on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195]: {}
io.netty.channel.StacklessClosedChannelException: null
	at io.netty.channel.AbstractChannel.close(ChannelPromise)(Unknown Source)
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, connection unexpectedly closed on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195], cause io.netty.channel.StacklessClosedChannelException.
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, connection unexpectedly closed on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195], cause io.netty.channel.StacklessClosedChannelException.
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, channel inactive event received on [id: 0x413e3757, L:/10.42.64.58:54394 ! R:/10.42.38.96:33142].
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] proxy 36dbf564-2b6d-43e2-b6b4-5e1e150be8d3, channel inactive event received on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195].
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] Outbound write on [id: 0xefcfa27d, L:::/0:0:0:0:0:0:0:0:28195]: {}
io.netty.channel.StacklessClosedChannelException: null
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)
03 Jan 2024 16:33:41 (Xrootd3-dcdoor32-external) [] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.StacklessClosedChannelException: null
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)

Any ideas.

Carlos

cfgamboa avatar Jan 03 '24 23:01 cfgamboa

Hello, I was able to reproduce the issue on https://fts.usatlas.bnl.gov:8449/fts3/ftsmon/#/job/e3cbb1ee-ab1e-11ee-8987-b49691292ed8

It was observed that the IPV6 socket was not created/died on the door.

Restarting the door enabled the process to bind to the IPV6 socket

The transfer works afterwards.

https://fts.usatlas.bnl.gov:8449/fts3/ftsmon/#/job/c0e60992-ab2b-11ee-b271-00163e1051a5

cfgamboa avatar Jan 04 '24 18:01 cfgamboa