[block-input-refactor] potential validateGossipDataColumnSidecar performance issue if node reconstructs columns at the same time

Open twoeths opened this issue 4 months ago • 0 comments
Describe the bug

there are discrepancy of validateGossipDataColumnSidecar, for some first columns it takes ~20ms to ~50ms, for some last columns it takes up to 0.6s which make the block very late to import
specifically, the last column was received at 3.33s, but only gets validated after 0.635s and it becomes late
I suppose it links to the fact that node has to reconstruct all data columns at the same time
662840:Sep-16 02:11:13.621[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=1, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.6019999980926514, gossipSubnet=5, columnIndex=5, recvToValLatency=0.0009999275207519531, recvToValidation=0.01900005340576172, validationTime=0.018000125885009766
662844:Sep-16 02:11:13.652[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=2, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.628999948501587, gossipSubnet=4, columnIndex=4, recvToValLatency=0.0010001659393310547, recvToValidation=0.023000001907348633, validationTime=0.021999835968017578
662848:Sep-16 02:11:13.654[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=3, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.6319999694824219, gossipSubnet=0, columnIndex=0, recvToValLatency=0.002000093460083008, recvToValidation=0.02200007438659668, validationTime=0.019999980926513672
662852:Sep-16 02:11:13.682[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=4, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.6589999198913574, gossipSubnet=3, columnIndex=3, recvToValLatency=0, recvToValidation=0.023000001907348633, validationTime=0.023000001907348633
662854:Sep-16 02:11:13.683[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=5, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.6610000133514404, gossipSubnet=7, columnIndex=7, recvToValLatency=0.002000093460083008, recvToValidation=0.02200007438659668, validationTime=0.019999980926513672
662856:Sep-16 02:11:13.691[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=6, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.6689999103546143, gossipSubnet=25, columnIndex=25, recvToValLatency=0.0010001659393310547, recvToValidation=0.02200007438659668, validationTime=0.020999908447265625
662858:Sep-16 02:11:13.709[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=7, currentSlot=39656, peerId=16Uiu2HAmHkUkKCGLedyDiYxhsPZf4jHQGbV5yM1iyPfA5s98TnSd, delaySec=1.680999994277954, gossipSubnet=14, columnIndex=14, recvToValLatency=0.002000093460083008, recvToValidation=0.0280001163482666, validationTime=0.026000022888183594
662861:Sep-16 02:11:13.733[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=8, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.696000099182129, gossipSubnet=13, columnIndex=13, recvToValLatency=0.0019998550415039062, recvToValidation=0.03699994087219238, validationTime=0.03500008583068848
662863:Sep-16 02:11:13.733[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=9, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.7009999752044678, gossipSubnet=20, columnIndex=20, recvToValLatency=0.003999948501586914, recvToValidation=0.032000064849853516, validationTime=0.0280001163482666
662865:Sep-16 02:11:13.734[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=10, currentSlot=39656, peerId=16Uiu2HAmHkUkKCGLedyDiYxhsPZf4jHQGbV5yM1iyPfA5s98TnSd, delaySec=1.7090001106262207, gossipSubnet=1, columnIndex=1, recvToValLatency=0.0009999275207519531, recvToValidation=0.02499985694885254, validationTime=0.023999929428100586
662868:Sep-16 02:11:13.743[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=11, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.7160000801086426, gossipSubnet=2, columnIndex=2, recvToValLatency=0.003000020980834961, recvToValidation=0.026999950408935547, validationTime=0.023999929428100586
662870:Sep-16 02:11:13.753[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=12, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.7309999465942383, gossipSubnet=49, columnIndex=49, recvToValLatency=0.003999948501586914, recvToValidation=0.021000146865844727, validationTime=0.017000198364257812
662872:Sep-16 02:11:13.753[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=13, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.7269999980926514, gossipSubnet=6, columnIndex=6, recvToValLatency=0.006999969482421875, recvToValidation=0.026000022888183594, validationTime=0.01900005340576172
662876:Sep-16 02:11:13.783[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=14, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.747999906539917, gossipSubnet=8, columnIndex=8, recvToValLatency=0.012000083923339844, recvToValidation=0.03500008583068848, validationTime=0.023000001907348633
662878:Sep-16 02:11:13.783[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=15, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.753000020980835, gossipSubnet=12, columnIndex=12, recvToValLatency=0.006999969482421875, recvToValidation=0.029999971389770508, validationTime=0.023000001907348633
662880:Sep-16 02:11:13.783[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=16, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.753999948501587, gossipSubnet=11, columnIndex=11, recvToValLatency=0.006000041961669922, recvToValidation=0.029000043869018555, validationTime=0.023000001907348633
662882:Sep-16 02:11:13.784[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=17, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.75, gossipSubnet=31, columnIndex=31, recvToValLatency=0.009999990463256836, recvToValidation=0.03399991989135742, validationTime=0.023999929428100586
662884:Sep-16 02:11:13.787[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=18, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.7569999694824219, gossipSubnet=23, columnIndex=23, recvToValLatency=0.003000020980834961, recvToValidation=0.029999971389770508, validationTime=0.026999950408935547
662886:Sep-16 02:11:13.788[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=19, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.755000114440918, gossipSubnet=41, columnIndex=41, recvToValLatency=0.004999876022338867, recvToValidation=0.03299999237060547, validationTime=0.0280001163482666
662888:Sep-16 02:11:13.793[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=20, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.7609999179840088, gossipSubnet=53, columnIndex=53, recvToValLatency=0.013000011444091797, recvToValidation=0.032000064849853516, validationTime=0.01900005340576172
662890:Sep-16 02:11:13.802[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=21, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.7820000648498535, gossipSubnet=35, columnIndex=35, recvToValLatency=0.0019998550415039062, recvToValidation=0.019999980926513672, validationTime=0.018000125885009766
662892:Sep-16 02:11:13.803[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=22, currentSlot=39656, peerId=16Uiu2HAm4AoFZx2KbUbKikVwcr2WYeCMq5SuuzX2nCvTqTzqDebk, delaySec=1.7720000743865967, gossipSubnet=55, columnIndex=55, recvToValLatency=0.0019998550415039062, recvToValidation=0.03099989891052246, validationTime=0.029000043869018555
662894:Sep-16 02:11:13.808[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=23, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.7860000133514404, gossipSubnet=9, columnIndex=9, recvToValLatency=0.0009999275207519531, recvToValidation=0.020999908447265625, validationTime=0.019999980926513672
662896:Sep-16 02:11:13.820[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=24, currentSlot=39656, peerId=16Uiu2HAm73NYeJFMjvL8pcjafSU19u5aBEW6RJUvQNKzVsDuHz9j, delaySec=1.805999994277954, gossipSubnet=52, columnIndex=52, recvToValLatency=0.0009999275207519531, recvToValidation=0.01399993896484375, validationTime=0.013000011444091797
662900:Sep-16 02:11:13.839[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=25, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.812999963760376, gossipSubnet=29, columnIndex=29, recvToValLatency=0.006999969482421875, recvToValidation=0.026000022888183594, validationTime=0.01900005340576172
662902:Sep-16 02:11:13.842[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=26, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.808000087738037, gossipSubnet=61, columnIndex=61, recvToValLatency=0.004999876022338867, recvToValidation=0.03399991989135742, validationTime=0.029000043869018555
662904:Sep-16 02:11:13.843[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=27, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.809999942779541, gossipSubnet=44, columnIndex=44, recvToValLatency=0.003000020980834961, recvToValidation=0.03299999237060547, validationTime=0.029999971389770508
662906:Sep-16 02:11:13.843[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=28, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.811000108718872, gossipSubnet=21, columnIndex=21, recvToValLatency=0.0029997825622558594, recvToValidation=0.031999826431274414, validationTime=0.029000043869018555
662908:Sep-16 02:11:13.845[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=29, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.815000057220459, gossipSubnet=59, columnIndex=59, recvToValLatency=0.004999876022338867, recvToValidation=0.029999971389770508, validationTime=0.02500009536743164
662910:Sep-16 02:11:13.847[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=30, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.816999912261963, gossipSubnet=22, columnIndex=22, recvToValLatency=0.003000020980834961, recvToValidation=0.029999971389770508, validationTime=0.026999950408935547
662912:Sep-16 02:11:13.848[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=31, currentSlot=39656, peerId=16Uiu2HAkvhUQuoWptkF1s65pYRd2JSVsMnW5r5U4tVAMdHsYtPxW, delaySec=1.8310000896453857, gossipSubnet=27, columnIndex=27, recvToValLatency=0.0009999275207519531, recvToValidation=0.01699995994567871, validationTime=0.016000032424926758
662914:Sep-16 02:11:13.868[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=32, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.8399999141693115, gossipSubnet=62, columnIndex=62, recvToValLatency=0.006999969482421875, recvToValidation=0.0280001163482666, validationTime=0.021000146865844727
662916:Sep-16 02:11:13.869[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=33, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.8410000801086426, gossipSubnet=40, columnIndex=40, recvToValLatency=0.00599980354309082, recvToValidation=0.0279998779296875, validationTime=0.02200007438659668
662918:Sep-16 02:11:13.869[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=34, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.8450000286102295, gossipSubnet=46, columnIndex=46, recvToValLatency=0.0019998550415039062, recvToValidation=0.023999929428100586, validationTime=0.02200007438659668
662920:Sep-16 02:11:13.873[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=35, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=1.8519999980926514, gossipSubnet=39, columnIndex=39, recvToValLatency=0.002000093460083008, recvToValidation=0.020999908447265625, validationTime=0.018999814987182617
662922:Sep-16 02:11:13.874[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=36, currentSlot=39656, peerId=16Uiu2HAmHkUkKCGLedyDiYxhsPZf4jHQGbV5yM1iyPfA5s98TnSd, delaySec=1.8469998836517334, gossipSubnet=10, columnIndex=10, recvToValLatency=0.006000041961669922, recvToValidation=0.02700018882751465, validationTime=0.021000146865844727
662924:Sep-16 02:11:13.876[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=37, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.8429999351501465, gossipSubnet=36, columnIndex=36, recvToValLatency=0.003999948501586914, recvToValidation=0.03299999237060547, validationTime=0.029000043869018555
662926:Sep-16 02:11:13.882[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=38, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.869999885559082, gossipSubnet=45, columnIndex=45, recvToValLatency=0.0010001659393310547, recvToValidation=0.012000083923339844, validationTime=0.010999917984008789
662928:Sep-16 02:11:13.901[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=39, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.875999927520752, gossipSubnet=51, columnIndex=51, recvToValLatency=0.005000114440917969, recvToValidation=0.02500009536743164, validationTime=0.019999980926513672
662930:Sep-16 02:11:13.903[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=40, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=1.875, gossipSubnet=16, columnIndex=16, recvToValLatency=0.005000114440917969, recvToValidation=0.0280001163482666, validationTime=0.023000001907348633
662932:Sep-16 02:11:13.907[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=41, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.877000093460083, gossipSubnet=50, columnIndex=50, recvToValLatency=0.003999948501586914, recvToValidation=0.029999971389770508, validationTime=0.026000022888183594
662934:Sep-16 02:11:13.907[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=42, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.878999948501587, gossipSubnet=26, columnIndex=26, recvToValLatency=0.002000093460083008, recvToValidation=0.0280001163482666, validationTime=0.026000022888183594
662936:Sep-16 02:11:13.909[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=43, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.8959999084472656, gossipSubnet=42, columnIndex=42, recvToValLatency=0, recvToValidation=0.013000011444091797, validationTime=0.013000011444091797
662938:Sep-16 02:11:13.922[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=44, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=1.9019999504089355, gossipSubnet=34, columnIndex=34, recvToValLatency=0.0010001659393310547, recvToValidation=0.019999980926513672, validationTime=0.018999814987182617
662940:Sep-16 02:11:13.929[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=45, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.8980000019073486, gossipSubnet=18, columnIndex=18, recvToValLatency=0.0009999275207519531, recvToValidation=0.03099989891052246, validationTime=0.029999971389770508
662942:Sep-16 02:11:13.929[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=46, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=1.9000000953674316, gossipSubnet=15, columnIndex=15, recvToValLatency=0.003000020980834961, recvToValidation=0.028999805450439453, validationTime=0.025999784469604492
662944:Sep-16 02:11:13.946[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=47, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=1.9179999828338623, gossipSubnet=63, columnIndex=63, recvToValLatency=0.003999948501586914, recvToValidation=0.0280001163482666, validationTime=0.024000167846679688
662946:Sep-16 02:11:13.950[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=48, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.9210000038146973, gossipSubnet=58, columnIndex=58, recvToValLatency=0.0009999275207519531, recvToValidation=0.029000043869018555, validationTime=0.0280001163482666
662948:Sep-16 02:11:13.950[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=49, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.9249999523162842, gossipSubnet=24, columnIndex=24, recvToValLatency=0.003999948501586914, recvToValidation=0.02500009536743164, validationTime=0.021000146865844727
662950:Sep-16 02:11:13.955[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=50, currentSlot=39656, peerId=16Uiu2HAkvEUXzjfXrcijLKe2zAhNp9b73sDZu1q88zbD943vxFrp, delaySec=1.931999921798706, gossipSubnet=28, columnIndex=28, recvToValLatency=0.005000114440917969, recvToValidation=0.023000001907348633, validationTime=0.017999887466430664
662952:Sep-16 02:11:13.956[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=51, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=1.9270000457763672, gossipSubnet=33, columnIndex=33, recvToValLatency=0.003999948501586914, recvToValidation=0.029000043869018555, validationTime=0.02500009536743164
662954:Sep-16 02:11:13.968[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=52, currentSlot=39656, peerId=16Uiu2HAm3L7vrDQqkTz3fXUN1pxBcJWKpMQng9WMKkPFmMAR8QMy, delaySec=1.9549999237060547, gossipSubnet=57, columnIndex=57, recvToValLatency=0.002000093460083008, recvToValidation=0.013000011444091797, validationTime=0.010999917984008789
662956:Sep-16 02:11:13.987[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=53, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=1.9709999561309814, gossipSubnet=60, columnIndex=60, recvToValLatency=0.002000093460083008, recvToValidation=0.016000032424926758, validationTime=0.01399993896484375
662958:Sep-16 02:11:13.987[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=54, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.9730000495910645, gossipSubnet=92, columnIndex=92, recvToValLatency=0.003000020980834961, recvToValidation=0.01399993896484375, validationTime=0.010999917984008789
662960:Sep-16 02:11:13.989[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=55, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=1.9700000286102295, gossipSubnet=32, columnIndex=32, recvToValLatency=0.003000020980834961, recvToValidation=0.01900005340576172, validationTime=0.016000032424926758
662962:Sep-16 02:11:14.003[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=56, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=1.9909999370574951, gossipSubnet=69, columnIndex=69, recvToValLatency=0.0010001659393310547, recvToValidation=0.012000083923339844, validationTime=0.010999917984008789
662964:Sep-16 02:11:14.012[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=57, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2, gossipSubnet=54, columnIndex=54, recvToValLatency=0.002000093460083008, recvToValidation=0.012000083923339844, validationTime=0.009999990463256836
662966:Sep-16 02:11:14.035[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=58, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=2.0209999084472656, gossipSubnet=19, columnIndex=19, recvToValLatency=0.003000020980834961, recvToValidation=0.014000177383422852, validationTime=0.01100015640258789
662981:Sep-16 02:11:14.113[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=59, currentSlot=39656, peerId=16Uiu2HAmKhMPLksZQC1YuCTAxHHV2TzHYj8VoMJVCd8yk4EAyH6T, delaySec=2.0230000019073486, gossipSubnet=79, columnIndex=79, recvToValLatency=0.0009999275207519531, recvToValidation=0.08999991416931152, validationTime=0.08899998664855957
662984:Sep-16 02:11:14.148[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=60, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.052000045776367, gossipSubnet=67, columnIndex=67, recvToValLatency=0.05900001525878906, recvToValidation=0.09599995613098145, validationTime=0.03699994087219238
662986:Sep-16 02:11:14.148[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=61, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.055999994277954, gossipSubnet=76, columnIndex=76, recvToValLatency=0.05500006675720215, recvToValidation=0.09200000762939453, validationTime=0.03699994087219238
662988:Sep-16 02:11:14.148[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=62, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=2.059000015258789, gossipSubnet=43, columnIndex=43, recvToValLatency=0.05200004577636719, recvToValidation=0.08899998664855957, validationTime=0.03699994087219238
662991:Sep-16 02:11:14.160[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=63, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.053999900817871, gossipSubnet=72, columnIndex=72, recvToValLatency=0.057000160217285156, recvToValidation=0.10600018501281738, validationTime=0.04900002479553223
662993:Sep-16 02:11:14.161[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=64, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.061000108718872, gossipSubnet=66, columnIndex=66, recvToValLatency=0.05099987983703613, recvToValidation=0.09999990463256836, validationTime=0.04900002479553223
662995:Sep-16 02:11:14.164[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=65, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.052999973297119, gossipSubnet=38, columnIndex=38, recvToValLatency=0.05800008773803711, recvToValidation=0.11100006103515625, validationTime=0.05299997329711914
662997:Sep-16 02:11:14.170[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=66, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.062999963760376, gossipSubnet=84, columnIndex=84, recvToValLatency=0.04900002479553223, recvToValidation=0.10700011253356934, validationTime=0.05800008773803711
662999:Sep-16 02:11:14.171[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=67, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=2.065999984741211, gossipSubnet=17, columnIndex=17, recvToValLatency=0.04699993133544922, recvToValidation=0.10500001907348633, validationTime=0.05800008773803711
663001:Sep-16 02:11:14.171[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=68, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=2.115000009536743, gossipSubnet=65, columnIndex=65, recvToValLatency=0.003999948501586914, recvToValidation=0.0559999942779541, validationTime=0.05200004577636719
663003:Sep-16 02:11:14.171[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=69, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=2.065000057220459, gossipSubnet=48, columnIndex=48, recvToValLatency=0.04799985885620117, recvToValidation=0.10599994659423828, validationTime=0.05800008773803711
663005:Sep-16 02:11:14.176[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=70, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=2.059000015258789, gossipSubnet=56, columnIndex=56, recvToValLatency=0.05299997329711914, recvToValidation=0.11700010299682617, validationTime=0.06400012969970703
663007:Sep-16 02:11:14.179[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=71, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.122999906539917, gossipSubnet=95, columnIndex=95, recvToValLatency=0.024000167846679688, recvToValidation=0.0559999942779541, validationTime=0.031999826431274414
663009:Sep-16 02:11:14.181[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=72, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=2.0799999237060547, gossipSubnet=37, columnIndex=37, recvToValLatency=0.03299999237060547, recvToValidation=0.10100007057189941, validationTime=0.06800007820129395
663011:Sep-16 02:11:14.184[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=73, currentSlot=39656, peerId=16Uiu2HAmVN2B91woYHVgAXvy9Vd1uWS7YRAFE5t6fRQBc8YogVV7, delaySec=2.117000102996826, gossipSubnet=47, columnIndex=47, recvToValLatency=0.0019998550415039062, recvToValidation=0.06699991226196289, validationTime=0.06500005722045898
663013:Sep-16 02:11:14.186[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=74, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=2.1389999389648438, gossipSubnet=75, columnIndex=75, recvToValLatency=0.00800013542175293, recvToValidation=0.04700016975402832, validationTime=0.03900003433227539
663015:Sep-16 02:11:14.186[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=75, currentSlot=39656, peerId=16Uiu2HAmVN2B91woYHVgAXvy9Vd1uWS7YRAFE5t6fRQBc8YogVV7, delaySec=2.1410000324249268, gossipSubnet=126, columnIndex=126, recvToValLatency=0.006000041961669922, recvToValidation=0.04500007629394531, validationTime=0.03900003433227539
663017:Sep-16 02:11:14.187[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=76, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.11899995803833, gossipSubnet=93, columnIndex=93, recvToValLatency=0.0280001163482666, recvToValidation=0.06800007820129395, validationTime=0.039999961853027344
663019:Sep-16 02:11:14.187[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=77, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.121000051498413, gossipSubnet=96, columnIndex=96, recvToValLatency=0.026000022888183594, recvToValidation=0.06599998474121094, validationTime=0.039999961853027344
663021:Sep-16 02:11:14.196[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=78, currentSlot=39656, peerId=16Uiu2HAkvhUQuoWptkF1s65pYRd2JSVsMnW5r5U4tVAMdHsYtPxW, delaySec=2.180000066757202, gossipSubnet=83, columnIndex=83, recvToValLatency=0.0009999275207519531, recvToValidation=0.016000032424926758, validationTime=0.015000104904174805
663023:Sep-16 02:11:14.196[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=79, currentSlot=39656, peerId=16Uiu2HAmFsFCpV32aqE7wDBCCgy1XT7gDDHJyCoTf7PNYDptaouQ, delaySec=2.177999973297119, gossipSubnet=71, columnIndex=71, recvToValLatency=0.003000020980834961, recvToValidation=0.018000125885009766, validationTime=0.015000104904174805
663025:Sep-16 02:11:14.214[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=80, currentSlot=39656, peerId=16Uiu2HAmKhMPLksZQC1YuCTAxHHV2TzHYj8VoMJVCd8yk4EAyH6T, delaySec=2.200000047683716, gossipSubnet=89, columnIndex=89, recvToValLatency=0.003000020980834961, recvToValidation=0.01399993896484375, validationTime=0.010999917984008789
663027:Sep-16 02:11:14.216[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=81, currentSlot=39656, peerId=16Uiu2HAmKhMPLksZQC1YuCTAxHHV2TzHYj8VoMJVCd8yk4EAyH6T, delaySec=2.2019999027252197, gossipSubnet=112, columnIndex=112, recvToValLatency=0.0010001659393310547, recvToValidation=0.014000177383422852, validationTime=0.013000011444091797
663029:Sep-16 02:11:14.242[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=82, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=2.2279999256134033, gossipSubnet=85, columnIndex=85, recvToValLatency=0.0010001659393310547, recvToValidation=0.014000177383422852, validationTime=0.013000011444091797
663031:Sep-16 02:11:14.245[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=83, currentSlot=39656, peerId=16Uiu2HAkvaPJ7YxKdXyzdrbzsADJvQEKMhgP1GU7WA7nPUNm5vLm, delaySec=2.2300000190734863, gossipSubnet=100, columnIndex=100, recvToValLatency=0.002000093460083008, recvToValidation=0.014999866485595703, validationTime=0.012999773025512695
663035:Sep-16 02:11:14.259[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=84, currentSlot=39656, peerId=16Uiu2HAmDxfZeNjvYHWNCdTVCxMm8g3cUMP4k3Ln96QaBcW5fzyX, delaySec=2.244999885559082, gossipSubnet=70, columnIndex=70, recvToValLatency=0.002000093460083008, recvToValidation=0.014000177383422852, validationTime=0.012000083923339844
663037:Sep-16 02:11:14.261[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=85, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.246000051498413, gossipSubnet=105, columnIndex=105, recvToValLatency=0.003000020980834961, recvToValidation=0.014999866485595703, validationTime=0.011999845504760742
663039:Sep-16 02:11:14.280[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=86, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.2660000324249268, gossipSubnet=110, columnIndex=110, recvToValLatency=0.0009999275207519531, recvToValidation=0.01399993896484375, validationTime=0.013000011444091797
663045:Sep-16 02:11:14.308[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=87, currentSlot=39656, peerId=16Uiu2HAmBK82GcN6zwoQGfEptzm8TE7CzSifjiakdG7kBPp6NCfF, delaySec=2.2769999504089355, gossipSubnet=90, columnIndex=90, recvToValLatency=0, recvToValidation=0.031000137329101562, validationTime=0.031000137329101562
663047:Sep-16 02:11:14.309[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=88, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.2799999713897705, gossipSubnet=115, columnIndex=115, recvToValLatency=0.003000020980834961, recvToValidation=0.029000043869018555, validationTime=0.026000022888183594
663049:Sep-16 02:11:14.309[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=89, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.2820000648498535, gossipSubnet=120, columnIndex=120, recvToValLatency=0.0009999275207519531, recvToValidation=0.026999950408935547, validationTime=0.026000022888183594
663051:Sep-16 02:11:14.310[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=90, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.2829999923706055, gossipSubnet=118, columnIndex=118, recvToValLatency=0.0009999275207519531, recvToValidation=0.026999950408935547, validationTime=0.026000022888183594
663053:Sep-16 02:11:14.322[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=91, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=2.2780001163482666, gossipSubnet=124, columnIndex=124, recvToValLatency=0.004999876022338867, recvToValidation=0.04399991035461426, validationTime=0.03900003433227539
663055:Sep-16 02:11:14.325[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=92, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.2980000972747803, gossipSubnet=97, columnIndex=97, recvToValLatency=0.003000020980834961, recvToValidation=0.026999950408935547, validationTime=0.023999929428100586
663057:Sep-16 02:11:14.329[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=93, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.309000015258789, gossipSubnet=104, columnIndex=104, recvToValLatency=0.002000093460083008, recvToValidation=0.019999980926513672, validationTime=0.017999887466430664
663061:Sep-16 02:11:14.351[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=94, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.3380000591278076, gossipSubnet=107, columnIndex=107, recvToValLatency=0.0019998550415039062, recvToValidation=0.013000011444091797, validationTime=0.01100015640258789
663063:Sep-16 02:11:14.376[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=95, currentSlot=39656, peerId=16Uiu2HAmVN2B91woYHVgAXvy9Vd1uWS7YRAFE5t6fRQBc8YogVV7, delaySec=2.364000082015991, gossipSubnet=81, columnIndex=81, recvToValLatency=0.0009999275207519531, recvToValidation=0.011999845504760742, validationTime=0.010999917984008789
663065:Sep-16 02:11:14.384[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=96, currentSlot=39656, peerId=16Uiu2HAm2BhenjVkS4TwN53whs4pL3y8RTxBRCaSn4Wphn5WSRG9, delaySec=2.371999979019165, gossipSubnet=123, columnIndex=123, recvToValLatency=0, recvToValidation=0.012000083923339844, validationTime=0.012000083923339844
663067:Sep-16 02:11:14.420[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=97, currentSlot=39656, peerId=16Uiu2HAmVN2B91woYHVgAXvy9Vd1uWS7YRAFE5t6fRQBc8YogVV7, delaySec=2.4030001163482666, gossipSubnet=64, columnIndex=64, recvToValLatency=0.0009999275207519531, recvToValidation=0.01699995994567871, validationTime=0.016000032424926758
663069:Sep-16 02:11:14.424[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=98, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.4110000133514404, gossipSubnet=99, columnIndex=99, recvToValLatency=0, recvToValidation=0.013000011444091797, validationTime=0.013000011444091797
663073:Sep-16 02:11:14.470[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=99, currentSlot=39656, peerId=16Uiu2HAmHkUkKCGLedyDiYxhsPZf4jHQGbV5yM1iyPfA5s98TnSd, delaySec=2.453000068664551, gossipSubnet=82, columnIndex=82, recvToValLatency=0, recvToValidation=0.01699995994567871, validationTime=0.01699995994567871
663075:Sep-16 02:11:14.495[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=100, currentSlot=39656, peerId=16Uiu2HAkzkjkAoZMjiFbgtYJdegF1csPm1bqay38zcPw6e435cty, delaySec=2.4769999980926514, gossipSubnet=86, columnIndex=86, recvToValLatency=0.002000093460083008, recvToValidation=0.017999887466430664, validationTime=0.015999794006347656
663077:Sep-16 02:11:14.501[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=101, currentSlot=39656, peerId=16Uiu2HAmDxfZeNjvYHWNCdTVCxMm8g3cUMP4k3Ln96QaBcW5fzyX, delaySec=2.4839999675750732, gossipSubnet=117, columnIndex=117, recvToValLatency=0, recvToValidation=0.01699995994567871, validationTime=0.01699995994567871
663079:Sep-16 02:11:14.535[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=102, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.5160000324249268, gossipSubnet=73, columnIndex=73, recvToValLatency=0.0009999275207519531, recvToValidation=0.01900005340576172, validationTime=0.018000125885009766
663081:Sep-16 02:11:14.568[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=103, currentSlot=39656, peerId=16Uiu2HAmNd3CDF32994mdruR4oJj6dNmCgSb8KqKKTK68xHUxc7F, delaySec=2.5490000247955322, gossipSubnet=30, columnIndex=30, recvToValLatency=0.0009999275207519531, recvToValidation=0.01900005340576172, validationTime=0.018000125885009766
663083:Sep-16 02:11:14.583[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=104, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.561000108718872, gossipSubnet=116, columnIndex=116, recvToValLatency=0.0019998550415039062, recvToValidation=0.021999835968017578, validationTime=0.019999980926513672
663085:Sep-16 02:11:14.607[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=105, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=2.5899999141693115, gossipSubnet=111, columnIndex=111, recvToValLatency=0.0010001659393310547, recvToValidation=0.017000198364257812, validationTime=0.016000032424926758
663087:Sep-16 02:11:14.630[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=106, currentSlot=39656, peerId=16Uiu2HAm9hA4Jztx2gJTLZ3qJB8HXtB19s1HfBJY7JbpngynAhAk, delaySec=2.617000102996826, gossipSubnet=77, columnIndex=77, recvToValLatency=0.0019998550415039062, recvToValidation=0.013000011444091797, validationTime=0.01100015640258789
663096:Sep-16 02:11:14.653[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=107, currentSlot=39656, peerId=16Uiu2HAm2BhenjVkS4TwN53whs4pL3y8RTxBRCaSn4Wphn5WSRG9, delaySec=2.634000062942505, gossipSubnet=113, columnIndex=113, recvToValLatency=0.0009999275207519531, recvToValidation=0.01900005340576172, validationTime=0.018000125885009766
663100:Sep-16 02:11:14.728[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=108, currentSlot=39656, peerId=16Uiu2HAmVN2B91woYHVgAXvy9Vd1uWS7YRAFE5t6fRQBc8YogVV7, delaySec=2.7060000896453857, gossipSubnet=108, columnIndex=108, recvToValLatency=0.0009999275207519531, recvToValidation=0.020999908447265625, validationTime=0.019999980926513672
663102:Sep-16 02:11:14.741[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=109, currentSlot=39656, peerId=16Uiu2HAm6x8rJL7an6BJP5hDBmGLTuGDskh5EDAuXtPGCC8RPED8, delaySec=2.7209999561309814, gossipSubnet=88, columnIndex=88, recvToValLatency=0.002000093460083008, recvToValidation=0.019999980926513672, validationTime=0.017999887466430664
663104:Sep-16 02:11:14.757[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=110, currentSlot=39656, peerId=16Uiu2HAmVN2B91woYHVgAXvy9Vd1uWS7YRAFE5t6fRQBc8YogVV7, delaySec=2.7330000400543213, gossipSubnet=91, columnIndex=91, recvToValLatency=0.0019998550415039062, recvToValidation=0.023999929428100586, validationTime=0.02200007438659668
663108:Sep-16 02:11:14.797[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=111, currentSlot=39656, peerId=16Uiu2HAm3L7vrDQqkTz3fXUN1pxBcJWKpMQng9WMKkPFmMAR8QMy, delaySec=2.7669999599456787, gossipSubnet=98, columnIndex=98, recvToValLatency=0.009000062942504883, recvToValidation=0.029999971389770508, validationTime=0.020999908447265625
663110:Sep-16 02:11:14.801[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=112, currentSlot=39656, peerId=16Uiu2HAm5PwXYLfXtzHBEy6LjVkJpq2atq7xbQ7jpxRXTJjXFf5y, delaySec=2.740000009536743, gossipSubnet=74, columnIndex=74, recvToValLatency=0.03299999237060547, recvToValidation=0.06100010871887207, validationTime=0.0280001163482666
663112:Sep-16 02:11:14.802[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=113, currentSlot=39656, peerId=16Uiu2HAm73NYeJFMjvL8pcjafSU19u5aBEW6RJUvQNKzVsDuHz9j, delaySec=2.7780001163482666, gossipSubnet=103, columnIndex=103, recvToValLatency=0.0029997825622558594, recvToValidation=0.023999929428100586, validationTime=0.021000146865844727
663114:Sep-16 02:11:14.804[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=114, currentSlot=39656, peerId=16Uiu2HAmKhMPLksZQC1YuCTAxHHV2TzHYj8VoMJVCd8yk4EAyH6T, delaySec=2.7720000743865967, gossipSubnet=101, columnIndex=101, recvToValLatency=0.003999948501586914, recvToValidation=0.031999826431274414, validationTime=0.0279998779296875
663116:Sep-16 02:11:14.808[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=115, currentSlot=39656, peerId=16Uiu2HAmFsFCpV32aqE7wDBCCgy1XT7gDDHJyCoTf7PNYDptaouQ, delaySec=2.7850000858306885, gossipSubnet=80, columnIndex=80, recvToValLatency=0.0009999275207519531, recvToValidation=0.023000001907348633, validationTime=0.02200007438659668
663120:Sep-16 02:11:14.903[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=116, currentSlot=39656, peerId=16Uiu2HAkvbn8CAJA16UoNS9GrCwzNdiDdrqtuPLPRaGjRxHPRVnX, delaySec=2.882999897003174, gossipSubnet=68, columnIndex=68, recvToValLatency=0.0010001659393310547, recvToValidation=0.020000219345092773, validationTime=0.01900005340576172
663122:Sep-16 02:11:14.908[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=117, currentSlot=39656, peerId=16Uiu2HAkxwc92eFewHchtBvZ6q1GTZNoZjmuQkP3ypwogMkkVzzz, delaySec=2.8889999389648438, gossipSubnet=127, columnIndex=127, recvToValLatency=0.0010001659393310547, recvToValidation=0.01900005340576172, validationTime=0.017999887466430664
663124:Sep-16 02:11:14.934[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=118, currentSlot=39656, peerId=16Uiu2HAmKhMPLksZQC1YuCTAxHHV2TzHYj8VoMJVCd8yk4EAyH6T, delaySec=2.9119999408721924, gossipSubnet=121, columnIndex=121, recvToValLatency=0.002000093460083008, recvToValidation=0.02200007438659668, validationTime=0.019999980926513672
663366:Sep-16 02:11:15.146[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=119, currentSlot=39656, peerId=16Uiu2HAm6nFqWxohf1T6c6dQSiX5b7i79E5AwqGP8m87Zd6WwYid, delaySec=3.121999979019165, gossipSubnet=114, columnIndex=114, recvToValLatency=0.003999948501586914, recvToValidation=0.023000001907348633, validationTime=0.01900005340576172
663370:Sep-16 02:11:15.173[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=120, currentSlot=39656, peerId=16Uiu2HAmDJysN7PCVdXQg7K8PGgxsNCxixy3sEtvkAVGR4SHmW1S, delaySec=3.1540000438690186, gossipSubnet=87, columnIndex=87, recvToValLatency=0.0009999275207519531, recvToValidation=0.01900005340576172, validationTime=0.018000125885009766
663374:Sep-16 02:11:15.185[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=121, currentSlot=39656, peerId=16Uiu2HAkvbn8CAJA16UoNS9GrCwzNdiDdrqtuPLPRaGjRxHPRVnX, delaySec=3.1670000553131104, gossipSubnet=106, columnIndex=106, recvToValLatency=0.0009999275207519531, recvToValidation=0.017999887466430664, validationTime=0.01699995994567871
663378:Sep-16 02:11:15.215[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=122, currentSlot=39656, peerId=16Uiu2HAkvbn8CAJA16UoNS9GrCwzNdiDdrqtuPLPRaGjRxHPRVnX, delaySec=3.197000026702881, gossipSubnet=109, columnIndex=109, recvToValLatency=0, recvToValidation=0.017999887466430664, validationTime=0.017999887466430664
663399:Sep-16 02:11:15.895[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=123, currentSlot=39656, peerId=16Uiu2HAkvbn8CAJA16UoNS9GrCwzNdiDdrqtuPLPRaGjRxHPRVnX, delaySec=3.239000082015991, gossipSubnet=122, columnIndex=122, recvToValLatency=0.004999876022338867, recvToValidation=0.6559998989105225, validationTime=0.6510000228881836
663405:Sep-16 02:11:15.928[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=124, currentSlot=39656, peerId=16Uiu2HAmC3Lbn9qiGeAAP4rTtUCTNnWau2EH3EwQc7tVD9p4As4t, delaySec=3.36899995803833, gossipSubnet=125, columnIndex=125, recvToValLatency=0, recvToValidation=0.5590000152587891, validationTime=0.5590000152587891
663407:Sep-16 02:11:15.929[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=125, currentSlot=39656, peerId=16Uiu2HAmKhMPLksZQC1YuCTAxHHV2TzHYj8VoMJVCd8yk4EAyH6T, delaySec=3.243000030517578, gossipSubnet=102, columnIndex=102, recvToValLatency=0.0019998550415039062, recvToValidation=0.685999870300293, validationTime=0.6840000152587891
663413:Sep-16 02:11:15.958[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=126, currentSlot=39656, peerId=16Uiu2HAkwGPzYUCEGKycKjeaeNrJ8y2VQVqwVnES5jVdfg3wgSnE, delaySec=3.4719998836517334, gossipSubnet=78, columnIndex=78, recvToValLatency=0.0010001659393310547, recvToValidation=0.48600006103515625, validationTime=0.4849998950958252
663417:Sep-16 02:11:15.960[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=127, currentSlot=39656, peerId=16Uiu2HAmSocKHjFvb4pFPtRw1sqfQEj65DSMfyRPq9HHJkX3viLs, delaySec=3.4700000286102295, gossipSubnet=94, columnIndex=94, recvToValLatency=0.003000020980834961, recvToValidation=0.49000000953674316, validationTime=0.4869999885559082
663421:Sep-16 02:11:15.971[network]         debug: Received gossip dataColumn slot=39656, blockRoot=0x8de7…f59f, timeCreatedSec=1757988673.602, expectedColumns=128, receivedColumns=128, currentSlot=39656, peerId=16Uiu2HAmKKwovwM2z8Tbyz5ehiBeTSC5JbZ2qbFkZGFcT5jkk65C, delaySec=3.3340001106262207, gossipSubnet=119, columnIndex=119, recvToValLatency=0.0009999275207519531, recvToValidation=0.6369998455047607, validationTime=0.6359999179840088
Expected behavior

need to review the timeline of reconstruct data columns. May only trigger it after 4s to make sure it does not affect validator performance
Steps to reproduce

No response
Additional context

No response
Operating system

Linux
Lodestar version or commit hash

mkeil/refactor-block-input-on-unstable
Sep 16 '25 08:09 twoeths