HDDS-10685. Short circuit read support in Ozone

[ChenSammi]

What changes were proposed in this pull request?

support short-circuit read for datanode.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10685

How was this patch tested?

new unit tests

• TestDomainSocketFactory
• TestXceiverServerDomainSocket
• TestShortCircuitChunkInputStream

manual test with freon

sammi@SAMMICHEN-MB0 ozone-1.5.0-SNAPSHOT % bin/ozone --jvmargs -Djava.library.path=/Users/sammi/workspace/hadoop-ozone/hadoop-hdds/client/src/test/resources freon rk --num-of-volumes=1 --num-of-buckets=10 --num-of-keys=1000 --replication=ONE --replication-type=RATIS --validate-writes --validate-channel short-circuit --num-of-validate-threads=10 --key-size 1MB
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.slf4j.reload4j.Reload4jServiceProvider@18ef96]
SLF4J(W): Found provider [org.slf4j.simple.SimpleServiceProvider@6956de9]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.slf4j.reload4j.Reload4jServiceProvider@18ef96]
2024-09-24 15:37:01,844 [main] INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2024-09-24 15:37:01,877 [main] INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2024-09-24 15:37:01,877 [main] INFO impl.MetricsSystemImpl: ozone-freon metrics system started
2024-09-24 15:37:02,469 [main] INFO storage.DomainSocketFactory: Trying to load the custom-built native-hadoop library...
2024-09-24 15:37:02,469 [main] INFO storage.DomainSocketFactory: Loaded the native-hadoop library
2024-09-24 15:37:02,469 [main] INFO storage.DomainSocketFactory: short-circuit local reads is enabled within 294209 ns.
2024-09-24 15:37:02,516 [main] INFO freon.RandomKeyGenerator: Number of Threads: 10
2024-09-24 15:37:02,524 [main] INFO freon.RandomKeyGenerator: Number of Volumes: 1.
2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Number of Buckets per Volume: 10.
2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Number of Keys per Bucket: 1000.
2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Key size: 1048576 bytes
2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Buffer size: 4096 bytes
2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: validateWrites : true
2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Number of Validate Threads: 10
2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: cleanObjects : false
2024-09-24 15:37:02,527 [main] INFO freon.RandomKeyGenerator: Data validation is enabled.
2024-09-24 15:37:02,527 [main] INFO freon.RandomKeyGenerator: Starting progress bar Thread.

 0.00% |█                                                                                                    |  0/10000 Time: 0:00:00|  2024-09-24 15:37:02,550 [pool-3-thread-1] INFO rpc.RpcClient: Creating Volume: vol-0-30931, with sammi as owner and space quota set to -1 bytes, counts quota set to -1
2024-09-24 15:37:02,616 [pool-3-thread-8] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-6-04589, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,616 [pool-3-thread-2] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-0-10826, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,616 [pool-3-thread-5] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-3-67064, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,616 [pool-3-thread-7] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-5-65340, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,616 [pool-3-thread-1] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-9-04405, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,616 [pool-3-thread-4] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-2-22496, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,619 [pool-3-thread-3] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-1-81033, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,619 [pool-3-thread-6] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-4-66996, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,621 [pool-3-thread-10] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-8-83075, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,626 [pool-3-thread-9] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-7-63535, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 
2024-09-24 15:37:02,817 [pool-3-thread-6] WARN impl.MetricsSystemImpl: ozone-freon metrics system already initialized!
2024-09-24 15:37:03,022 [pool-3-thread-4] INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl
 0.10% |█                                                                                                    |  10/10000 Time: 0:00:01|  2024-09-24 15:37:03,773 [pool-5-thread-7] INFO scm.XceiverClientShortCircuit: Pipeline-df315eb9-30ee-4765-9e68-7fc16335ce48-XceiverClientShortCircuit is created
2024-09-24 15:37:03,775 [pool-5-thread-7] INFO storage.DomainSocketFactory: DomainSocket(fd=311,path=/Users/sammi/ozone_dn_socket) is created within 1572500 ns
 100.00% |█████████████████████████████████████████████████████████████████████████████████████████████████████|  10000/10000 Time: 0:00:24|  
2024-09-24 15:37:27,539 [main] INFO freon.RandomKeyGenerator: Data generation is completed
2024-09-24 15:37:27,539 [main] INFO freon.RandomKeyGenerator: Data validation is completed
2024-09-24 15:37:31,231 [Pipeline-df315eb9-30ee-4765-9e68-7fc16335ce48-XceiverClientShortCircuit-ReceiveResponse] INFO scm.XceiverClientShortCircuit: receiveResponseTask is closed with java.nio.channels.ClosedChannelException
2024-09-24 15:37:31,243 [main] INFO scm.XceiverClientShortCircuit: DomainSocket(fd=311,path=/Users/sammi/ozone_dn_socket) is closed for 249539d9-f678-4ecf-aa6d-fcbcea75f627(localhost/127.0.0.1)
2024-09-24 15:37:31,243 [Pipeline-df315eb9-30ee-4765-9e68-7fc16335ce48-XceiverClientShortCircuit-SendRequest] INFO scm.XceiverClientShortCircuit: sendRequestTask is interrupted

***************************************************
Status: Success
Git Base Revision: bab77db646e17201dd953cc9f0a805502d24ea45
Number of Volumes created: 1
Number of Buckets created: 10
Number of Keys added: 10000
Replication: RATIS/ONE
Average Time spent in volume creation: 00:00:00,008
Average Time spent in bucket creation: 00:00:00,014
Average Time spent in key creation: 00:00:03,689
Average Time spent in key write: 00:00:02,927
Total bytes written: 10485760000
Total number of writes validated: 10000
Writes validated: 100.0 %
Successful validation: 10000
Unsuccessful validation: 0
Total Execution time: 00:00:29,326
***************************************************

Still working on whether it's possible to test it with robot test.