alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

Alluxio Fuse should improve the sequential write handling with Libfuse 3

Open yyongycy opened this issue 2 years ago • 2 comments

Is your feature request related to a problem? Please describe. It is possible the workload writes sequentially a file with a large buffer, eg 1M, in Fuse side, the offset is not handled in order, it could be a slightly out of order, for example, the second 128k of 1M is firstly handled by Fuse thread, then the first 128k of 1M is shown. So far Alluxio Fuse will just error out.

Describe the solution you'd like Alluxio Fuse can simply buffer the data received out of order, and write them if all data ahead in logic offset is received.

Describe alternatives you've considered It is possible to check if the fuse kernel buffer can be increased to cater the same size of write https://github.com/libfuse/libfuse/blob/master/ChangeLog.rst#libfuse-360-2019-06-13 But that depends on the linux kernel

Urgency This is important

Additional context Fuse debug log which is indicating out of order write request write[4] 131072 bytes to 47054848 flags: 0xc002 write[4] 131072 bytes to 46399488 flags: 0xc002 write[4] 131072 bytes to 47054848 write[4] 131072 bytes to 46399488

yyongycy avatar May 18 '22 03:05 yyongycy

@yyongycy did you see this behavior again? and is it reproducible? are we able to close this issue?

LuQQiu avatar Jul 14 '22 23:07 LuQQiu

This is tested with libfuse3, maybe keep it open and add libfuse3 label there.

yyongycy avatar Jul 15 '22 12:07 yyongycy

I reproduce this bug.

huanghua78 avatar Sep 01 '23 11:09 huanghua78