flatbuffers

flatbuffers copied to clipboard

Stems from discussion in #8183. This should allow users to be able to access a Uint8List directly without copying over from a list, thus actually offering zero-copy access (for this class, at least) in both the lazy path and the non-lazy path. This is pretty important for signal processing and image processing where we want to avoid any unnecessary list copies.

ByteBuffer has an asInt8List method along with asUint8List, so for symmetry perhaps a Int8List _asInt8List(int offset, int length) method could be added Buffeto rContext, and _FbInt8List could be removed?

ByteBuffer has an asInt8List method along with asUint8List, so for symmetry perhaps a Int8List _asInt8List(int offset, int length) method could be added Buffeto rContext, and _FbInt8List could be removed?

I'm not opposed to that, but I'd like to see some insight from one of the Dart maintainers to see if this aligns with their view for the Dart side. Ideally, we could expand the same to Uint16ListReader to use their more space-efficient dart:typed_data alternatives (assuming those two follow the same pattern of being subtypes of List<int>), but then the question arises should those classes also offer lazy/greedy paths?

Ready for review. I tested using @tompark's example code, with the latest version of flatbuffers on pub.dev as well as this fork, and this fork prints the expected bundle1.image1 is already a Uint8List, len=29149 while the latest version on pub.dev does not. The only difference in my test path is that I manually executed the steps from setup.sh instead of writing a script. All tests are passing as well.

I can provide a test repo if you'd like as well, although setting up Tom's example code also takes <10 minutes, assuming you have Flutter and Python already installed.

I applied the same changes to Int8List and Uint16List, added a missing test for Int8List and applied type checking to each test.

NOTE: I changed the hex values in the expect() to match their correct decimal value to better differentiate between the Int8List/Uint8List tests

I squashed the changes and removed a change related to my other PR to prevent conflicts between the two.

@vaind Hi Ivan, is there any way you might be able to spare some bandwidth in the relatively near future to help review and potentially land both of my open PRs? Cheers.

Thanks a lot. It looks pretty good to me, although there need to be some changes to make the code correct on Big endian machines (I know these are basically non-existent, but since Dart supports them, we should too).

I think I'd rather add a separate path or built-in for big-endian, otherwise the newly added conditions to the tests will cause the tests to fail on big-endian, as the old classes do not return Uint8List, Int8List, Uint16List, etc on the lazy path. I think the only supported test environment (theoretically) would be ARM on Linux running in big-endian mode? I'll try and set up a qemu environment to test.

Thanks a lot. It looks pretty good to me, although there need to be some changes to make the code correct on Big endian machines (I know these are basically non-existent, but since Dart supports them, we should too).

I think I'd rather add a separate path or built-in for big-endian, otherwise the newly added conditions to the tests will cause the tests to fail on big-endian, as the old classes do not return Uint8List, Int8List, Uint16List, etc on the lazy path. I think the only supported test environment (theoretically) would be ARM on Linux running in big-endian mode? I'll try and set up a qemu environment to test.

I think it's OK to have different actual return type on big endian, the only contract is that it is compatible with List, everything else is "just" an optimization. It's OK for tests to assert the return value is the expected type based on endianness though

Thanks a lot. It looks pretty good to me, although there need to be some changes to make the code correct on Big endian machines (I know these are basically non-existent, but since Dart supports them, we should too).

I think I'd rather add a separate path or built-in for big-endian, otherwise the newly added conditions to the tests will cause the tests to fail on big-endian, as the old classes do not return Uint8List, Int8List, Uint16List, etc on the lazy path. I think the only supported test environment (theoretically) would be ARM on Linux running in big-endian mode? I'll try and set up a qemu environment to test.

I think it's OK to have different actual return type on big endian, the only contract is that it is compatible with List, everything else is "just" an optimization. It's OK for tests to assert the return value is the expected type based on endianness though

Sounds good to me. Sorry for the delay. I'll get this updated this week, hopefully on the earlier half as opposed to the later half.

LGTM.

Are you going to do the other lists (int32, ...) too or is this PR finished?

I'll extend this to other types! I'll expand the tests as well.

LGTM.

Are you going to do the other lists (int32, ...) too or is this PR finished?

For this PR specifically, it has been expanded out to the existing List types in the code already. I don't know enough about the flatbuffers library to understand how to expand this out for types that do not already have a Reader class, eg Int16ListReader, Int32ListReader. Note though, the float implementation needs to be corrected, but after that the PR should be complete.

@vaind I couldn't find out what is causing Float64List to fail, so I am going to revert the changes to the Float types and mark them with a todo to return their dart:typed_data equivalent so the PR can be restored to a working state and will be ready for review.

@dbaileychess Ping. Let me know if anything should be changed.