Have you considered looking at System.IO.Pipelines ?

[pbolduc]

Have you looked at using System.IO.Pipelines? Marc Gravell has a blog post about them.

Pipelines - a guided tour of the new IO API in .NET, part 1

It could help reduce the amount data movement in the circular buffer due to Pipelines' Zero copy buffers. I also wonder if there an opportunity to squeeze more performance in the particular case where the files are pure ASCII and avoid converting bytes from files/memory into strings when not required.

[jehugaleahsa]

Yes. I was investigating a Span-based implementation in a local branch. I'm excited for spans and all the parse overloads becoming part of .net standard. Right now, the branch is targeting .net core 2.1.

On Sun, Jul 8, 2018, 3:22 PM Phil Bolduc [email protected] wrote:

Have you looked at using System.IO.Pipelines? Marc Gravell has a blog post about them.

Pipelines - a guided tour of the new IO API in .NET, part 1 https://blog.marcgravell.com/2018/07/pipe-dreams-part-1.html

It could help reduce the amount data movement in the circular buffer due to Pipelines' Zero copy buffers. I also wonder if there an opportunity to squeeze more performance in the particular case where the files are pure ASCII and avoid converting bytes from files/memory into strings when not required.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jehugaleahsa/FlatFiles/issues/35, or mute the thread https://github.com/notifications/unsubscribe-auth/ABTgPrISDBNm_fen9TH2fhx-D_QXLQoWks5uElvcgaJpZM4VG1sb .

[jehugaleahsa]

I recently did some more research in this area. The PipeReader class looks promising: https://docs.microsoft.com/en-us/dotnet/api/system.io.pipelines.pipereader?view=dotnet-plat-ext-5.0.

Minimally, for newer platforms, even using the ArrayPool class could speed up performance for newer platforms and eliminate the need for a custom circular array queue: https://docs.microsoft.com/en-us/dotnet/api/system.buffers.arraypool-1?view=net-5.0

It's a little unclear what platforms will support what.