Add parallel support (TPL and async)

[MarcosMeli]

Let discuss what is the best approach to make the library Task friendly and Thread Safe for concurrent work

[ghost]

Can I Jump in into this?

[MarcosMeli]

Hi @hanu412 of course you can join in, welcome to the team

What idea did you have, maybe return Task<List<TRecord>> in some methods ? Thanks

[ghost]

I kind of built a similar framework where we had to process ~20million records every night in a fixed length files. It purely works on TPL. You're absolutely spot on in saying return a Task. One more thing is we implemented chunking-where we don't load the whole of the file into memory but small chunks using MemoryMapped IO from .Net. We can talk/discuss more on it.

[MarcosMeli]

Great @hanu412 at work we have a TPL Dataflow that uses a FileHelperAsyncEngine as producer and make some transformations and bulkcopy it into the SqlServer

You are using the Microsoft TPL DataFlow package or just the Task Framework of .net ?

[ghost]

It is the simple Task (Task Parallel Library), we have been using for our work. It is NOT the data flow package.