CsvHelper icon indicating copy to clipboard operation
CsvHelper copied to clipboard

dynamic reading speed

Open JoshClose opened this issue 6 years ago • 5 comments

Dynamic reading is really slow. Look into performance.

JoshClose avatar Mar 29 '18 16:03 JoshClose

ExpandoObject locks every time during an add operation. As a result, dynamic reading seems to be very slow.

To solve this problem will need an implementation that does not use ExpandoObject. (Like SqlMapper.DapperRow class in Dapper)

I think there are two implementation plans.

  1. Implement inherit Dictionary<string, object> that implements IDynamicMetaObjectProvider.
  • It is easy to implement, but it is not memory efficient.
  1. Create a class that implements IDictionary<string, object> and IDynamicMetaObjectProvider specialized for CSV
  • Improve memory efficiency by sharing CSV header information.

I write and test the code to improve this problem. Thanks.

shibayan avatar Jun 25 '19 09:06 shibayan

Itterating with foreach over IEnumerable<dynamic> is extremely slow and takes around 5 minutes for 50 thousand records.

superrnovae avatar Oct 03 '22 08:10 superrnovae

@superrnovae An auxiliary library has been created for fast handling. I believe it will help. https://github.com/shibayan/CsvHelper.FastDynamic

shibayan avatar Oct 03 '22 10:10 shibayan

@shibayan So this makes using dynamic faster than using types?

JoshClose avatar Oct 04 '22 14:10 JoshClose

@JoshClose No, it will not be faster than accessing with types because of the overhead present. However, performance should be significantly improved compared to dynamic using ExpandoObject.

shibayan avatar Oct 04 '22 17:10 shibayan