ChoETL icon indicating copy to clipboard operation
ChoETL copied to clipboard

Missing Field Value error when using POCO

Open JimF42 opened this issue 3 years ago • 8 comments

I am new to ChoETL, so maybe I am doing something wrong. I have CSV that I am trying to read with a header. I have created a class and I have decorated it with ChoCSVFileHeader and numerous iterations of ChoCSVRecordObject along with decorating the individual properties with ChoCSVRecordField(FieldName = "...")

If I run this with new ChoCSVReader<VlanRecord>(@"<my file name>.csv") in my for loop, I will get a consistent error reporting that I have missing field values for one particular field. If I open the same file up in Excel, everything lines up correctly and there are values in the columns I am looking for (I don't need or want all the columns from this system generated CSV).

As a test, I tried removing unneeded columns to see if some unusually long values were throwing something off, but the only thing different is that the "missing" field changes. I even eventually removed all of the columns except the ones I needed (not practical in an on-going basis, but fine for testing), and, again, I get the same results, just a different field value is called out as "missing".

If, as an additional test, I run this into a dynamic object instead of a POCO, it all runs fine. No errors. But I wanted a POCO for better support for the rest of my application.

I'm not sure what I am doing wrong or how to diagnose this any further.

I have not uploaded the file as it would have to be heavily sanitized to be able to do that. If required, I can do that for a couple of lines (happens right away after the header).

I have updated to version 1.2.1.57 and I still see the issue.

c# 11 and .NET 7 running on Windows 10 64-bit.

JimF42 avatar Apr 11 '23 18:04 JimF42

pls post sample csv with POCO class. Or use dotnetfiddle to share sample.

Cinchoo avatar Apr 11 '23 20:04 Cinchoo

@Cinchoo, try this sample. Thanks ChoEtlSample.zip

JimF42 avatar Apr 12 '23 12:04 JimF42

ok, got it. csv comes with quoted values including header. In order to handle it, use QuoteAllFields = true in ChoCSVRecordObjectAttribute

    [ChoCSVFileHeader,
     ChoCSVRecordObject(ErrorMode = ChoErrorMode.ReportAndContinue, ThrowAndStopOnMissingField = true, QuoteAllFields = true)]

Cinchoo avatar Apr 12 '23 12:04 Cinchoo

Interesting. Yup, that works.

One more question though. I have opened this file in Excel and re-saved it and Excel removed all of the quotes (header and data). Why does it still only work unless I use QuoteAllFields = true?

FYI, your documentation on Code Project says that QuoteAllFields doesn't apply to Reader. Apparently, it does apply. Listing 6.1, the bullets below the code, "QuoteAllFields - N/A for reader."

Thanks for the fix though.

JimF42 avatar Apr 12 '23 13:04 JimF42

I'll fix the document. thx for ur input.

Excel, I'm not sure about it.

Cinchoo avatar Apr 12 '23 13:04 Cinchoo

As far as Excel, it isn't specifically an Excel question. It was just the tool that I used to read in the CSV and resaved it. When it resaved it the raw .csv file no longer had even one quote in it (when opened in Notepad++). But, to get the program to work, I still need to use QuoteAllFields=true.

As long as I have this knowledge to make it work, I'm fine with it. It just seemed strange that even with zero double quotes in the file, I still need to use that property.

JimF42 avatar Apr 12 '23 14:04 JimF42

if the csv header comes with quotes, you must use this flag to get it work. Pls share the csv when u can.

Cinchoo avatar Apr 19 '23 19:04 Cinchoo

ChoEtlSample.zip

Here you go. Not a single double quote in the entire file. Thanks.

JimF42 avatar Apr 25 '23 14:04 JimF42