Question: Why do string arrays throw an exception?

[mgoldste1]

 CodeNode GenerateAssignment(KeyValuePair<FieldLayoutAttribute, PropertyInfo> pair)
        {
            var fieldType = pair.Key.FieldType;
            //if this is an array, defer to GenerateArrayAssignment
            if (pair.Key is ArrayFieldLayoutAttribute)
            {
                // TODO: Why do we need these fieldType checks at all?
                if (fieldType == typeof(string))
                {
                    // TODO: Accept string arrays
                    throw new InvalidOperationException(Resources.NoStringArrays);
                }
                if (fieldType == typeof(BitArray))
                {
                    throw new InvalidOperationException(Resources.NoBitArrayArrays);
                }
                return GenerateArrayAssignment(pair);
            }
...
}

The stdf files I'm parsing have some record types that aren't in the spec, but when I search for them on the internet I see other projects implemented them. See https://sprysoftware.net/javadoc/javadoc/spry/reader/stdf/VUR.html I am adding support for these record types:

0-30 - VUR - Version Update Record 1-90 - PSR - Pattern Sequence Record

The VUR record has a string array in it and the code above threw an exception. I commented out the InvalidOperationException for string arrays and it worked fine... so why is it there?

[marklio]

Great question. Given the TODO, I'd say that at some point string arrays didn't work, and that they they started working due to some other change and since string arrays aren't used in any of the other record types that use converter generators, this exception wasn't blocking anything. If string arrays work, then I don't see any need for this exception.

Feel free to submit a PR for that.

Also, I don't know anything about VUR or PSR. Is there an updated spec for these? Those aren't type/subtypes that are in the "custom extensions" area of the spec. I've been out of the semiconductor test space for about 17 years now :).

I'm curious what environment you're running in. I've considered making an updated version of this library that targets .NET 6+ and uses source generators instead of Reflection.Emit to generate the converters, but I have no idea if anyone would use it.

[mgoldste1]

• I found a draft for one here - https://docplayer.net/127928234-Background-statement-for-semi-draft-document-4782-new-standard-specification-for-standard-test-data-format-stdf-memory-fail-datalog.html
• This might be the official 2007 revision's spec - https://github.com/lantianjialiang/stdf_spec/blob/master/stdf_V4-2007_spec.pdf
• Also this paper goes into detail on the changes from STDF V4-2007. https://ieeexplore.ieee.org/document/6252114

I'm curious what environment you're running in. I've considered making an updated version of this library that targets .NET 6+ and uses source generators instead of Reflection.Emit to generate the converters, but I have no idea if anyone would use it.

I converted it to .net 6 earlier today but haven't tested it much. So far I know that I can iterate over all the records and parse them. The code I wrote when it was still on .net framework to convert stdf files to json and xml still works too.

The only thing that wasn't straight forward to fix was with the assembly builder. .net core doesn't support "RunAndSave" yet, but it does work with it set to Run. This was the code change in the record converter factory (also commented out the save method under it) -

        void InitializeDynamicAssembly() {
            _DynamicAssembly = AssemblyBuilder.DefineDynamicAssembly(
                new AssemblyName("DynamicConverters"),
                AssemblyBuilderAccess.Run);
            _DynamicModule = _DynamicAssembly.DefineDynamicModule(
                "DynamicConverters");
        }

How bad is it to use "Run" instead of "RunAndSave"?

It looks like as of 3 weeks ago someone started the code for saving them. The comment on the latest check-in of AssemblyBuilder.cs says "Initial step for adding AssemblyBuilder.Save implementation" https://github.com/dotnet/runtime/tree/main/src/libraries/System.Private.CoreLib/src/System/Reflection/Emit

IMO, if it ain't broke, don't fix it. :)

[marklio]

Cool. There's been support for ".net core style" ref emit since the Silverlight days (I had a working Silverlight builld as part of our internal app building effort for SL (I work on the .NET team). Most everything should work fine. It doesn't use AssemblyBuilder, but the raw DynamicMethod. I have a local branch that works on .net 6 and starts the work on source generators that a started last year. I'll take a look again this week. There's also some good "Span-ification" experiments that I've made and the .NET 6 stream perf stuff makes things REALLY fast.

[mgoldste1]

If you wanted to transition it to .net 6 it'd probably be better for you to do it than me. What I have works, but it could probably be better considering I've barely got my toes wet with STDF files and your project.

If my boss lets me contribute, I can put in a PR for my documentation additions and for some of the new record types. I've been trying to make the project more beginner friendly seeing as I'm a beginner with this stuff... We've used our own proprietary format over here for sort data but we now have a need to parse STDF files and passed into our systems.

[marklio]

If you want, you can take a look at the marklio/net7 branch, which (for some reason) targets .Net 8 (maybe I should rename it). My thinking was that 8 would be out by the time we really got it where we wanted it. If that doesn't work for your environment, I'd be interested to know. It basically moves everything to .Net 8 and attempts to do full nullability annotations, which yielded one bug that's I think has been in the library since 2005 or so (FTRs were straight up missing a field that was described in the attributes, but missing in the type).

The tests pass, but the tests really only round trips records. Since I'm not in the semiconductor industry, I don't have any STDF files to test against. This has been a long-standing issue with me being able to do much meaningful with the library. Most folks are stingy with their test data ;).

There are a few things things I'd like to do as part of a .Net modernization effort

• Gut the IO and update it with modern IO patterns. That's likely to yield a far amount of perf improvement. That likely means moving to System.IO.Pipelines.
• Move as much as possible to predictable allocation patterns. That's likely to yield alot of perf as well.
• Add async codepaths. Much of this will come from the IO work above. Alot of the LINQ-ish constructs don't make much sense in this world, but there are probably some interesting things to do here.
• I'd like to move the record parsers to source generators rather than the reflection.emit patterns. I've actually prototyped this already in the marklio/SourceGeneratorExperiment branch, but I'd like to start over because I think the IO update is likely to yield some architectural changes that simplify things quite a bit.
• Incorporate the new record types in the 2007 spec update. Feel free to make PRs for this work either in the main branch or the net7 branch.
• Address endianness for a broader set of cross-platform scenarios.
• Revisit some fundamental design decisions I made almost 20 years ago (yikes)

I'll probably close this particular issue and make some tracking issues for the work above. I'm likely to make very bursty progress, but this kind of thing is super fun for me when there is shared interest in the outcome.