What is the current state of BDW+RDW?

[mark-weghorst]

Question

In 2019 https://github.com/AbsaOSS/cobrix/issues/109 @yruslan stated that BDW+RDW was not currently supported.

Is this still the case?

I have a use case with variable blocked records, and I wanted to check on the current status before embarking on a PR to add support.

Has there been any progress on this front, or is a custom record parser still the way to go here?

[yruslan]

BDW+RDW is still not supported, so as you correctly pointed out custom record header parsers are the way to go there. This is mainly because we don't have example files that contain both BDW and RDW so we don't have a way to test it on mainframe-generated files.

Llet me know if there are any issues when using a custom record header parser. Maybe we can make that record header parser builtin and make direct support for such files.

[mark-weghorst]

Here is the challenge as I see it, when we have a 1:1 relationship between blocks and records, i can work around this quite easily by just treating the BDW as the RDW and applying a record_start_offset of 4 to the job to skip over the RDW.

However when we multiple records in a block it gets harder. I'm not quite sure how to tackle this in a header parser, but maybe you can give me a rough idea of where to go from here.

In the examples here is code that demonstrates how to achieve larger sized RDW's. But the thing I'm stuck on is how does the record parser know when we are on a block boundary?

For example consider the following:

[BDW][RDW]Record[RDW]Record[RDW]Record[RDW][BDW]Record[RDW]Record[RDW]Record[BDW]........

Does this situation require a new reader entirely?

[yruslan]

I see. Do I understand it right that BDW is present every n records (where n is fixed for a file)? If that is the case, the reader can provide a current record number as a constructor parameter (or as a setter). Then, depending on the current record number the record header parser could determine if there is a BDW.

Do you have an example of such a file? Something small synthetic, maybe even manually created? I think since currently, the functionality of custom record header parsers is not flexible enough to handle BDW+RDW, I could play with the parser/reader and come up with something.

[yruslan]

Added a feature request for a way to add support for a more flexible way of parsing records. It would allow implementing files that have BDW+RDW.

[mark-weghorst]

@yruslan I can build you some test data, but can you be more specific about the field types you need in your sample? Would something composed of just X(N) work or do you need some comps?

I was planning on asking this a separate issue, but the question has relevance here too I think:

Is Cobrix only an input Codec or is it possible to create a data file and copybook from an existing data frame?

if such an undocumented animal exists, it would be easy way to build you some test data.

[yruslan]

We are planning to release 2.1.4 today which will have record extractors, which we will use to support BDW+RDW. So it is a big step towards it.

We appreciate if you can provide an example BDW+RDW data file. Even X(1) will do 😄

Regarding the other question. Unfortunately, writing data files is not supported [yet].