Cobrix error handling of bad records

[1mjbaig]

Background The goal of the process is to process an ebcdic mainframe data file with copybook and load it into Azure Data Lake Gen 2 in a readable text file (UTF-8)

Question @yruslan During processing of mainframe file if Cobrix encounters an issue with a record does it fail the entire process or does it write the error record to a file and process remaining records. Secondly how can we enable error handling option in databricks leveraging cobrix?

Thanks

[yruslan]

Errors are handled in the usual Spark way.

• Errors that affect all records, such as parsing errors, errors in the schema, or record layout will cause the process to stop.
• Errors that affect individual fields, such as incorrectly encoded BCD number, result in these fields being null. The process won't stop.

Currently, Cobrix doesn't have an error record it writes errors to. This is a good idea, and we should implement it at some point.

[1mjbaig]

Thank you for the quick response