devoplib

Forget to mention running the ebcdic conversion in Databricks environment.

Definition of the fields is PIC S9(9)V9(2) COMP-3 Input data is EBCDIC EOD_METRICSCARD_TPD_SVER_AMT EOD_METRICSCARD_TPD_SVER_AMT_debug 367876.90 00036787690C ===> This should be 00036787690D Initially the values are Negative and Suddenly getting POSITIVE...

Thank you very much for confirming it. We got the negative values earlier and data on s3 is not changed. It seems odd that suddenly negative sign is missing. I...

Initially we created one task and that task runs a note book that reads the EBCDIC files and converts to ASCII and creates dataframe. We save the dataframe onto Databricks...

EBCDIC to ASCII conversion process is running in parallel for each table since layout is same like you mentioned above. We have couple of tables that needs to be processed...

Thank you very much. When we retries the task it completes or it fails couple of times and completes finally. Databricks is saying since we are using Cobrix and the...

Definitely I will share If we find anything related to this error. We have been working [sharing logs] with Databricks team and they are saying that it is due to...

Thank you very much for looking into it. Note: In order to complete the job we are retrying the job multiple times. Since we have many streams running parallel they...

Please find below the code that creates the dataframe code_page_name = 'cp037_extended' cobolDataFrame = spark .read.format("za.co.absa.cobrix.spark.cobol.source") .option("ebcdic_code_page", code_page_name) .option("schema_retention_policy", "collapse_root") .option("string_trimming_policy", "none") .option("pedantic", "true") .option("improved_null_detection", "false") .option("copybook", tmp_copybook_file) .load(ebcdic_table_data_file) We...

Excellent. Thank you very much @Vinodkc. @yruslan: Please advise. Thank you.