[Bug]: Execution of pipeline fails when CSV File Input transform has Lazy Conversion flag ON

[basusk]

Apache Hop version?

2.7.0 (2023-11-17 12.19.09)

Java version?

openjdk version "11.0.22" 2024-01-16 LTS

Operating system

macOS

What happened?

When CSV File Input transforms used in this attached pipeline ((rename to .hpl); with input csv files) is set ON, execution of this pipeline fails with following errors while tried on Spark engine -

2024/03/16 15:07:00 - aggr1 - ERROR: java.lang.RuntimeException: java.lang.IllegalStateException: Error decoding bytes for coder: WindowedValue$ValueOnlyWindowedValueCoder(KvCoder(BigEndianLongCoder,PCollectionViews$ValueOrMetadataCoder(HopRowCoder,OffsetRange$Coder))) 2024/03/16 15:07:00 - aggr1 - Caused by: java.lang.IllegalStateException: Error decoding bytes for coder: WindowedValue$ValueOnlyWindowedValueCoder(KvCoder(BigEndianLongCoder,PCollectionViews$ValueOrMetadataCoder(HopRowCoder,OffsetRange$Coder)))2024/03/16 15:07:00 - aggr1 - Caused by: java.io.EOFException 2024/03/16 15:07:00 - aggr1 - at java.base/java.io.ObjectInputStream$BlockDataInputStream.readBoolean(ObjectInputStream.java:3370)

Entire error stack trace attached.

Same pipeline with Lazy Conversion flags turn off in all 3 CSV File Input transforms, pipeline runs fine on Spark -

2024/03/16 15:11:17 - aggr1 - Executing this pipeline using the Beam Pipeline Engine with run configuration 'Spark' WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/saikatbasu/hop/hopClient/plugins/engines/beam/lib/spark-unsafe_2.12-3.4.0.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2024/03/16 15:11:45 - aggr1 - Beam pipeline execution has finished.

SparkExecFineWithLzConvOff.txt SparkExecErrorWithLzConvOn.txt aggr1.hpl.txt SSB.DATES.csv SSB.LINEORDER.csv SSB.CUSTOMER.csv

Issue Priority

Priority: 3

Issue Component

Component: Beam, Component: Hop Gui, Component: Pipelines