[Bug]: Dataflow throws out data with cryptic, debug-proof error

[j3pic]

Related Template(s)

I don't know.

What happened?

I set up a Datastream and Dataflow pipeline to move data from MySQL to BigQuery. Immediately, I ran into problems. Some tables were not copied at all, even though the Avro files containing their data were created by Dataflow. I can't verify the contents of these files because Avro files require a schema to be read, and I don't know where to find the schema.

The following error message was logged:

Avro File Read Failure <the path to an Avro file>

After a great deal of searching, I found this repo, and grepped for the error message, finding this:

      /** A class to handle errors which occur during file reads. */
      public static class ReadFileRangesFnExceptionHandler implements Serializable {
    
        /*
         * Applies the desired handler logic to the given exception and returns
         * if the exception should be thrown.
         */
        public boolean apply(ReadableFile file, OffsetRange range, Exception e) {
          LOG.error("Avro File Read Failure {}", file.getMetadata().resourceId().toString());
          return false;
          // return true;
        }
      }

I had to guess where this was being called. I believe this is it:

      public void process(ProcessContext c) throws IOException {
        ReadableFile file = c.element();
    
        FileBasedSource<T> source =
            CompressedSource.from(createSource.apply(file.getMetadata().resourceId().toString()))
                .withCompression(file.getCompression());
        try (BoundedSource.BoundedReader<T> reader = source.createReader(c.getPipelineOptions())) {
          for (boolean more = reader.start(); more; more = reader.advance()) {
            c.output(reader.getCurrent());
          }
        } catch (RuntimeException e) {
          if (exceptionHandler.apply(file, null, e)) {
            throw e;
          }
        }
      }

I looked into one possible defintiion of reader.advance() (the one in KafkaUnboundReader.java) and found that it's a massive method that calls many other methods on several different objects, so it could fail in numerous possible ways. To make matters worse, I can't even be sure that this is the method that is being called. It could be some other method on another class being pulled in from somewhere else.

If it wasn't for the commented-out "return true" in the apply method, a stack trace would have possibly been logged, and maybe I'd know something more.

I downloaded the file, removed the copy in the bucket, and then re-uploaded the file. Then Dataflow successfully read the file.

Beam Version

2.35.0

Relevant log output

2022-08-04 01:24:52.616 PDT Avro File Read Failure gs://<REDACTED>