BigQuery chart does not display the Dataproc icon in Lineage

[ROM-mm]

I recently changed the connector version to the spark-3.5-bigquery-0.41.1.jar (Dataproc Serveless) version with the flag: spark.dataproc.enabled=true and noticed that the data is being sent to the DATA LINEAGE API, in However, the dataproc icon does not appear in the graph in bigquery, much less linking the raw table with the trusted one, but the bq icon does. I went back to version 0.39.0 and it worked again.

version of dataproc-runtime 2.2 version of spark-connector 0.41.1

[ngorchakova]

Could you please raise a problem in the public issue tracking for Lineage? There you can provide more technical details that will help us quicker investigate the issue and fix it (what is public issue tracker)

[davidrabinowitz]

@codelixir please take a look

[codelixir]

Hi @ROM-mm,

When you say you changed the bq connector version, are you using the same dataproc serverless version and manually changing the connector version? Or do the two cases you described above run on different dataproc serverless vesions? I just want to understand the different environments and check if we need to account for more variables.

[ROM-mm]

Hi @ROM-mm,

When you say you changed the bq connector version, are you using the same dataproc serverless version and manually changing the connector version? Or do the two cases you described above run on different dataproc serverless vesions? I just want to understand the different environments and check if we need to account for more variables.

[ROM-mm]

@codelixir version of dataproc-runtime 2.2 version of spark-connector 0.41.1

[codelixir]

version of dataproc-runtime 2.2 version of spark-connector 0.41.1

Thanks @ROM-mm, in the example where you use connector version 0.39.0, is it still dataproc runtime 2.2?

[ROM-mm]

@codelixir