Write to bigquery using DataframeWriter

[fbbergamo]

Is there any problem to implement the writing to Bigquery using DataFrameWriter,

What Do you guys think to implement like the redshift lib?

https://github.com/databricks/spark-redshift/blob/master/src/main/scala/com/databricks/spark/redshift/DefaultSource.scala#L68

I want to use the spark like other formats:

df.write.format("com.samelamin.spark.bigquery").option("writeDisposition", "WRITE_TRUNCATE").save("bigquery-dw:poc_data_pipeline.pricing")

This will help to use this library using the same interface that is already build in my project.

Is there any other way to achieve this?

[samelamin]

Sorry for the late reply @fbbergamo, It is a good idea, there isnt a specific reason we cant do this, as far as I know its only the redshift connector that uses this, so there isnt really a spark standard to writing a connector

I think we did start writing a class but it really didn't go anywhere

You can look here and send a pr

[fbbergamo]

@samelamin thanks, I will work on and let you know. :)