[FAQ] Every time the program runs, it performs a “delete-and-import” kind of operation, as increments are not well supported in BigQuery (no UPDATE/DELETE)

[kirantpatil]

trafficstars

Hi,

I found this project reference from here.

There writer says that, it every time delete whole data and import all data once again, which is not acceptable for production setup where thousands of records are being created per day.

May I know is this issue resolved since now Bigquery table has update api, patch api and delete api ?

I thought this program will sync the data from postgresql db to bigquery everytime it updates and only updated data is sent over the wire.

Please let me know, how this works ?

Regards, Kiran.

[csadai]

Hey @kirantpatil - We haven't had a look at these changes yet, as the current setup still supports our use case pretty well. We will try to have a look in the coming weeks. In the meantime, we'd be happy to review any pull request to improve this project! Very best, Cédric

[kirantpatil]

Hi @csadai,

Our worry is it should not choke the network bandwidth, since every time it tries to import all DB data to bigquery, if my understanding correct here ?

Thanks.