hamilton icon indicating copy to clipboard operation
hamilton copied to clipboard

Published 20 hours ago • verified publisher icon DAGWorks-Inc

pandas gbq

Open skrawcz opened this issue 1 year ago • 4 comments

Implement https://pandas.pydata.org/docs/reference/io.html#google-bigquery as a data loader & saver - https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_gbq.html#pandas.DataFrame.to_gbq.

  1. It should go into pandas_extensions.py, one class for the writer, on class for the reader, e.g. much like how the pandas pickle reader and writer are structured.
  2. There should be requisite tests to exercise the functionality. Though this will be need to be mocked, since we don't want to connect to GCP to run unit tests.
  3. We will want a separate example for this one. Since it requires having a GCP account.
  4. If there's an issue with type hints, let me know and we can chat through it.

skrawcz avatar Sep 22 '23 20:09 skrawcz

working on this

bryangalindo avatar Sep 22 '23 22:09 bryangalindo

@skrawcz for the third requirement, assuming we're talking about the materialization example, where do you want to store the example? (e.g., examples/pandas/materialization/gbq, examples/pandas/gbq/materialization, examples/pandas/materialization/notebook-gbq.ipynb)

bryangalindo avatar Sep 23 '23 19:09 bryangalindo

Yeah let's just put it under examples/pandas/materialization/gbq and have similar script & notebook ? That way it can have its own README too.

skrawcz avatar Sep 25 '23 16:09 skrawcz

I am working on this 👍

Yeah let's just put it under examples/pandas/materialization/gbq and have similar script & notebook ? That way it can have its own README too.

I'm working on this 👍

bryangalindo avatar Sep 30 '23 21:09 bryangalindo