gcp-census

gcp-census copied to clipboard

GCP Census should be able to list 10000+ projects, by scheduling next tasks with iteration token

marcin-kolda

We need to make sure the views are created in the correct order, e.g. create ```datasets_cost_estimation_v1_0``` after ```table_metadata_v1_0``` view. Currently the first run might returns an error and the call...

marcin-kolda

We should update our model to support [partitioned tables](https://cloud.google.com/bigquery/docs/partitioned-tables#partitioned_tables) with `field` field. Requires also #10.

marcin-kolda

Retrieve BQ metadata asynchronously to reduce GAE costs

1

We could group retrieving table metadata into larger tasks (i.e. 10 API calls per task) and then retrieve them asynchronously. This should reduce GAE costs.

marcin-kolda

marcin-kolda

Model in version 1.0 doesn't store [partitions summary](https://cloud.google.com/bigquery/docs/querying-partitioned-tables#last-modified) from queries anymore. We should store it in a separate table

marcin-kolda

Both [google-api-python-client](https://github.com/google/google-api-python-client/releases) and [oauth2client](https://github.com/google/oauth2client/releases) have new versions. We should update to the most recent versions. In fact oauth2client seems not to be supported anymore, so should switch to the alternative.

marcin-kolda

Whenever we will update model it should be automatically updated in BigQuery. Example scenarios: - [ ] updated query in view, - [ ] updated description in table/view, - [...

marcin-kolda

marcin-kolda