frappe
adding Google BigQuery to Insights as data sources and use queries to visualize data
Hi, I was working on adding Google's BigQuery as an Insights data source but have been confused while doing so. I was expecting the entire flow to be bit modular, but turns out I would have to make a lot more changes than presumed. It was kind of obvious to make a whole lot of changes given the fact that PostgresDB / MariaDB / Frappe site-db provide username and password based connection. Connection string can also be used to establish the connection in these cases. But when it comes to BigQuery, the connection is established using a service-account based authorization method.
I'm having some issues primarily with the understanding of what are the steps after a connection is established.
Until now, I have created a doctype called custom-data-source which takes a project ID and service-account as parameters and establishes a connection to the database.
I have made the changes to this fork. I'm looking for a quick review of the progress I've made. There is a lot of clutter to clean up, bear with me🙂
I have a few more questions:
- Am I following the right path?
- What are the steps further?
- How do I display the tables of my BigQuery DB to Insights dashboard from here?
- How do query builders work?
- Are query builders generic to all relational databases? Or is there a need to create new ones for BigQuery?
- Once I have the data from a data source, how are they visualized on the dashboard?
- Is there a complete workbook of how things work behind the scene for reference?
follow-up, I have made some tweaks around the system and was able to list tables and fetch data from those tables using the underlying methods, had to add google-bigquery-dialect in order to make it work.
Here's the implementation for the same: Internal Insights
