Flow On-Chain Analytics Dashboard

[Guisong-Fu]

<Flow On-Chain Analytics Dashboard>

Grant category

Developer service

Description

Similar to Dune. A general analytics tool for Flow blockchain.

Problem statement

Dune does not support Flow blockchain yet, and there is no other tool that provides similar services for Flow blockchain.

Proposed solution

• Build a data warehouse that stores chain state
• Build a platform where everyone could do some analysis on Flow blockchain activities

Impact

• Everyone would be able to easily do some analysis on Flow blockchain.

Milestones and funding

Milestone	Deliverables	Timeline	Risks	USD requested
1 - Data Analysis P1; Initial Setup	Determine what data should be collected and how the data is preferred to be visualized by surveying the community; Setup the domain and the landing page	1 week	May need some support from Flow Team	3000
2 - Data Analysis P2	Determine the best way to collect/index data	1 week	May need to consult Flow Team and set up an access node	3000
3 - Data Storage	Determine the suitable table structure	2 weeks	Time	3000
4 - Job Implementation	Build up cloud infrastructure; Implement the corresponding ETL jobs and data pipeline	4 weeks	Time	30000
5 - Standalone Historical Event Indexer Opensourced	Refine the indexer job and make it opensource	1 week	Time	2000
6 - Visualization Tool Set up	Configure SuperSet	3 weeks	May need to tweak SuperSet a bit; may involve some visual desgin work	20000
7 - Integration Test	Dashboard being connected to data warehouse	1 week		3000
8 - Pre-build some dashboards	A few dashboards can be seen	1 week	Some bugs may pop out and some rework maybe needed	3000
9 - Open for testing	Platform are ready to be tested; bounty program is released	2 weeks	May encounter performance issue thus need to scale up the infra	6000
10 - Documentation	Docmentation is ready	1 week		3000
11 - Maintainence	Maintain Cloud Service; Support	3 months		18000

Team

Name	Role	Bio	Contact
Guisong Fu	Data Architect/Engineer	CTO of SoulMade; muti-year experience as Data Architect	[email protected]
SkyOne	Senior Flow Blockchain Developer	Founder of 7DayReFundNFT; core developer of SoulMade	[email protected]

Milestone completion

• [x] Data Analysis (P1+P2)
• [x] Data Storage
• [x] Job Implementation
• [ ] Standalone Historical Event Indexer Opensourced
• [ ] Visualization Tool Set up
• [ ] Integration Test
• [ ] Pre-build some dashboards
• [ ] Open for testing
• [ ] Documentation
• [ ] Maintenance

[alxflw]

thanks for the submission @Guisong-Fu! we'll review this application soon

[alxflw]

@Guisong-Fu we were just chatting about this and we want to see how you're thinking about using an existing indexing service for flow (as opposed to building your own in milestones 2-4). Currently, there's: flowscan, graffle, bitquery and flip side.

Maybe, you could explore with these team if they could offer access to their data warehouse? If not, we suggest you refine your application to account for the projected maintenance cost (given that the data collection will be large).

[Guisong-Fu]

Hi @alxflw , thanks a lot for your suggestion! We do have thought about collaborating with other teams as well. But according to our knowledge and experience, we may still have to set up our own indexer. It's mainly because that if we want to have complete data to analyse, we would need to index all the historical events by crawling all the available blocks, and none of the existing services seems to have done such kind of job, so we have to do it by ourselves. And we had done some tests, it seems that it won't be much more complicated to index the events from the newly coming blocks on our own compared with reading data from for example Flowscan's database and parse data from there. But indeed, maintaining the entire data collection pipeline maybe very costly. I have adjusted our application! Thanks for pointing it out! We have already kicked off this project. And we just learned that for different ranges of blocks, we have to use different access node~ 😄 We will try to crawl all the blocks and extract all the data related to TopShot, and launch the bare bone platform so users could use to build dashboards for TopShot in the up coming weeks. What a milestone TopShot has achieved! 1 billion sales! 👍 I know many have been long for some more thorough analysis on Topshot. It would be fantastic if we could help! We can always have a call at any time and have a deeper discussion. 😃

[Guisong-Fu]

Hi @alxflw , we have developed a proof of concept. Please let me know if you would like to try our prototype and have a further discussion. 😃

[alxflw]

Hey @Guisong-Fu, would love to check it out. You can find me on Discord: alx-flw.find#6198.

One thing that is coming up quite frequently is the need to index the state. I was wondering if you could build your indexer as a standalone opensource project, so other developers could re-use/improve it. Happy to chat more!

[alxflw]

Hi @Guisong-Fu, thanks for the demo. We have a few follow up questions:

1. It would be great if the indexing project could a standalone opensource project that any other developer could use for their own need. Do you think you could refine the milestone to account for that? Essentially, milestone 1-4 would lead up to that deliverable

2. Can you confirm that SuperSet can be set up in a way that allows anyone without an account to access a shared dashboard? We'd consider this viz UI "public infrastructure" for Flow, so it should have the least friction for anyone to view

3. Could you extend milestone 1 to include surveying the community to see what data and visualization needs they have?

4. Could you specify the time you'd be able to maintain the project in milestone 10? I'd suggest we start with 3/6 months. Follow up grants for maintenance are possible.

[Guisong-Fu]

Hi @alxflw , thanks a lot for all the great suggestions! 😄 We now have updated the proposal accordingly!

1. We have added this "opening up the indexer job as a standalone opensource project" as milestone 5, as it will take a bit of refinement work. The idea is that everyone could use this repo as a template so to crawl/index the target events. We are also thinking about setting up a central place(e.g. a shared cloud storage) where users could upload/download/share the scanned data for other use cases. We can take this as our next step.

2. Yes, it can be open to the public audience(anonymous users, users who do not login) if the dashboard builders permit so. Authentication and authorisation are big topics thus we didn't have time to take them in with our demo. But it can definitely be configured in whatever ways we want.

3. Sure. And the idea is that everyone can build the dashboard in whatever ways they want, so we probably do not need to worry too much about how the data should be visualised. But sure, we can pre-build some dashboards in the ways that community would like to see as examples and references for other builders.

4. Thanks for pointing it out! We now have set it to 3 months. The main cost would be coming from the cloud infrastructure, according to our rough estimation, the infra would cost around 1,500 USD per month. But it may go up if there are more needs.

[alxflw]

Fantastic, thanks for the updates and confirmations!

WRT 2: It was pointed out to me that there was pre-existing work done on the indexing. Maybe you could see how that would impact your milestone 5? If the projects below already cover what is needed for indexing, maybe you could use them directly?

• https://github.com/rayvin-flow/flow-scanner
• https://github.com/rayvin-flow/flow-scanner-lib

[Guisong-Fu]

Hi @alxflw , maybe there is a bit of misunderstanding here 😃

In terms of milestone 5, I'm actually more thinking of "an indexer job that indexes all the historical events" rather than the new events. The reason why I think of it in this way is that when we did the proof of concept, we found it actually a bit tricky to crawl all the historical blocks and index the target events. We want to make it easy so everyone could easily crawl all the historical blocks and retrieve the data wanted. I do believe there will be needs on the historical data for other use cases as well besides analysis.

And yes! We know Flow-scanner, it's a pretty cool project! And there are also a few other projects that do the similar thing. Have you seen this one? It's very thorough. 👍 https://github.com/ph0ph0/Get-The-Flow-Down#event-listenersmonitorsindexers

For indexing new events, we will look into all of these options and pick the one that suits us most. And maybe we could work with Graffle as well. 😄

[alxflw]

wonderful, that makes a lot of sense! thanks for clarifying

[Guisong-Fu]

Hi @alxflw , any update on this grant? 🙂 Thanks!

[alxflw]

hi @Guisong-Fu, this proposal was just accepted! 🚀

I'll follow up with you with next steps on discord.

[flipsidejack]

Hi, just dropping in here to spread some awareness - I just found this developer request in Flow's year-end grant summary and list of requests. It's mentioned that there are no tools like Dune that support Flow yet, but we actually do at Flipside. We have the complete chain indexed back to mainnet launch, with curated tables for easy blockchain querying and analysis.

You can see the tables by trying out our query editor at: flipside.new

Here are some recent dashboards to give you a gist of the analysis possible:

• Flow NFT explorer: https://app.flipsidecrypto.com/dashboard/6ty6DJ
• NFL All Day Deep Dive: https://app.flipsidecrypto.com/dashboard/nfl-all-day-round-3-digging-deeper-tuZIEV

Regarding the milestone steps, Flipside has completed every single one. This is not a request for grant, just wanted to make you aware that we've been working on this

[bluesign]

@flipsidejack I just tried your app yesterday, I can honestly say, I am beyond amazed.