Migrate to d3 visualizations

[djrtwo]

Issue

• matplotlib visualizations from the command line aren't super sweet
• d3 visualizations + a web interface for running & rerunning simulations are super sweet

@karlfloersch implemented this in March in PR 1, but it has serious merge conflicts.

Proposed implementation

Use PR 1 as a reference for implementing d3 + web interface in the current repo.

Hold off on implementation until blockchain branch is merged into master

[naterush]

I think @karlfloersch mentioned that the reason the original PR was not merged in was b/c of some performance concerns.

Not sure if this is an issue - but it might be worth thinking about other visualizations as well.

[seanavery]

A web based d3 visualizer sounds awesome! I have decent amount web & d3 experience, and would love to take a crack at rebasing this PR and extending out. Is anyone currently working on this and/or is it still in the roadmap? @zramsay I see you are working on graphing features right now.

@naterush As far as scalability of the visualization goes.. Matplotlib uses canvas under the hood, so it has to re-render the entire view on each change (not good for frequently changing graphs). SVG utilizes the underlying dom is better for dynamic data. However, there is a hard limit on the number of DOM nodes and any change requires a complete re-render.

This is where I think react can come in. Since virtual-dom allows subsets of the view tree to be updated independently (only re-renders what changes). Plus the new react-fiber implementation has given a huge performance boost to apps w/ large component tree. https://www.youtube.com/watch?v=Qu_6ItnlDQg

Uber has a nice d3-react library with higher level api's for tree maps :) https://github.com/uber/react-vis

[djrtwo]

Hi Sean! Thanks for popping in.

No one is working on this currently. I think it'd be excellent to port to d3. Could end up making the visualizations interactive :)

I'll be generally AFK for the next week and a half so be in touch with @naterush as you come up with a plan. In general, I'd like to keep any additional web component completely optional and require minimal modifications to the core of the repo. I'm not a front-end guru so I'll defer to your judgement on tooling.

Some notes on potential features:

• Run/visualize
  • configure simulation from web interface (pick params like the ones you can pass in to casper.py from command line)
  • run specific from a test-string
• export
  • as a test string
  • as a gif
• handle binary and blockchain protocol
  • Make tool as general as possible to handle both protocols. Will need some specific config depending on protocol. See our parent plot_tool vs the children binary_plot_tool and blockchain_plot_tool

[seanavery]

Hey Danny, thank you for the welcoming! huge fan of this project :)

Features above noted. Will dig more into how test_lang is working, would be cool to generate tests from the ui and export test_lang sequence from simulations.

Still getting feel for the repo and plot tool functionality, but basically, I am thinking that the plot tool could remain as is but emit JSON data instead of saving images & interacting with matplot if --web is present in command line call.

Overview

1. Plot Tool Modifications

• add another argument option --web that will initiate the web/d3 simulation process
• append a boolean self.web to the plot_tool class as a gate on msg emissions-- sending json blob to express server

2. Server

• a simple express server that serves the webapp payload on /
• a websocket that connects to webapp client and forwards emitted messages from the python shell

3. App v.1

• form to run simulations. specifying [rand, rrob, full, nofinal, binary], [#validators], [#rounds], [#interval]
• d3 visualization of simulation output. I am still testing out different libraries and apis for this and am not sure which will be best at this point. Perhaps using d3-force.. (https://bl.ocks.org/mbostock/4062045)
• the app could save an array of of states [ {interval1}, {interval2}, {interval3}..] allowing traversal back and forth between rounds

I started building up a webapp yesterday and am making python shell calls on form submissions. Here is what the form looks like.

Goal today is to start emitting json data from the simulation to the webapp via websocket pushes. Once I have data loaded up in the app, I can start to toy with the visualizations 👍

[seanavery]

Update

I have a minimum working hack for pushing json data from the python shell to the web app over a websocket. Simple redux state accepting the pushes and updating the visualization.

The current graph is pretty naive and only renders nodes and edges. I want to get more nuanced features like sizing according to weight, some sort of clique clustering, styling on the message passing & conical chain highlight.

Also, will work on features noted above such as test_lang integration and gif exports. Would be helpful to know what features are the biggest priority for you guys.

Fully connected network looks pretty cool as of now :)

[djrtwo]

So cool!!

Can you open up a PR with "[WIP]" (work in progress) in the title so we can take a look?

On Thursday, November 16, 2017, Sean Pollock [email protected] wrote:

Update

I have a minimum working hack for pushing json data from the python shell to the web app over a websocket. Simple redux state accepting the pushes and updating the visualization.

The current graph is pretty naive and only renders nodes and edges. I want to get more nuanced features like sizing according to weight, some sort of clique clustering, styling on the message passing & conical chain highlight.

Also, will work on features noted above such astest_lang integration and gif exports. Would be helpful to know what features are the biggest priority for you guys.

Fully connected network looks pretty cool as of now :)

[image: screen shot 2017-11-16 at 9 31 30 am] https://user-images.githubusercontent.com/12690131/32899795-8a578ba0-cab1-11e7-8eb6-226cf519cb2b.png

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ethereum/cbc-casper/issues/17#issuecomment-344965523, or mute the thread https://github.com/notifications/unsubscribe-auth/ABXf--plwFL1h1qwtRLBERFHCPRt1G_1ks5s3Fn6gaJpZM4Px7Yt .