Ability to process and render unknown data sources without modifying core files

[robhawkes]

Loading, processing and rendering of data sources is currently handled by classes baked into the core application. This is fine for world-wide sources of data that the majority will find useful (like OpenStreetMap), but it isn't a good approach for allowing people to use their own data that may only touch a small geographic area, or require unique processing.

For example, rendering the London Underground network is specific to an area of London and is also incredibly intricate and specific, so it should be handled outside of the core application. The same would apply for visualising air traffic over Heathrow Airport.

This could be achieved by prototyping the data classes from the outside, but that seems complicated. Perhaps there's a method of abstracting data collection, processing and output that can be defined outside of the core app and added on runtime. Or perhaps something even simpler.

Maybe something like this:

var city = new VIZI.City();
city.init({coords: [-0.01924, 51.50358]});

var data = new VIZI.Data("http://url.com/to/data.json");

// Data.process() is called automatically when data is loaded
data.process = function(data) {
    // Do something with the data
};

I think inspiration could be taken from the way Three.js allows you to write your own handlers for processing and rendering, meaning the library itself does nothing without your own setup. I think ViziCities currently does too much by default and should scale back a bit and have things set up on runtime outside of the library.

Also look into how D3 handles this as it's more like ViziCities in purpose and use, just in 2D.

Either way, ViziCities needs to be able to easily handle unknown and complex data at runtime and without too much hassle for the developer.

[robhawkes]

A method of doing this via prototyping has been partly enabled by 028ddb62bddd4dc6c58140f9b94d32f923a817db and the work done for the Heathrow Airport demo.

[DavidBruant]

Yes. Struggling with this one at the moment. I think this issue is actually several issues at once. Continuing the discussion here. Feel free to split this issue into several as you think is appropriate.

Decoupling loading and processing

Right now, it looks like the data loading part cannot be done separately. The VIZI.City.prototype.loadOverpass function will not only load the data from the Overpass API (and there is no way to choose another API), but also do the rendering. Part of the problem lies in src/client/data/Data.js which does way too much especially when it calls loadProcess which calls generateFeatures.

Proposal

• Have one function/module to fetch the data (with a URL as parameter) but nothing else (just returns a promise for the result)
• Have one function/module that processes the data to be in the intermediate format you want to vizualize
• Have one function/module to draw with Three.js based on the intermediate data

Then let the user combine these functions/modules as they wish. You provide one such combination in the form of VIZI.DataOverpass.prototype.update. For instance, if I want to load data from another source, I can load it, but it's hard to feed it to something without copy/pasting a good share of the current Data.js.

Data expressiveness

ViziCities needs to be able to easily handle unknown and complex data at runtime

Yes and no. ViziCities cannot predict the shape of data of unknown sources, because it would be too much work to support everything. In my case, I have data in .3ds format. It seems unrealistic that Vizicities would support that format (especially because the long/lat data is missing, so you'd need a specific API to account for that, etc.)

Proposal

However, what you could be doing is define, specify and document the data format you want to work with in Vizicities. Then, different tools could be created to

1. validate that the data is in the proper format (so people understand that they have a bug at the data level before looking any further)
2. convert existing formats (osm, .3ds, etc.) to the format you expect so that these formats are understood by Vizicities.

My understanding is that the current Vizicities data format is pretty much the OSM data format (nodes, ways, some ways to describe simple buildings). Is that enough for what you plan to use vizicities for? What more do you want to visualize?

This is tedious work, but it's sort of "infrastructure" work. Based on it, lots of things will come for free or cheap (like the decoupling explained above).

Tell me what you think.

[groovecoder]

I'm struggling with this too. I'd love to easily add a layer of GeoJSON to mash up, e.g., Tulsa Transit GTFS and SNAP/EBT data onto Tulsa.

Is there at least a doc somewhere that explains how to process and render new data sources in the core code?

[robhawkes]

@groovecoder: Have you tried the 0.2.0 branch yet? There have been huge improvements there around the methods of pulling in and using external data sources in known formats (eg. GeoJSON). You may also appreciate the (unfinished) documentation being written for 0.2.0, as well as the examples that show you some ways of using the new approach.

I'm actually working on an improvement to this new system (known as the Blueprint API) that takes it a step further and allows for near unlimited control over data sources and how data is pulled in and transformed before output. This is still at an early stage and likely won't be of any use for a while yet.

Regardless, I'd like to hear more about exactly what you'd like to visualise as it will help me ensure that the new Blueprint API can support it.

[groovecoder]

I hadn't seen 0.2.0 yet, but it looks good. I'll read up on it today and this week.

We want to visualize a few layers of data, and be able to toggle them on and off:

• Grocery stores (point data)
• SNAP/EBT withdraw locations (point data stores or ATMs)
• Tulsa Transit bus lines (line data)

The goal is to visualize "food deserts" in Tulsa.

[robhawkes]

Nice, well 0.2.0 should be able to partly handle that out of the box but you'll hit a couple of issues should you want useful UI on the POIs, etc. I've already factored in POIs with UI and GTFS / route data into the new Blueprint API so that'll help when it lands.

Let me know how you get on with 0.2.0.