kepler.gl dynamically add to dataset

I'm trying to keep the data I display in kepler updated with my database in real time. The only way I've been able to do this so far is by deleting the current layers with the associated dataID, adding the data with the new row to the map with addDataToMap, and then restyling the layers.

This works with smaller datasets, but seems very inefficient with larger ones. It would be nice if it was possible to dynamically add (or delete) a row from a dataset currently displayed on the map- and see those changes reflected in the layers. I'm imagining something like addToDataset(dataId, newRow).

If there is a way to currently do this that I'm overlooking or other suggestions I'd love to hear them. Thanks!

Jul 25 '18 14:07 aakim26

thank you for using Kepler.gl.

Currently we don't have a straight way to update an existing dataset.

In the meanwhile, you can change your flow by skipping removeDataSet and call addDataToMap with your new data. Make sure the new dataset id that you are passing as addDataToMap call matches the existing one; by doing so you are going to override existing data with your new data.

Jul 27 '18 21:07 macrigiuseppe

Hey, thanks for the response and suggestion!

I've tried not removing the layers associated with the dataset and just calling addDataToMap- while it does update the existing layers it also creates new ones (for example, when working with icon layers I end up with a point layer and an icon layer in addition to my original icon layer). Is there a way to set the configuration so that this doesn't happen (when I call addDataToMap I generally have config: {})?

Jul 27 '18 21:07 aakim26

@aakim26 you need to pass the same configuration that is currently in your kepler state as part of your addDataToMap call.

You can use KeplerGlSchema to export the configuration before you call addDataToMap

Aug 03 '18 17:08 macrigiuseppe

@aakim26 you can find a tutorial on how to dynamically update a dataset with new data here: http://vis.academy/#/kepler.gl/3-load-config

please see section "2. Replace an existing dataset with new data with the same format , keeping the current config."

Aug 20 '18 01:08 macrigiuseppe

@macrigiuseppe thanks for the link! I'm going to leave the issue open because I think the ability to modify datasets without completely replacing them would still be very useful. Especially using large datasets it could really increase the efficiency.

Aug 20 '18 15:08 aakim26

any idea how to get the data from a remote URL instead of a file ?

Sep 11 '18 21:09 prune998

@prune998 You could fetch the csv and config, and then just do something like: dataset = csv file config = json map configuration file understood by kepler

loadSampleConfigurations(dataSet,dataConfig){ var dataId = dataConfig.config.visState.layers[0].config.dataId; const data = Processors.processCsvData(dataSet); const dataset = { data, info: { id: dataId } }; this.props.dispatch(addDataToMap({datasets: dataset, config: dataConfig})); };

Nov 22 '18 13:11 varunpant

@prune998 We just added a new feature to load maps/datasets from remote URL. Please feel free to check out the pull request here with instructions on how to load maps/datasets from url: https://github.com/uber/kepler.gl/pull/260.

P.S. please do not use the same issue for multiple topics

Nov 22 '18 21:11 macrigiuseppe

@aakim26 I have embed kepler.gl locally and want to fetch data directly from database which is not stored locally. I am new to react.js so having difficulty to resolve it. So can you tell me how can i fetch it from database directly.

May 30 '19 12:05 vararp25

Is this going to re-render the tiles each time from the beginning or progressively add new data? My scenario is such that a 200mb dataset might be small. I'd like to be able to stream in the dataset in chunks and add them to the map as they become available, but not force a full re-render of the tiles each time I add a new chunk.

Jul 15 '19 19:07 derek-nascent

This is a bigger effort we will tackle later, to support data streaming in chucks, and not update the entire dataset

Jul 19 '19 18:07 heshan0131

Hey, thanks for the response and suggestion!

I've tried not removing the layers associated with the dataset and just calling addDataToMap- while it does update the existing layers it also creates new ones (for example, when working with icon layers I end up with a point layer and an icon layer in addition to my original icon layer). Is there a way to set the configuration so that this doesn't happen (when I call addDataToMap I generally have config: {})?

Hi,

I've saved the Kepler map that I created and customized using Python as an HTML file (populated with some default data). I needed to update data in the map on an event and thus I'm updating data in __KeplerDataConfig using an API on the occurrence of that event. Could you please help me with calling addDataToMap function to refresh the map component so that it reads the updated __KeplerDataConfig (with updated data) and thus the updates the data reflected on the map.

Thanks!