velox-modelserver icon indicating copy to clipboard operation
velox-modelserver copied to clipboard

Published 20 hours ago • verified publisher icon amplab

Better Broadcasting Strategies

Open tomerk opened this issue 9 years ago • 1 comments

The current broadcasting strategy is to just write into etcd and then read it back from all nodes. Etcd isn't designed for this and this is probably bad for anything over a megabyte in size or so.

Strategies to introduce:

  • Spark filesystem (kinda like how the observations and user weights reading & writing currently work)
  • Torrent

This issue also involves decoupling the Model code from the exact type of broadcast being provided.

tomerk avatar Apr 14 '15 05:04 tomerk

As of Issue #51 Broadcast via the spark filesystem exists. In the future it would be nice to have broadcast via HTTP, and broadcast via Torrent

tomerk avatar Apr 21 '15 22:04 tomerk