glint
glint copied to clipboard
rjagerman
Glint: High performance scala parameter server
Hello Rolf! I have had a look about your code, what troubles me is how glint is interface with spark,i even don't see a line code related to spark. Best...
`glint.exceptions.PullFailedException: Failed 10 out of 10 attempts to push data at glint.models.client.async.PullFSM.glint$models$client$async$PullFSM$$execute(PullFSM.scala:67) at glint.models.client.async.PullFSM$$anonfun$request$1.applyOrElse(PullFSM.scala:79) at glint.models.client.async.PullFSM$$anonfun$request$1.applyOrElse(PullFSM.scala:76) at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:136) at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:134) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)...
`BigMatrix` supports pulling a set of row indices, i.e. `matrix.pull(rows: Array[Long])`. For push, only `matrix.push(rows: Array[Long], cols: Array[Int], values: Array[V])` is supported. For common use cases of pull/push a set...
Artery will be the new Akka remoting, based in Aeron IPC. It will be interesting to see the performance implications for communication. https://github.com/akka/akka/pull/21587
Would there be interest in some random initialisation mechanism for vectors / matrices? Often it is ok to init with 0s (the implicit case with new Glint data structures). But...
I've been unexpectedly busy these last summer months, so I unfortunately did not find much time to work on Glint. There has been some interesting discussion from users in issues...
Hi @rjagerman, this project looks very interesting and I'd like to explore it a bit more. You mention Spark integration as a goal, has there been work done on that?...
I have millions of discrete 64-byte features, remapping features to continuous ids is expensive. Can glint support that?
In order to facilitate an easier setup for localhost testing it would be nice to spawn a glint subsystem from within an application that uses the framework. E.g.: ``` val...
By default the parameter server aggregates new values by addition. This is ideal for some algorithms, but not for others. I've made some changes to make aggregation more flexible in...