h5spark

h5spark copied to clipboard

The code does not compile in Cori@Nersc

1

I followed the instructions: > Scala version: > > `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:path_to_h5spark/lib` > `module load sbt` > `sbt assembly` > `sbatch spark-scala.sh` The step `sbt assembly` incurred the following errors: >...

wangshusen

Using H5Spark in Jupyter

5

A user requests to use H5Spark in Jupyter, which is a popular notebook used anywhere. The current issue is that the class path is not recognized in the notebook, i.e.,...

valiantljk

Currently, H5Spark parallelize the IO along the slowest dimension, i.e., the dimension that changes slowest on disks. For example, for a 2D C array x[10][200], the h5spark will choose the...

valiantljk

Load Balancer in h5spark

2

When loading multiple files, the file size can have a long-tailed distribution(see the figure), or an even distribution. In case of even distribution, we don't need to balance the load....

valiantljk

Needs to maintain the coordinates during the read, then figure out the global offset during parallel write.

valiantljk