Parallelisation with Dask does not work

[thesomamyth]

Hi,

The readme for jMetalPy mentions that it can be parallelised with PySpark or Dask, but doesn't mention how to do so. Is there documentation anywhere for how to configure that?

Thanks

[thesomamyth]

Hi,

I can't find any documentation on how to successfully parallelise jmetalpy. The closest thing I could find was this example here: https://jmetal.github.io/jMetalPy/api/algorithm/multiobjective/eas/nsgaii_distributed.html

But even running that exact code produces a stream of errors and doesn't actually work. I've attached a log of what happens. This is me running the program, letting it sit for a while with no seeming activity, then hitting Ctrl+C to kill it.

paralleltest.log

[benhid]

Hello Dominik,

To be clear, which version of Dask are you using? In the examples folder you can find some examples about how to use PySpark and Dask to evaluate solutions in parallel, as well as a distributed version of the NSGA-II algorithm.

[thesomamyth]

Hi,

Thanks for the response! I have no idea how I got to that other version of the distributed algorithm, but using the links you provided produces a working example using ZDT1. Thanks for that!

I am, however, running a custom FloatProblem, and using it with the distributed NSGA-II algorithm gives me the following error:

4 cores available
Traceback (most recent call last):
  File "main.py", line 69, in <module>
    main()
  File "main.py", line 57, in main
    algorithm.run()
  File "/home/dominik/.local/share/virtualenvs/multigroup-8rF_qus2/lib/python3.8/site-packages/jmetal/algorithm/multiobjective/nsgaii.py", line 267, in run
    auxiliar_population = r.replace(auxiliar_population, offspring_population)
  File "/home/dominik/.local/share/virtualenvs/multigroup-8rF_qus2/lib/python3.8/site-packages/jmetal/util/replacement.py", line 26, in replace
    self.ranking.compute_ranking(join_population)
  File "/home/dominik/.local/share/virtualenvs/multigroup-8rF_qus2/lib/python3.8/site-packages/jmetal/util/ranking.py", line 60, in compute_ranking
    dominance_test_result = self.comparator.compare(solutions[p], solutions[q])
  File "/home/dominik/.local/share/virtualenvs/multigroup-8rF_qus2/lib/python3.8/site-packages/jmetal/util/comparator.py", line 155, in compare
    raise Exception("The solution1 is None")
Exception: The solution1 is None

Any idea as to what this might be? My custom problem inherits from FloatProblem and is only using custom init and evaluate methods. Is there anything else I need to implement to get it working in a distributed/parallel way? It works fine single-threaded.

[benhid]

Hello,

I just tested the algorithm with a custom problem and it is working as expected. Can you send me an email (<antonio.benitez (at) lcc.uma.es>) with your problem implementation so I can test it on my machine? (if so, I will update this issue for future references once we find the root cause for raised exception).

Thank you!!

[ajnebro]

Hi. Does the error appear when using the sequential NSGA-II, or only when the distributed version is executed?

Antonio

[thesomamyth]

Hi,

With sequential NSGA-II it works fine.

I'm not sure I can send over my custom problem because it contains my company's IP. Would you be able to post the working problem that you used? Perhaps I can then work backwards to discover what the difference is and post a minimally reproducible example.

[erl987]

DistributedNSGAII works well for me with dask. However, I was fully unaware of this possibility and had previously hacked my own solution to get jmetalpy working with dask.

I think it would be really useful to clarify in the documentation how the library can be used with dask in a massive-parallel way.