Multiprocessing fails

[mabdelrazek]

Smoothing failed with beginCluster()

[mabdelrazek]

Hello, One nice improvement would be to renable parallel processing. Runing the example smoothing produce the following error:

Error in val[, a] : object of type 'closure' is not subsettable

Thanks, Mohammad

[fdetsch]

Confirmed. Parallel processing has been disabled on the 'release' branch as of 0e03a53ccb826cb0e7ad83476c2124edf7aae705, but its re-implementation is definitely planned for the medium term.

[mabdelrazek]

Thanks @fdetsch looking forward to having it back!

[mabdelrazek]

@fdetsch any updates about the parallel processing?

[fdetsch]

Yeah, @MatMatt wanted to have a look into this within the next few weeks.

[mabdelrazek]

Thanks @fdetsch and @MatMatt , looking forward to that :-D

[kimofos]

hi, is this function already re-implemented?

[Muesgen]

Hi, I came across this issue, is there an update regarding parallel processing? That would be very useful!

[Muesgen]

Hi again,

in my opinion this function is very useful and would like to use it in parallel because my rasters are quite large. I have already tried it with fork and sock clustering, but it did not work so far.

Actually I try mclapply, but that does not work either. See here:

My Idea was to process the values on multiple cores and send the processed blocks everytime to the same core, that only one core opens one file.

Do you have an idea here? or an other idea in general? So that I could try it. @fdetsch @MatMatt

[fdetsch]

Sorry for being so unresponsive. Parallel processing would definitely be nice to have, but I am currently lacking capacities. @MatMatt how about you?

[MatMatt]

Ups, I see I have been mentioned several times here, sorry I did not realise that. What about substituting the raster package with the terra package, at least for some parts of MODIS? terra would make in many cases the parallel processing obsolete... https://cran.r-project.org/web/packages/terra/index.html

[fdetsch]

I have hardly worked with terra so far, do you have any experience yet? If we decided to go down that road, it would in my opinion make sense to replace all related code (whittaker.raster() etc.) to avoid explicit imports of raster and sp altogether. However, I believe this would be a longer-term project..

[MatMatt]

Yes I have played with terra recently and it is very similar to raster (obviously), so if, than the shift should not be too difficult. I have just looked into whittaker.raster(), and is is soaked full of raster::* conjunctions...I first need to understand again what is going on in there... I come back to you later today.

[MatMatt]

Ah one more thing, should I look into the 'develop', correct?

[fdetsch]

Since this will likely involve massive changes, I'd rather create a new branch 'terra' from 'develop' to keep things as clean as possible.

[MatMatt]

ok, On the mid/long term the shift will be inevitable. But I want to avoid to trigger a too big thing an leaving you alone with the dealing of it. Yes that would be good, if you do create the terra-branch, I can play without endangering anything, and if it becomes to big, I will look at solving it within raster. Thx

[fdetsch]

Done.