devito icon indicating copy to clipboard operation
devito copied to clipboard

Devito `Data` operations: RHS's with data distributed across different ranks don't work the same way as serial

Open rhodrin opened this issue 4 years ago • 0 comments

Consider the following example distributed across, say, 4 ranks:

grid = Grid(shape=(8, 8, 8))
f = Function(name='f', grid=grid)
f.data[:] = 1
f.data[:, 5, :] = f.data[:, 1, :] + f.data[:, 7, :]

Currently this does not work since f.data[:, 1, :] and f.data[:, 7, :] have their data stored on different ranks and hence the addition on the RHS is 'not well defined' and fails. The above operation would currently have to be carried out via:

f.data[:, 5, :] = f.data[:, 1, :]
f.data[:, 5, :] += f.data[:, 7, :]

For convenience, we may want f.data[:, 5, :] = f.data[:, 1, :] + f.data[:, 7, :] to work directly for distributed arrays. Since f.data[:, 1, :] + f.data[:, 7, :] will require data to be transferred across ranks prior to __setitem__ being called we could do this through overriding __add__.

rhodrin avatar Nov 20 '19 15:11 rhodrin