Memory Layout of Ghost Cell API is undocumented.

[rainwoodman]

The Ghost Cell part of the API never explicitly confirmed the ghost cell data is appended to the end of the allocated storage space.

[mpip]

Thanks for the hint. I will fix it at the end of this week.

-------- Ursprüngliche Nachricht -------- Von: Yu Feng [email protected] Datum:
An: mpip/pfft [email protected] Betreff: [pfft] Memory Layout of Ghost Cell API is undocumented. (#12)

The Ghost Cell part of the API never explicitly confirmed the ghost cell data is appended to the end of the allocated storage space.

— Reply to this email directly or view it on GitHub.

[rainwoodman]

Thanks. Looks like a common pattern of ghost cells is

• 'Exchange' to create ghost cells on 'other ranks'
• 'other ranks' modify the ghost cells
• 'Reduce' collects ghost cells from 'other ranks' back to 'myself'.

The current documentation did not mention the second step at all. Did I understand the GC wrongly? If not, then besides documentation, it may be useful to include some API to ease the second step. Looks like it is non-trivial to access those extra slabs on the side with different levels of symmetry...

do you have a production code that uses ghost cells? What is the pattern over there?

[rainwoodman]

Also, if arbitrary strides are supported, we probably can come up with a 'different' way of doing GCs, by enforcing a dense representation of the entire volume (from below to above), instead of appending ghost cells after host. That can save a few 'if's when looking up the storage location of a cell from its index.

[mpip]

PFFT already uses a dense representation of the entire volume. This is what page 46 of the manual states. Maybe, I have to explain it more clearly there. The ghost cell module is used a lot within PNFFT and P2NFFT (a particle-mesh framework based on PFFT and PNFFT). The typical workflow of a particle mesh code is as follows:

• interpolate particle data to grid points locally on each process
• use a ghost cells reduce to merge the local interpolations on the corresponding MPI processes
• use an equispaced Poisson solver like FFT - convolution in Fourier space - inverse FFT
• duplicate the grid values at the borders via ghost cell exchange
• interpolate the grid values to particle positions

From the mathematical point of view you can write the ghost cell send as a matrix-vector product. Hereby, the entries of the ghost cell matrix are either 0 or 1. The matrix has more rows than columns. Then, the ghost cells reduce corresponds to an application of the transposed ghost cell matrix. This is explained on page 47 of the manual. But again, I can write it more clearly.

[rainwoodman]

You mean the section 6.2 (Ghost cell Communication) of the manual?

It would be nice to include the mathematical point view in the manual, together with a few formula -- it is hard to decipher what this means from just words..

[rainwoodman]

A follow up on this. Can I use pfft to help me managing the communication of edges of finite differentiation? It appears the ghost cells API is for this?