A comparison of the field energy evolution between the `master` and the current `dev` show massive differences

[PrometheusPi]

A comparison of the field energy evolution between the master and the current dev show massive differences

Thus something changed the evolution of the KHI between our last 0.6.0 release and today's dev.

Originally posted by @PrometheusPi in https://github.com/ComputationalRadiationPhysics/picongpu/issues/4402#issuecomment-1362907768

[psychocoderHPC]

Grr this will be a nightmare to search the problem. We did a full rewrite of the field solver.

We need to check if this is produced by the field solver or current deposition.

[PrometheusPi]

We should do a git bisect next year to find the origin of the difference and I should test whether the dev or the master is correct.

[kimjohn1]

Screen shot of the Laser Wakefield electric charge density (scalar field) using the dev branch on 8 December

Screen shot of the Laser Wakefield electric charge density (scalar field) using the dev branch on 31 December and on 1 January

I don't have finer resolution on the particular day the change occurs since I didn't capture screenshots during full system testing between those dates.

I'm very glad to see that you posted this discovery. I had just started to troubleshoot whatever I must have messed up to produce such a completely garbled visualization.

Kim

[psychocoderHPC]

@kimjohn1 Thanks for the report. Do you use the default settings of the laser wakefield simulation? I am running the latest dev from today and can not see any issues when checking the png files.

./bin/picongpu -d 1 1 1 -s 2000 --periodic 1 0 1 -g 192 1024 192 --e_png.period 10 --e_png.axis yx --e_png.slicePoint 0.5 --e_png.folder pngElectronsYX

[psychocoderHPC]

I checked the KHI where I changed the ions to positrons (shape=TSC current deposition=Esirkepov) by setting the mass ratio to 1.0.

If I compare these to results from 2017 https://github.com/ComputationalRadiationPhysics/picongpu/pull/2275#issuecomment-331383352 I can not see an issue or the tests do not cover the issue:

./bin/picongpu -d 1 1 1 -g 128 512 12 -s 3000 --periodic 1 1 1 --e_png.period 10 --e_png.axis yx --e_png.slicePoint 0.5 --e_png.folder pngElectronsYX --fields_energy.period 10 --e_energy.period 10 --e_energy.filter all --i_energy.period 10 --i_energy.filter all --openPMD.period 250 --openPMD.file simData --openPMD.ext h5

charge conservation step 2500

src/tools/bin/plot_chargeConservation.py simOutput/openPMD/simData_002500.h5

charge conservation over time

src/tools/bin/plot_chargeConservation_overTime.py simOutput/openPMD/simData_%T.h5 --export charge_conservation.png

heating

src/tools/bin/plotNumericalHeating . --export heating.png

[psychocoderHPC]

OK looks like something is broken, using the test KHI (positrons/electron) and changing the drift to z-direction instead of x-direction is showing different images.

lds_energy.period 10 --e_energy.period 10 --e_energy.filter all --i_energy.period 10 --i_energy.filter all --openPMD.period 250 --openPMD.file simData --openPMD.ext h5

drift in z-direction plane YZ step 10

drift in x-direction plane YX step 10

The x drift looks correct, from my experience I know if there are clouds in the image after 10 steps something is wrong. The instability in advanced time steps with the z-drift is totally wrong :-(

[psychocoderHPC]

I was going back with my KHI-z direction test case until 18th Jan 2022 https://github.com/ComputationalRadiationPhysics/picongpu/tree/e48017d498ef9f1645b484ac1678fabc9bf36786 and have everywhere the some issue that changing the drift into z-direction produces wrong results.

[kimjohn1]

@psychocoderHPC I use the Laser Wakefield example simulation, but I use a custom config file that packages and sends output data using openPMD SST to the visualization application. It is pretty interesting to note that the .png images are normal looking. Do you know if the field energy evolution image that you included in the original post uses data supplied by openPMD? Perhaps there is an openPMD issue and not a solver issue at work here. Kim

[kimjohn1]

@psychocoderHPC I believe the openPMD SST implementation uses ADIOS2 to format the data. I should have included that in my last post. Kim

[psychocoderHPC]

@PrometheusPi and me performed some more checks with the dev branch.

default KHI pos/el drift in X step 600

same as default but drift in Z and supercell 4x8x8 cells

[psychocoderHPC]

I switched back to a branch where we still had EsirkepovNative and FieldInterpolatin (no coordinate shift for the field interpolation) native in our code.

git checkout ce7aa8ff91f22777dd56d901f9c01d3395359486

The results are equal to the latest dev results, so I assume that the coordinate shift in the field interpolation is not the root of the evil.

[psychocoderHPC]

@psychocoderHPC I believe the openPMD SST implementation uses ADIOS2 to format the data. I should have included that in my last post. Kim

Could you please open an issue with you image from https://github.com/ComputationalRadiationPhysics/picongpu/issues/4407#issuecomment-1368556570 It is possible that we break something in the openPMD output and therefore you see wrong densities. IMO your issue is independent of the numerical issue we observe and is an issue in the derived field implementation or an openPMD-plugin issue.

[kimjohn1]

@psychocoderHPC Will do. Kim

[psychocoderHPC]

@PrometheusPi and me performed some more checks with the dev branch.

default KHI pos/el drift in X step 600

same as default but drift in Z and supercell 4x8x8 cells

OK the issue is not exiting, the problem was that the example was not fully permutated. I missed to change in png.param that we need to plot b.x() green/blue and not b.z() what we plot in the default example. To fully permutate the example you need to do

• permutate x and z for the global volume size
• particle.param
  • set numParticlesPerDimension to 1,5,5
  • permutate DriftParamPositive_direction and DriftParamNegative_direction
• speciesDefinition.param change ions to poitron, change MassRatioIons to 1.0
• memory.param change supercell size to 4,8,8
• png.param change preChannel1 and preChannel2 to b.x()

KHI pos/el step 600 drift into Z-direction

[PrometheusPi]

Thus, only the initial issue remains and might be not significant since measuring the (global) growth showed the same results.

[psychocoderHPC]

offline discussed with @PrometheusPi two weeks ago. Maybe we can trigger the start of the KHI differently and not via numerical effects to become a more reliable setup.

[BrianMarre]

@PrometheusPi any new updates?

[PrometheusPi]

@BrianMarre not yet - the main (global) physics (growth rate of the KHI) was reproduced fine. Only the minor field components deviate (relative to themselves significantly, not significantly to the governing field dynamic)