etmc
Clover overhead in inverter ~50%
Okay, so I went ahead and did the long overdue measurement of the overhead of using the clover term in the inverter.
The result is that 50% of the time spent in the operator is spent in the hopping matrix, 25% in clover_inv and 25% in clover_gamma5.
As a comparison, without the clover term 50% of time is spent in H_eo_tm_inv_psi and 50% of time is spent tm_sub_H_eo_gamma5.
Keeping this in mind I'm a bit surprised that with the clover term the performance reported in the inverter, not taking into account the clover overhead, is about 50-60% lower than for the standard operator...
Unfortunately I can't really give a recommendation for improving the situation.
Actually, I think I know one sure-fire way of improving performance. If we prepare a copy of the sw field with the correct ordering (even-odd, I guess), this should help quite a bit. Other than that I'm not sure what to do.
Hi Bartek, can you please show me the directory where you did these tests? Maybe I can start from there and take more measurements, with the help of the tuning experts.
Hi Luigi, the scalasca output is in:
/work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/invert_test_clover /work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/invert_test_no_clover
The inversions of course have differing iteration numbers so only relative weights between the "hot-spots" are meaningful.
They should be world-readable, at least according to the unix permissions.
I get:
[hch02b@juqueen2 hch02]$ cd /work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/invert_test_clover /work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/invert_test_clover: Permission denied. [hch02b@juqueen2 hch02]$ cd /work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/ /work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/: Permission denied. [hch02b@juqueen2 hch02]$ more /work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/invert_test_clover /work/hch02/hch028/invert_test/invert_test_hybrid_hs_scalasca_512_64/invert_test_clover: Permission denied
oh, the grand-grand-parent directory had the wrong permissions, it should be fixed now