Feature: Add PEXSI Support for Gamma Only LCAO Calculation

[Flying-dragon-boxing]

Reminder

• [x] Have you explained the changes of codes in core modules of ESolver, HSolver, ElecState, Hamilt, Operator or Psi? (ignore if not applicable)

What's changed?

• new diago method (ks_solver) pexsi added to hsolver (diago_pexsi.cpp/.h and folder module_pexsi in module_hsolver)
• when ks_solver set to pexsi, pexsi uses DM from its calculation to calculate charge density rho (PEXSI calculates DM instead of psi)
• new FindPEXSI.cmake module added to folder cmake
• inputs and outputs related to pexsi
• Makefile and CMakeLists.txt

Any changes of core modules? (ignore if not applicable)

• class DiagoPexsi inherited from hsolver::DiagH
• new function dmToRho in class ElecStateLCAO to calculate rho from density matrix given by pexsi solver

[mohanchen]

We need tests and examples, and documents for users.

[LiuXiaohui123321]

Hi! How do you design the PEXSI's parallel scheme? Is there a new communication domain created separately for it, or is it shared with other communication domains?

[Flying-dragon-boxing]

Hi! How do you design the PEXSI's parallel scheme? Is there a new communication domain created separately for it, or is it shared with other communication domains?

PEXSI uses the global communicator DIAG_WORLD. Before and after the PEXSI process, the matrix conversion (pexsi uses compressed column but abacus uses block coordinate descent to store matrices) splits DIAG_WORLD into groups. There are also parameters to configure the parallelization inside the PEXSI process (i.e. pexsi_nproc and pexsi_nproc_pole, i'll add docs for them).

[Flying-dragon-boxing]

Based on current code:

1. docs about new parameters and implementation should be added before approving.
2. UnitTests about new inputs and code in module_pexsi should be added before approving.
3. double* pexsi_DM is key data for other developers who want to use PEXSI. Please add more docs to explain in detail what * pexsi_DM is and how the data in * pexsi_DM is arranged. I do not think current dmToRho() works for NSPIN=2 correctly.
4. Some local changes in Makefile should not be pulled remote.
5. Some temp codes for test should be deleted.
6. I think some examples should be added into abacus-develop/tests/integrate for nspin=1|2|4 to prove it correctness and protect it in the future.

• nspin = 2 condition should have been fixed, the pexsi_DM now is a std::vector of double(std::complex<double> has not been implemented) pointer and stores the density matrix
• unit tests for the new inputs have been added in the new commit
• local changes in Makefile have been removed

[Flying-dragon-boxing]

Integrate tests are added and tested locally in my environment, but I don't think they will pass unless this pr gets merged and the docker image rebuilds, so they have not been added to this pr yet

[Flying-dragon-boxing]

Baese on current code (2024/2/23)

1. I think it is better to define pexsi_EDM in DensityMatrix instead of elecstate_lcao, and remove unnecessary dependencies in FORCE_gamma_edm.cpp.
2. I do not think dmToPsi() works for nspin=4 now.
3. explain why you make modifications in psiToRho.
4. UnitTests of module_pexsi is still missed.

• In abacus, nspin=4 (soc) feature currently use complex hamiltonians, which is not configured in our pexsi interfaces.
• Code structures still needs modifications, especially in the MD process.
• Yes, unit tests are needed.

[Flying-dragon-boxing]

Is there some supporting information that can help other developers get an overview of your overall approach to designing and implementing PEXSI functionality in ABACUS, for example, Word document or PPT?

Good question. I have some messy documents waiting for organizing.

[Flying-dragon-boxing]

An unit test has been added for DiagoPexsi. But it won't run on github, as the required libraries are not ready yet in the docker image and switch ENABLE_PEXSI is set to off.

[WHUweiqingzhou]

1. please add a test report from the developers' side.
2. please add an example in abacus-develop/example to show how to use PEXSI.

[Flying-dragon-boxing]

• pexsi_dm and pexsi_edm are dmk/edmk, which should be indicated
• deconstruct methods and memory management may be dealt with better

[Flying-dragon-boxing]

Test report (based on previous tests): https://xmywuqhxb0.feishu.cn/file/HH0wbFEUuozhYIxPmbGcJZYQnIJ?from=from_copylink

Please let me know if you need further information

[mohanchen]

We will soon accept this PR, and continue updating PEXSI-related codes.

[scott-5]

Test report (based on previous tests): https://xmywuqhxb0.feishu.cn/file/HH0wbFEUuozhYIxPmbGcJZYQnIJ?from=from_copylink

Please let me know if you need further information

@Flying-dragon-boxing hi, in the test report, PEXSI and GENELPA were compared. What I want to ask is whether PEXSI is also one of the ks_solvers? How do we call PEXSI ? Is the main purpose of PEXSI features performance or something else?

[kirk0830]

@scott-5 hi, you can find all related parameters here: https://abacus.deepmodeling.com/en/latest/advanced/input_files/input-main.html#pexsi, and yes, pexsi is a kind of ks_solver, examples in examples/pexsi folder may help.

[scott-5]

@kirk0830 Thanks. After reading the doc, would it be better to add the pexsi option here: https://abacus.deepmodeling.com/en/latest/advanced/input_files/input-main.html#ks-solver and statements about gamma only lcao calculation.