abnormal stopped of DCU jobs (Device & Memory)

[pxlxingliang]

Describe the bug

Some jobs on DCU are stopped abnormal.

1. Stopped before SCF beforescf.zip

The last line of screen output is:

 START CHARGE      : atomic
 DONE(12.4994    SEC) : INIT SCF
 ITER   ETOT(eV)       EDIFF(eV)      DRHO       TIME(s)

The last lines of running_scf.log

 -------------------------------------------
 SELF-CONSISTENT
 -------------------------------------------
                                 init_chg = atomic
 DONE : INIT SCF Time : 12.4993 (SEC)

2. Stoppend when calculating stress stress.zip

3. Stoppend at beginning start.zip

 Init Non-Local PseudoPotential table :
 Init Non-Local-Pseudopotential done.
 DONE : NON-LOCAL POTENTIAL Time : 10.011598924 (SEC)


 Make real space PAO into reciprocal space.
       max mesh points in Pseudopotential = 1001
     dq(describe PAO in reciprocal space) = 0.01
                                    max q = 1204

 number of pseudo atomic orbitals for Sr is 0

 number of pseudo atomic orbitals for Al is 2

 Warning_Memory_Consuming allocated:  PW_B_K::ig2ixyz 8.63247299194 MB

Expected behavior

No response

To Reproduce

No response

Environment

No response

Additional Context

No response

Task list for Issue attackers (only for developers)

• [ ] Verify the issue is not a duplicate.
• [ ] Describe the bug.
• [ ] Steps to reproduce.
• [ ] Expected behavior.
• [ ] Error message.
• [ ] Environment details.
• [ ] Additional context.
• [ ] Assign a priority level (low, medium, high, urgent).
• [ ] Assign the issue to a team member.
• [ ] Label the issue with relevant tags.
• [ ] Identify possible related issues.
• [ ] Create a unit test or automated test to reproduce the bug (if applicable).
• [ ] Fix the bug.
• [ ] Test the fix.
• [ ] Update documentation (if necessary).
• [ ] Close the issue and inform the reporter (if applicable).

[WHUweiqingzhou]

@denghuilu, could you have a look?

[denghuilu]

I have reviewed each STDOUTER.log file and found that the abnormal stops were caused by an Out of Memory error.

COMMAND: echo ks_solver cg >> INPUT; bash run.sh -o 1 -n 4 -d 1 -s 0WARNING: Total thread number on this node mismatches with hardware availability. This may cause poor performance.
Info: Local MPI proc number: 4,OpenMP thread number: 1,Total thread number: 4,Local thread limit: 32
 Unexpected Device Error /public/home/abacus/abacus-dcu/source/module_psi/kernels/rocm/memory_op.hip.cu:48: hipErrorOutOfMemory, out of memory
 Unexpected Device Error /public/home/abacus/abacus-dcu/source/module_psi/kernels/rocm/memory_op.hip.cu:48: hipErrorOutOfMemory, out of memory
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing

[WHUweiqingzhou]

@dyzheng we need to check the usage of memory in these cases.