slurmdb_jobs().get() does not return complete list of jobs

[0luhancheng0]

Details

• Slurm Version:19.05.4
• Python Version:3.6.2
• Cython Version:0.29.14
• PySlurm Branch:commit c50467cd84b8b2dfaed45298ae7d81043dae009d
• Linux Distribution:centos7

Issue

slurmdb_jobs().get() does not seem to return the complete list of jobs history records. The following script run both sacct -X and slurmdb_jobs and compare the number of records got.

import pyslurm
from subprocess import check_output


def call_pyslurm(username, **kwargs):
    jobs = pyslurm.slurmdb_jobs().get(**kwargs)
    #user_history = list(filter(lambda x : x['user'] == username, jobs.values())) 
    user_history = list(filter(lambda x : x['user'] == username, jobs.values())) 
    return  user_history

def call_sacct(args):
    sacct_output = check_output( 'sacct ' + args + ' -n -X', shell=True).decode().split(' \n')
    for i in range(len(sacct_output)):
        sacct_output[i] = sacct_output[i].split()
    return sacct_output
userlist = check_output('squeue -O username -h', shell=True).decode().split('\n')[:-1]

for i in range(len(userlist)):
    userlist[i] = userlist[i].rstrip()

userlist = list(set(userlist))
starttime = '2019-12-01T00:00:00'
print('get list of jobs start from {0}'.format(starttime))
for i in range(len(userlist)):
    sacct_output = call_sacct('-S '+starttime+ ' -u ' + userlist[i])
    pyslurm_output = call_pyslurm(userlist[i], starttime=starttime.encode('utf-8'))
    print("pyslurm return {0} records, sacct return {1} records for user {2}".format(len(pyslurm_output), len(sacct_output), userlist[i]))

[0luhancheng0]

any update on this?

[tazend]

This should now work correctly via the new Interface for querying database Jobs. Please check out here for more infos :)