ELM-Python-Client icon indicating copy to clipboard operation
ELM-Python-Client copied to clipboard

oslcquery speed up large --percontribution querying by parallelising query with processing of results from previous query

Open barny opened this issue 1 year ago • 0 comments

One example doing counting of artifacts/module in a very large GC project with 1253 contributions containing 25886 modules and total of 6722401 bindings was spending about 50% of the time processing results, sequentially with the queries; if the processing can be done in parallel with the next query that's a significant speedup. Probably doesn't need anything more fancy/complicated than processing in parallel with the next query, but I suppose some more speedup might be possible with a controlled number of queries in parallel, at the cost of server load.

-q rdm_types:ArtifactFormat=jazz_rm:Module -s oslc_rm:uses,dcterms:title,dcterms:identifier

barny avatar Sep 28 '22 08:09 barny