qiskit-ibm-runtime icon indicating copy to clipboard operation
qiskit-ibm-runtime copied to clipboard

Stop caching Qobj in the job

Open jyu00 opened this issue 3 years ago • 5 comments

What is the expected enhancement?

Today we cache Qobj in the job after submission, so that it's readily available if the user wants to access it. The problem is, as the size of the machines increases, so does the size of the Qobjs. Caching Qobj in the job means Python cannot garbage collect it. This gets much worse if one submits many jobs in a single Python session, like many algorithms do.

Instead of keep a reference to the Qobj in the job, we can just retrieve it on demand. It'd be slower for users who references the Qobj multiple times, but they can easily keep a local reference rather than doing job.circuits() each time.

jyu00 avatar Oct 29 '21 15:10 jyu00

image

zlatko-minev avatar Nov 01 '21 13:11 zlatko-minev

image

zlatko-minev avatar Nov 01 '21 13:11 zlatko-minev

Qobj has not been cached for a while, closing

kt474 avatar Nov 30 '23 20:11 kt474

The same thing applies to inputs (circuits) for primitives though. There should be an issue for that if you don't want to repurpose this one. cc @drew-distefano

jyu00 avatar Dec 01 '23 16:12 jyu00

I'm running into memory issues again it seems.

When I use the qiskit_ibm_provider.job.ibm_circuit_job.IBMCircuitJob, it called self._get_params(), which retrieves a large objector utility-sized circuits. This creates a large object in the _params['circuits']. I can't seem to get rid of it anywhere, maybe it is cached elsewhere? image

zlatko-minev avatar Dec 12 '23 00:12 zlatko-minev

We don't have Qobj anymore, but it looks like we still cache inputs

jyu00 avatar Jun 21 '24 20:06 jyu00