simple-salesforce
simple-salesforce copied to clipboard
simple-salesforce
Bulk not working well with large data
results = sf.bulk.instance.query(query,lazy_operation=True)
all_results = []
for result in results:
all_results.extend(result)
The above is not working, I have 19000 records but it is only returning 230 when I check:
print(len(all_results) )
Or when I loop through all_results
@yestouscool - Something seems off as I am not receiving the same issue when I perform on my end. Can you provide python version and simple-salesforce version?
In addition, instance should be the name of the custom object (ex. "Account" or "Custom_Object__c")
@jon-wobken 3.10.11 (main, Apr 5 2023, 14:15:10) [GCC 9.4.0] That's the version of the python I am using.
Yes actually I replaced instance__c but still it didn't work.
@drbjim would query_all still use Bulk API under the hood not to hit the limit of salesforce?
@jon-wobken 3.10.11 (main, Apr 5 2023, 14:15:10) [GCC 9.4.0] That's the version of the python I am using.
Yes actually I replaced instance__c but still it didn't work.
What version of simple Salesforce? A bit odd. Can you try doing a second loop where you extend the result to all_results?
@drbjim would query_all still use Bulk API under the hood not to hit the limit of salesforce?
It does not. It uses rest API but you are unlikely to hit the limits in my experience using this vs bulk api which has a limit of 15k batches per 24 hours.
It finaly worked, magically I would say but the problem is for a large data object 6GB, the program does not move from the line:
results = sf.bulk.instance.query(query,lazy_operation=True)