memory usage improvements?

[Illumina-Todd]

I was running into troubles with memory when attempting a migration very similar to the pricebookentry migration in the samples. To make sure it wasn't my error, I tried running MigrateProducts.java from the samples and got the same result

Below is the tail of what was in the console (for MigrateProducts) when it stopped running. I let it run for 3 hours or so. I also had turned on the traceLog file which had grown to about 2GB at that point. It looks like it queries all of my products (about 10k of them). And then tries again. I assume one query is for source and the other is for target though the logging isn't very clear about that. When I looked at the trace log the last entry is a retrieve more so it looks like at some point the amount of data retrieved is so much that it really crawls.

I'd guess the only solutions would be either some kind of database or a poor man's database using flat files and lots of streams.

BTW, for my own effort (before I switched to the sample) I was doing something very similar. I was migrating pricing data but with custom tables from a managed package. I only wanted to move pricing for a few products. But since the table equivalent to pricebookitem has a lookup to product2, the tool still wanted to download the whole product table (even though I really only needed 3 rows.

I was successful migrating a few hundred rows from a smaller table.

2020-12-02 18:55:16 DEBUG SfdcApiServiceImpl:214 - Lookup: Running SF query for [Product2] 2020-12-02 18:55:16 DEBUG SfdcApiServiceImpl:950 - sforceObject.query=SELECT [...] FROM Product2 2020-12-02 18:55:16 DEBUG SfdcApiServiceImpl:881 - query:SELECT [...] FROM Product2 2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:899 - Logged-in user can see 10664 Product2 records. 2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:269 - SF query for type [Product2] took [25903]ms 2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:214 - Lookup: Running SF query for [Product2] 2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:950 - sforceObject.query=SELECT T[...] FROM Product2 2020-12-02 18:55:42 DEBUG SfdcApiServiceImpl:881 - query:SELECT [...] FROM Product2

[anoop-singh1]

Thanks for the feedback.

Did you try the filter option in the json?