firegento
Error on importing 100k products
Hi
My solution involves having products data coming from a queue system. In that I sent 100k products gathered in packages of 100/500 depending on configurations. Having that said after importing I would say something like 4000 products i start having - "General system exception happened". The strange part is - I catched those 100 products and rerun the process and booooom - everything works just fine. - So i have no idea on how to reproduce this problem.
Specs: Localy no indexers or crons are in place.
What I've tried:
ran those 100 products 7 times - 700 products - no errors ran 100 products with repeated products - no errors.
Hints:
I've turned general log and realised that magento rollsback sometimes the save of categories - urls or products may lead to this. Magento 2 has some triggers which could be messing with this process.
Can somebody help me? TY
Hello jtcosta,
I assume that this error occurs not on the Magento-side, but on the database. "General Error" is mostly just passed from the database to Magento. Your database could be not able to handle that amount of data in a small period of time. If you are unable to scale on the Database Size, you should try to make some "sleep" periods before importing a new bunch of data.
Greetings Elias
Hello Elias,
After some digging I got to this function ->
public function processImport($dataArray) { if($this->_validateData($dataArray)) { $this->_importData(); } }
Which seems that if by any reason the validation of rows fails the Importer will exit and not import that bunch. ValidateData will return $result = !$errorAggregator->getErrorsCount();
So what I'm trying to do is basically get the rows which are invalid and trying to skip/remove them. My question is -> Isn't the module already doing this? If so where is that logic?
