DoctrineWriter: fix setting associations

[buddhaCode]

trafficstars

The associations set in loadAssociationObjectsToEntity() method will be overwritten by the original value/foreign key. This will lead to an Doctrine exception, coz Doctrine is expecting an entity object and not a scalar value.

[Baachi]

Can you provide a failing test case?

[buddhaCode]

Sorry, what do you mean?

[Baachi]

Can you provide a short snippet which shows the error, so i can reproduce your issue.

[buddhaCode]

Sure.

I have a target model Store with an association to a model Industry

CSV source:

Industry id: 1 name: industry1

Store id: 1 name: store1 industry: 1

Here is the Error:

Found entity of type on association Shopware\CustomModels\StoreLocator\Store#industry, but expecting Shopware\CustomModels\StoreLocator\Industry in vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php on line 783 Stack trace: #0 vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(679): Doctrine\ORM\UnitOfWork->computeAssociationChanges(Array, 1) #1 vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(740): Doctrine\ORM\UnitOfWork->computeChangeSet(Object(Doctrine\ORM\Mapping\ClassMetadata), Object(Shopware\CustomModels\StoreLocator\Store)) #2 vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(297): Doctrine\ORM\UnitOfWork->computeChangeSets() #3 vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(389): Doctrine\ORM\UnitOfWork->commit(NULL) #4 Shopware/Plugins/Local/Frontend/NetcomTypo3StoreLocator/vendor/ddeboer/data-import/src/Ddeboer/DataImport/Writer/DoctrineWriter.php(179): Doctrine\ORM\EntityManager->flush() #5 Shopware/Plugins/Local/Frontend/NetcomTypo3StoreLocator/vendor/ddeboer/data-import/src/Ddeboer/DataImport/Workflow.php(304): Ddeboer\DataImport\Writer\DoctrineWriter->finish() #6 Shopware/Plugins/Local/Frontend/NetcomTypo3StoreLocator/Controllers/Frontend/StoreImport.php(312): Ddeboer\DataImport\Workflow->process() #7 Shopware/Plugins/Local/Frontend/NetcomTypo3StoreLocator/Controllers/Frontend/StoreImport.php(52): Shopware_Controllers_Frontend_StoreImport->importStores('m...') #8 Enlight/Controller/Action.php(159): Shopware_Controllers_Frontend_StoreImport->importAction() #9 Enlight/Controller/Dispatcher/Default.php(528): Enlight_Controller_Action->dispatch('importAction') #10 Enlight/Controller/Front.php(228): Enlight_Controller_Dispatcher_Default->dispatch(Object(Enlight_Controller_Request_RequestHttp), Object(Enlight_Controller_Response_ResponseHttp)) #11 Shopware/Kernel.php(141): Enlight_Controller_Front->dispatch() #12 vendor/symfony/http-kernel/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(472): Shopware\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #13 Shopware/Components/HttpCache/AppCache.php(256): Symfony\Component\HttpKernel\HttpCache\HttpCache->forward(Object(Symfony\Component\HttpFoundation\Request), true, NULL) #14 vendor/symfony/http-kernel/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(429): Shopware\Components\HttpCache\AppCache->forward(Object(Symfony\Component\HttpFoundation\Request), true) #15 vendor/symfony/http-kernel/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(329): Symfony\Component\HttpKernel\HttpCache\HttpCache->fetch(Object(Symfony\Component\HttpFoundation\Request), true) #16 Shopware/Components/HttpCache/AppCache.php(178): Symfony\Component\HttpKernel\HttpCache\HttpCache->lookup(Object(Symfony\Component\HttpFoundation\Request), true) #17 vendor/symfony/http-kernel/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(193): Shopware\Components\HttpCache\AppCache->lookup(Object(Symfony\Component\HttpFoundation\Request), true) #18 Shopware/Components/HttpCache/AppCache.php(113): Symfony\Component\HttpKernel\HttpCache\HttpCache->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #19 shopware.php(109): Shopware\Components\HttpCache\AppCache->handle(Object(Symfony\Component\HttpFoundation\Request)) #20 {main}

And here is the xDebug output.

$value should not be "1". It should be an object of type Industry. Actually, all associations were already set in the loadAssociationObjectsToEntity() method, which was called directly before updateEntity() method. So the associations does not have to be set again.

[webdevilopers]

:+1: Fixes https://github.com/ddeboer/data-import/issues/188.

[Baachi]

This might fix your issue but as travis reported it completely break the current workflow. I can't really produce your issue. @webdevilopers Maybe you can submit a PR with an testcase which demostrate the bug.

[webdevilopers]

As mentioned in https://github.com/ddeboer/data-import/issues/188#issuecomment-93343799 I was wondering why the fix worked. My issue looked more like a reading - not writing - issue in https://github.com/ddeboer/data-import/issues/188.

I will try the same import with a new xlsx file from scratch to prevent any formatting that is done by the tool that exported it, @Baachi .

[webdevilopers]

I played around with the xlsx file. I removed some lines and tried again. After some attempts it looks like my script will break every 20 items. Is there a number set for this batch action?

Here are the first rows of my xlsx file:

The strange thing is the debug output. While this output relates to the first element with ID 3152

 at ORMInvalidArgumentException ::invalidAssociation (object(ClassMetadata), array(
'fieldName' => 'creditAccount', 'joinColumns' => array(array('name' => 'credit_account_id', 'unique' => false, 'nullable' => true, 'onDelete' => null, 'columnDefinition' => null, 'referencedColumnName' => 'id')), 'cascade' => array(), 'inversedBy' => 'creditAccountJournals',
'targetEntity' => 'AppBundle\Entity\AccountsCode', 'fetch' => '2', 'type' => '2', 'mappedBy' => null, 'isOwningSide' => true, 'sourceEntity' => 'AppBundle\Entity\Journal', 'isCascadeRemove' => false, 'isCascadePersist' => false, 'isCascadeRefresh' => false, 'isCascadeMerge' => false, 'isCascadeDetach' => false, 'sourceToTargetKeyColumns' => array('credit_account_id' => 'id'),
'joinColumnFieldNames' => array('credit_account_id' => 'credit_account_id'), 'targetToSourceKeyColumns' => array('id' => 'credit_account_id'), 'orphanRemoval' => false),
'3152')

in vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php at line 844  +
at UnitOfWork ->computeAssociationChanges (array(
'fieldName' => 'creditAccount', 'joinColumns' => array(array('name' => 'credit_account_id', 'unique' => false, 'nullable' => true, 'onDelete' => null, 'columnDefinition' => null, 'referencedColumnName' => 'id')), 'cascade' => array(), 'inversedBy' => 'creditAccountJournals',
'targetEntity' => 'AppBundle\Entity\AccountsCode', 'fetch' => '2', 'type' => '2', 'mappedBy' => null, 'isOwningSide' => true, 'sourceEntity' => AppBundle\Entity\Journal', 'isCascadeRemove' => false, 'isCascadePersist' => false, 'isCascadeRefresh' => false, 'isCascadeMerge' => false, 'isCascadeDetach' => false, 'sourceToTargetKeyColumns' => array('credit_account_id' => 'id'),
'joinColumnFieldNames' => array('credit_account_id' => 'credit_account_id'), 'targetToSourceKeyColumns' => array('id' => 'credit_account_id'), 'orphanRemoval' => false),
'3152')
in vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php at line 741

some outputs later relates to the last item with IDs 9008 and 10130

at DoctrineWriter ->writeItem (array('' => null,
'creditAccount' => '9008', 'debitAccount' => '10130'))
in vendor/ddeboer/data-import/src/Ddeboer/DataImport/Workflow.php at line 261

All the IDs are quoted so I guess they are interpreted as strings.

My Mapping:

        $converter = new MappingItemConverter();
        $converter
            ->addMapping('Belegdat.', 'documentDate')
            ->addMapping('Sollkto', 'creditAccount')
            ->addMapping('Habenkto', 'debitAccount')
        ;
$workflow->addItemConverter($converter);

And my entity mapping:

class Journal
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    private $id;

    /**
     * @ORM\Column(type="date", name="document_date")
     */
    private $documentDate;

    /**
     * @ORM\ManyToOne(targetEntity="AppBundle\Entity\AccountsCode", inversedBy="creditAccountJournals")
     * @ORM\JoinColumn(name="credit_account_id", referencedColumnName="id", nullable=true)
     */
    private $creditAccount;

    /**
     * @ORM\ManyToOne(targetEntity="AppBundle\Entity\AccountsCode", inversedBy="debitAccountJournals")
     * @ORM\JoinColumn(name="debit_account_id", referencedColumnName="id", nullable=true)
     */
    private $debitAccount;
}

Pretty basic.

[Baachi]

Ah okay i think your issue is related to the doctrine entity manager clear behaviour. The DoctrineWriter have a batch processor builtin. If the batch of 20 items are reached, the DoctrineWriter will flush the entity manager and remove all objects in the entity manager.

This should solve your problem:

use Ddeboer\DataImport\Writer\DoctrineWriter as BaseDoctrineWriter;

class DoctrineWriter extends BaseDoctrineWriter {
     public function flushAndClear()
     {
              $this->entityManager->flush();
     }
}

[webdevilopers]

I'm currently not working on that project but this looks like the main issue, thanks @Baachi . Is this behaviour documented anywhere? Is it a real use case regarding my use case? Should I open a new one or will the BaseDoctrineWriter be updated with a different fix for this original issue by @buddhaCode ?

[Baachi]

A fix is difficult because it depends on your needs. We are working on the documentation.