OneToManyReader

[MichaelKubovic]

Hey there, I've spent some hours lately fighting with onetomany reader (nested!) and unfortunatelly I hit the brick wall when tried to fix it!

Firstly, my use-case involves two nested onetomany readers. The workflow looks like this.

$productsReader = new CsvReader($productsFile);
$productsReader->setColumnHeaders(array('reference','unit','shortDescription','name','description','tags','attributes','categories'));

$variantsReader = new CsvReader($variantsFile);
$variantsReader->setColumnHeaders(array('productReference', 'reference', 'name', 'minimalQuantity', 'boxQuantity'));

$pricesReader = new CsvReader($pricesFile);
$pricesReader->setColumnHeaders(array('productReference','variantReference','minQty','price','group'));

$variantPriceReader = new OneToManyReader($variantsReader, $pricesReader, 'prices', 'reference', 'variantReference');
$productVariantReader = new OneToManyReader($productsReader, $variantPriceReader, 'variants', 'reference', 'productReference');

$writer = new DoctrineWriter($entityManager, 'MadnessCoreBundle:Product', 'reference');
$writer->disableTruncate();

$workflow = new Workflow($productVariantReader);
$workflow->addItemConverter(new ProductConverter($entityManager));
$workflow->addWriter($writer);

There are several problems with onetomany reader:

1. by default, the last line of right reader is ommited
2. if sequence of "matched" csv lines is interrupted by one that doesn much, then the matching is over and lines that occur later in a file are not matched.

The root cause comes from OneToManyReader.php, method current(). https://github.com/ddeboer/data-import/blob/master/src/Ddeboer/DataImport/Reader/OneToManyReader.php#L75

First issue comes from the fact, that inside of the loop, iterator moves pointer to the position for next loop cycle. When iterator moves pointer to the last item (in the penultimate cycle), valid() returns false, indicating EOF and the valid() condition on while returns false (despite the fact that $rightRow holds valid data)

The second issue is due to $leftId == $rightId condition on the while loop. Once we get to line that doesn't match, loop is over and following lines are not matched. Solution in this case appeared to be moving the condition inside the cycle, allowing iterator to move on.

And now the scary part. I've spent some great number of hours fixing method and I couldn't make it work. This is my take, please have a look and give me some hint why my fixed method doesn't work.

public function current()
{
    $leftRow = $this->leftReader->current();

    if (array_key_exists($this->nestKey, $leftRow)) {
        throw new ReaderException(
            sprintf(
                'Left Row: "%s" Reader already contains a field named "%s". Please choose a different nest key field',
                $this->key(),
                $this->nestKey
            )
        );
    }
    $leftRow[$this->nestKey] = array();

    $leftId     = $this->getRowId($leftRow, $this->leftJoinField);

    while ($this->rightReader->valid()) {

        $rightRow = $this->rightReader->current();
        $rightId = $this->getRowId($rightRow, $this->rightJoinField);

        if ($leftId == $rightId) {
            $leftRow[$this->nestKey][] = $rightRow;
        }

        $this->rightReader->next();
    }

    var_dump($leftRow);

    return $leftRow;
}

The problem now is, that the upper onetomany in the hierarchy only iterates once, and then becomes invalid (eof == true). (it prints out that iterator's key is 0 and count is 4).

Testing CSV:

products.csv

"roll-stand","1ks","vždy k dispozícii na našom sklade;kvalitná zaklápacia lišta;vysúvacie nožičky pre lepšiu stabilitu;dodávané so štandardnou prenosnou taškou","Roll-Up Standard","Náš najpredávanejší bannerový systém, ktorý je cenovo ekonomický a zároveň veľmi kvalitný a stabilný.","paravan rollup; roletka; mosquito","Dodacia lehota|1 – 3 pracovné dni;Použitie|interiér;Príprava grafiky|PDF, 85x200cm, 100 DPI, 1:1, spadávka 6 cm zo spodnej strany;Rozmer grafiky|85/100/120/150 x 200 cm (š x v);Záruka|2 roky;Minimálny odber|1","7;33"

variants.csv

"roll-stand","roll-stand-85","85x200cm s tlačou","1","1"
"roll-stand","roll-stand-100","100x200cm s tlačou","1","1"
"roll-bullshit","roll-bullshit-10","bullshit","1","1"
"roll-stand","roll-stand-120","120x200cm s tlačou","1","1"
"roll-stand","roll-stand-150","150x200cm s tlačou","1","1"

prices.csv

"roll-stand","roll-stand-85","1","63","1"
"roll-stand","roll-stand-85","3","61","1"
"roll-stand","roll-stand-85","6","59","1"
"roll-stand","roll-stand-85","10","57","1"
"roll-stand","roll-stand-100","1","83","1"
"roll-stand","roll-stand-100","3","81","1"
"roll-stand","roll-stand-100","6","79","1"
"roll-stand","roll-stand-100","10","77","1"
"roll-stand","roll-stand-120","1","99","1"
"roll-stand","roll-stand-120","3","97","1"
"roll-stand","roll-stand-120","6","95","1"
"roll-stand","roll-stand-120","10","93","1"
"roll-stand","roll-stand-150","1","109","1"
"roll-stand","roll-stand-150","3","107","1"
"roll-stand","roll-stand-150","5","105","1"
"roll-stand","roll-stand-150","10","103","1"