Query::all() with interator result

[kamarton]

What steps will reproduce the problem?

Yii 2.x (and also Yii 1.x) Query::all() result is very limited.

Main problems:

• loading all record it once
  • larger memory requirement
  • Many records require batch and similar solutions
• does not support scanning

If result change to Iterator class:

• ActiveRecord populate only in sequence, only at one per time
• store only raw attributes array
• support scan based database
• rewind disabling option: less memory, allowing scan by infinity
• include batch and each functionality
• ActiveRecord populate error is skippable
• optional meta data
  • eg. google cloud bigquery add more meta fields to result (example: processed bytes, next page token, cache life)

What is the expected result?

example: Console process read normally, and support infiniti of records.

$iterator = $query->all();
if($iterator->databaseSpecificMetadaData /* limited result */) {
 
}
foreach($iterator as $key => $model) {
  // loop:
  // 1. next
  // 2. ActiveRecord popuplate - if required
  // 3. use model
  // next loop
}

What do you get instead?

example Console process require batch reading

Additional info

example scanning from Google Cloud Bigtable.

Q	A
Version	1.x 2.x
PHP version	-
Operating system	-

[kamarton]

Suggest

class Query {
  public function all() {
    return new QueryDataProvider($this);
  }
  public function one() {
    $dp = new QueryDataProvider($this);
    return $dp->first();
  }

  public function count() {
    $dp = new QueryDataProvider($this);
    return $dp->count();
  }
}

class QueryDataProvider extends DataProvider implements \Iterator { // or best practive DataProvider is \Iterator (this may already be true.)
  protected $query;
  protected $rowCache;   // raw data cache (attributes only)
  function __counstruct($query) {
     $this->query = $query;
  }
  public function count() {
    return $this->query->(yii2)count();
  }
  public function rewind() {
    if($disabledrewind && $rowCache !== null) {
       throw new NotSupportedException('rewind disabled!');	// For a simpler example
    }
    ...
    $this->next();
  }

  public function valid() {
    return sizeof($this->rowCache);
  }

  public function next() {
    if($this->rowCache === null) {
      $this->rowCache = $this->query->(yii2)asArray()->(yii2)all();
    } else {
      array_shift($this->rowCache);   // rewind disabled, drop previous elements
    }
  }

  public function key() {
    reset($this->rowCache);
    return key($this->rowCache);
  }

  public function current() {
    $row = reset($this->rowCache);
    // ActiveRecord::populate($row) if ActiveQuery (populate on the fly each element)
    return $row;
  }

}

[Tigrov]

Try this

$dataReader = $query->createCommand()->query();
// or
$dataReader = $query->each();

foreach ($dataReader as $row) {
   ...
}

[Tigrov]

Also QueryDataReader can be found in yiisoft/data-db package

https://github.com/yiisoft/data-db/blob/master/src/AbstractQueryDataReader.php