Add possibility to add metadata to each batch

[ragulka]

I'm implementing this a second time, and have run into the same thing twice now - I usually need to save some common data that is shared between all the items in a batch. For example, when generating a CSV file with all the users, I would need to make sure that all the generated lines are saved to the same file. This means I need to add the path to the output file to the batch data.

Right now, I would need to save that file path to every single item, which is quite pointless (same information repeated over & over again) and it can make the option field size quite large in case of thousands of items.

Ideally, I could do something like this:

$this->example_process = new WP_Example_Process();
foreach ( $user_ids as $user_id ) {
    $this->example_process->push_to_queue( $user_id );
}
$this->example_process->set_options( array( 'file_path' => '/myfile.csv' ) ); 
// and in the handler
function task( $item, $options ) {
  // $options will be array( 'file_path' => '/myfile.csv' )
}

But at the moment I need to do this:

$this->example_process = new WP_Example_Process();
foreach ( $user_ids as $user_id ) {
    $this->example_process->push_to_queue( array( 'user_id' => $user_id, 'file_path' => '/myfile.csv' ) );
}
// will result in something like
array(
 array( 'user_id' => 1, 'file_path' => '/myfile.csv' ),
 array( 'user_id' => 2, 'file_path' => '/myfile.csv' ),
 array( 'user_id' => 3, 'file_path' => '/myfile.csv' ),
 array( 'user_id' => 4, 'file_path' => '/myfile.csv' ),
)

As you can see, I am saving a lot more info about each item than I really need to. And it gets much worse quickly when handling thousands of items. PHP serialize/unserialize will max out server memory pretty quicklly when serializing huge arrays like this.

[brocheafoin]

Why not use a property of you WP_Example_Process subclass? Either a static one or a dynamic one which you can populate with the constructor?

[ianmjones]

Your WP_Background_Process subclass that implements task() can pull in data from anywhere based on the item that was queued.

So you can queue an array of data like your example, and then pull in extra information from a related table that holds metadata for the task at hand. You can also add objects to the queue that when task() hydrates them they pull in data you need.

And as @brocheafoin said, you can specialise your WP_Background_Process subclasses so that when they are created you give them specific information that then sets the stage for all the background processing it does, including setting the $action so it only looks for a set of batches for a particular task, and then uses them to process data in the background to complete that task.