Easy way to get includes for eager loading

[thomasvanlankveld]

I want to be able to do something like this in my controllers:

$users = User::with($eagerLoads)->get();

In my transformers, I want to be able to use both $defaultIncludes and $availableIncludes. Also, I want to be able to set an "eager loading key" to separate my internal logic from the HTTP API interface.

protected $defaultIncludes = [
    'invite',
    'login',
];

protected $availableIncludes = [
    'posts',
    'group.admin' => 'admin',
];

I wrote a bit of code in my Controller to make this work:

class UsersController extends \BaseController {

    /**
     * Display a listing of users
     *
     * @return Response
     */
    public function index()
    {
        $fractal = new Manager();
        $fractal->setSerializer(new \League\Fractal\Serializer\EmberSerializer());

        $transformer = new UserTransformer();

        $requestedIncludes = Input::get('include');

        if(!is_array($requestedIncludes))
            $requestedIncludes = array($requestedIncludes);

        $availableRequestedIncludes = array_intersect($transformer->getAvailableIncludes(), $requestedIncludes);
        $defaultIncludes = $transformer->getDefaultIncludes();

        $includes = array_merge($availableRequestedIncludes, $defaultIncludes);

        $eagerLoads = array();

        foreach ($includes as $includeKey => $includeName) {
            if (gettype($includeKey) === "string") {
                unset($includes[$includeKey]);
                array_push($eagerLoads, $includeKey);
            } else {
                array_push($eagerLoads, $includeName);
            }
        }

        $users = User::with($eagerLoads)->get();

        return $fractal->createData(new Collection($users, $transformer, 'users'))->toArray();
    }

}

I'd like for this code to be a part of Fractal somewhere. It could work maybe like this:

$eagerLoads = $manager->getEagerLoads(new UserTransformer, Input::get('include'));

Or like this:

$transformer = new UserTransformer;

$eagerLoads = $transformer->getEagerLoads(Input::get('include'));

Now, I'm not sure if maybe this implementation is too Laravel-specific, but I've not come across any guide so far on how to do eager loading with Fractal. I think the way includes work is much too intertwined with Fractal for it to leave the eager loading implementation entirely up to the users.

Also, there were quite a couple of tricky things I needed to solve here. The Transformer needs to exist before we query the database, we need to check if a key exists for the includes or if it just has a value, and we need to combine $defaultIncludes and $availableIncludes. I don't think all users of Fractal should have to reinvent the eager-loading wheel.

Let me know what you think!

[codivist]

:+1: I agree with @thomasvanlankveld, the eager loading should not be re-invented each time. I'm about to implement eager loading in my api. I'll see if I can't figure something out for this.

@thomasvanlankveld have you tried adding in your ideas into the fractal and going from there?

[philsturgeon]

Yeah, I do know what you mean. The problem is trying to work out how to implement this feature, and not have it be so opinionated that it will only work on Eloquent or other specific ORMs, and not be so vague that its no use.

Ideas?

[thomasvanlankveld]

tl;dr: The implementation I suggested above should solve eager loading for almost all ORMs.

Research

Allright, I thought I'd look up how a couple different ORM's handle their eager loading and I think it we should be safe. I found 11 ORMs that have eager loading support. Other ORM's I found either don't have eager loading, or their documentation has gone offline (Idiorm & Paris, Qcodo, QCubed, Rocks, Torpor).

$eagerLoads implementations

In order from 'easy to implement' to 'aaaaaarrghhh'.

These 6 work out of the box:

// Laravel's Eloquent ORM
$books = Book::with($eagerLoads)->get();

// CakePHP's ORM
$query->contain($eagerLoads);

// Yii's ORM
$authors = Author::model()->with($eagerLoads)->findAll();

// Granada ORM
$users = User::with($eagerLoads)->find_many();

// Spot ORM (Not sure about nested relationships here, but only has 95 gh stars so I don't care to much)
$posts = $posts->all()->with($eagerLoads);

// PHPixie's ORM (Although this one might only work as ->with($eagerLoads[0], $eagerLoads[1], etc.))
$fairies=$pixie->orm->get('fairy')->with($eagerLoads)->find_all();

Then for RedBeanPHP we need 1 extra operation:

// RedBeanPHP ORM - Preloader plugin
R::preload(implode(",", $eagerLoads));

For CodeIgniter and Doctrine it would work with a simple foreach loop:

// CodeIgniter (Places properties of related models on the original model. To each his own I guess)
foreach($eagerLoads as $eagerLoad) {
    $post->include_related($eagerLoad);
}

// Doctrine ORM (Doesn't seem to support nested relationships)
foreach($eagerLoads as $eagerLoad) {
    $query->setFetchMode("MyProject\User", $eagerLoad, \Doctrine\ORM\Mapping\ClassMetadata::FETCH_EAGER);
}

FuelPHP should work with a foreach loop as well. For nested relationships to work, we'd need to give it some extra love though: It needs "parent" relationships to be loaded as well, and in the right order. I've seen code like this in Fractal Manager's parseIncludes(), but we would need some refactoring to be able to use it for eager loads as well.

// FuelPHP wants this
$post = Model_Post::query()
    ->related('articles')
    ->related('articles.user')
    ->related('articles.user.profile')
    ->get_one();

// Which means that if Fractal include 'articles' and 'articles.user' with 'articles.user.profile', it could work like this
foreach($eagerLoads as $eagerLoad) {
    $query = $query->related($eagerLoad);
}

Symfony's ex girlfriend Propel seems to be the trouble case. It wants its eager loading defined with the 'from' side of the relationship as well. No idea how to do this:

// Propel wants this to happen
$review = ReviewQuery::create()
  ->joinWith('Review.Book')
  ->joinWith('Book.Author')
  ->joinWith('Book.Publisher')
  ->findOne();

// Or this?
$authors = AuthorQuery::create()
  ->leftJoinWith('Author.Book')
  ->find();

Conclusion

An $eagerLoads array in the form of ['post', 'author', 'publisher.country'] would work for almost all ORMs, for as far as their eager loading awesomeness stretches. Some ORMs just don't support multiple relations or nested relations. For FuelPHP it would have to be ['post', 'author', 'publisher', 'publisher.country'], which we could implement by stealing some of Fractal Manager's magic. Propel is just weird.

In order of appearance

• Laravel's Eloquent ORM
• CakePHP's ORM
• Yii's ORM - Nested & multiple
• Granada ORM
• Spot ORM
• PHPixie's ORM - Multiple
• RedBeanPHP ORM - Preloader plugin
• CodeIgniter's DataMapper ORM
• Doctrine ORM - Nested set
• FuelPHP's ORM
• Propel ORM

[codivist]

I've been thinking about this, and I think this could be done the same way the Paginator Adapters are done. There is a home for all of them, but they are seperated out i.e. IlluminatePaginatorAdapter, ZendFrameworkPaginatorAdapter, etc.

Then the work doesn't have to be reinvented, just extended if it doesn't fit into one of their frameworks.

Right now I use my own extended version of IlluminatePaginatorAdapter to get the pagination we require for our project. I assume it could be very similar, and this should solve the opinionated issue.

my two cents :moneybag:

[jasonlewis]

Just an update but I still think this is relevant and something we'll work towards for a 1.0 release.

[mattkenefick]

+1

[willishq]

Thinking about this, it might be worth going the same way as the pagination adapters and roll out an eager loader adapter for popular ORMs, that way if people want to roll out their own they can create their own adapter to return the data in a way their ORM can handle.

[ababkov]

If it is done, definitely think it should be done with seperate eager loading adapters.

[thybag]

Thought i may as well mention, one fairly easy way to do it is just to get the fractal manager to do the work for you entirely (then grab it's resolved list of includes via the getRequestedIncludes).

So for example in Laravel you'd just have something like this.

 // Use fractal to resolve the valid includes
$manager = new \League\Fractal\Manager();
$manager->parseIncludes(request()->has('include') ? request()->get('include') : '');

// Ask it what it found & apply them as eager load relations on the query.
$withs = $manager->getRequestedIncludes();

if (!empty($withs)) {
        $query = $query->with($withs);
}

[ababkov]

@thybag that assumes your include strings map exactly to your db relationships

[thybag]

Additionally (which is obvious in hindsight) that the includes were correct to begin with as it's not validated against the avaiableIncludes.

Realised the day after I posted & been giving it some further thought.

Seems the big blocker is figuring out which transformer is being returned from each include ahead of actually having the data, since otherwise you can't check the whether the nested includes are valid etc.

Only work around I can think of so far is to override the TransformerAbstract and use lazy Eager loading from within the callIncludeMethod stuff after fractals already checked a given include is valid for whatever transformer it is. Have yet to properly experiment with the idea though.

Would be grateful for any thoughts on how to try and approach it you may have?

[travisfisher]

👍

[lbm-trentm]

Did this ever come close to a working idea? Happy to help build it if we can agree on a solution.

[lubobill1990]

+1

[wensonsmith]

+1 waiting for this feature

[gdespiritorflex]

+1

[Firtzberg]

I think the transformer should have an additional method fired from the Scope before executing the tranformer over a collection. https://github.com/thephpleague/fractal/blob/master/src/Scope.php#L360 In there you can do the eager loading. The method should also be provided with details about the applied includes.

[matthewtrask]

I have to read above, but can someone tl;dr the concise of the group? Once I know that, I can start scoping out the work and getting this feature in.

[Firtzberg]

@matthewtrask I would like to share my approach. I use Laravel (Eloquent). I have implemented a custom scope that prepares the data.

<?php

namespace App\Contracts\Transformers;

use League\Fractal\Scope;

interface PreparingTransformerContract
{
    /**
     * Prepares item data for transformation.
     * @param mixed $data Item data to be prepared.
     * @param Scope $scope Transformation scope.
     * @return void
     */
    public function prepareItem($data, Scope $scope): void;

    /**
     * Prepares collection data for transformation.
     * @param mixed $data Collection data to be prepared.
     * @param Scope $scope Transformation scope.
     * @return void
     */
    public function prepareCollection($data, Scope $scope): void;
}

<?php

namespace App\Extensions\Fractal;

use App\Contracts\Transformers\PreparingTransformerContract;
use League\Fractal\Scope;
use League\Fractal\Resource\Collection;
use League\Fractal\Resource\Item;

class PreparingScope extends Scope
{
    /**
     * {@inheritdoc}
     */
    protected function executeResourceTransformers()
    {
        $this->prepareResource();
        return parent::executeResourceTransformers();
    }

    /**
     * Prepares data if the transformer supports it.
     *
     * @return void
     */
    private function prepareResource(): void
    {
        $transformer = $this->resource->getTransformer();
        if ($transformer instanceof PreparingTransformerContract) {
            $data = $this->resource->getData();
            if ($this->resource instanceof Item) {
                $transformer->prepareItem($data, $this);
            } elseif ($this->resource instanceof Collection) {
                $transformer->prepareCollection($data, $this);
            }
        }
    }
}

<?php

namespace App\Extensions\Fractal;

use League\Fractal\Manager;
use League\Fractal\Resource\ResourceInterface;
use League\Fractal\ScopeFactory as LeagueScopeFactory;

class ScopeFactory extends LeagueScopeFactory
{
    /**
     * {@inheritdoc}
     */
    public function createScopeFor(Manager $manager, ResourceInterface $resource, $scopeIdentifier = null)
    {
        return new PreparingScope($manager, $resource, $scopeIdentifier);
    }
}

I set my extended ScopeFactory to be used when the ScopeFactoryInterface is required. I have a trait to provide a default implementation for Eloquent.

<?php

namespace App\Transformers\Concerns;

use League\Fractal\Scope;
use Illuminate\Database\Eloquent\Collection as EloquentCollection;

trait EagerLoads
{
    /**
     * Names of relations to be eager loaded.
     * Use key to name the scopeSegment if they don't match.
     * @var string[]
     */
    protected $relations = [];

    /**
     * Gets relations to be eager loaded.
     * @param Scope $scope Transformation scope.
     * @return string[] Relation names to be eager loaded.
     */
    protected function getRequiredRelations(Scope $scope): array
    {
        $requiredRelations = [];
        foreach ($this->relations as $key => $relation) {
            $scopeSegment = $relation;
            if (is_string($key)) {
                $scopeSegment = $key;
            }
            if(!$scope->isExcluded($scopeSegment)) {
                $requiredRelations[] = $relation;
            }
        }
        return $requiredRelations;
    }

    /**
     * Eager loads model relations for transformation.
     * @param mixed $model Model whose relations should be eager loaded.
     * @param Scope $scope Transformation scope.
     * @return void
     */
    public function prepareItem($model, Scope $scope): void
    {
        $model->loadMissing($this->getRequiredRelations($scope));
    }

    /**
     * Eager loads relations of models for transformation.
     * @param mixed $models Collection of models whose relations should be eager loaded.
     * @param Scope $scope Transformation scope.
     * @return void
     */
    public function prepareCollection($models, Scope $scope): void
    {
        // Cast to eloquent collection if necessary.
        if (!($models instanceof EloquentCollection)) {
            $models = EloquentCollection::make($models);
        }
        $models->loadMissing($this->getRequiredRelations($scope));
    }
}

There could be custom implementations for the different ORMs. Can you confirm that executeResourceTransformers will be called once?

[matthewtrask]

Can you confirm that executeResourceTransformers will be called once?

I can't confirm that just yet but I will find out.

There could be custom implementations for the different ORMs.

Yea, I would imagine we would need an Eloquent implementation/Doctrine implementation and any other ORM that is out there. However for this I am inclined to start with Eloquent and Doctrine, and then go from there.

[Firtzberg]

Yea, I would imagine we would need an Eloquent implementation/Doctrine implementation and any other ORM that is out there. However for this I am inclined to start with Eloquent and Doctrine, and then go from there.

This part could remain framework agnostic and you can leave the communities to write their own integration packages. I was astonished to discover that you have pagination adapters for the different frameworks. Pagination adapters are one thing. Adding ORM dependencies might be heavy. Also the maintenance might prove difficult. The implementation I provided works nicely, but can be improved since it doesn't support nesting. Imagine you had a PostTransformer including a collection of comments using the CommentTransformer. The CommentTransformer eager loads the commentator (user/creator). With my implementation you have sorted out the N+1 problem when transforming a single post, but you have it again when you transform a collection of posts. Or when the CommentTransformer uses a TinyUserTranformer that requires the 'user_settings' relationship.

[joelharkes]

@Firtzberg is there currently any way to record in your transformers which includes need to eager load what entities?

it would be nice to somehow calculate from which fractal transformer includes which eager loads needs to be done (recursively), before any data is retrieved from the database.

like in a form of:

public function includeUser(){
	return $this->collection(new UserTransformer(), fn ($model) => $model->user); //optional third argument eager load string. or payload data. 
}

now we can call includeUser() without needing the user beforehand, but we can recursively figure out what data/relationships are needed. based on all the 'included' ResourceAbstract

[Firtzberg]

Hi @joelharkes,

is there currently any way to record in your transformers which includes need to eager load what entities?

You partially can. In the provided code you can make $relations an associative array where the key is the inclusion name and value the relation name. It doesn't eager load if the key is marked for exclusion. You have to adapt getRequiredRelations to make it fully work. It would probably look similar to figureOutWhichIncludes from the TransformerAbstract. You probably need $defaultRelationships and $availableRelationships instead of $relations.

it would be nice to somehow calculate from which fractal transformer includes which eager loads needs to be done (recursively), before any data is retrieved from the database.

Dealing with recursive eager loading would require me to write a whole new package to handle possibly circular inclusion and merging of queries. Again it would have to do with getRequiredRelations

BTW, I just swapped load() with loadMissing()

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 4 weeks if no further activity occurs. Thank you for your contributions.