Pipelined fetching of multiple objects from ModelSet

[jkbrzt]

I've implemented simple pipelined fetching of multiple objects from ModelSet. Not sure if it's useful for others in this form, but you can see the code bellow. For efficiency reason, you can also pass raw=True to get plain dicts instead of model instances.

class ModelSet(Set):

    def get_in_pipeline(self, raw=False):        
        key = self.model_class._key
        ids = list(self._set)
        pipe = self.db.pipeline()

        for id in ids:
            pipe.hgetall('%s:%s' % (key, id))

        for i, hash in enumerate(pipe.execute()):
            id = ids[i]
            if raw:
                hash['_id'] = id
                yield hash
            else:
                model = self.model_class(**hash)
                model._id = id
                yield model

[kiddouk]

I find that globally useful but I'd rather have to configure the modelset to have all operations fetched in a (or in fact two) pipeline(s).

But Redisco should offer such a thing.

[dmk23]

So is this going to be merged into the code base?

[kiddouk]

Not yet.

@jkbr and I discussed the need for such a function but I would like something a bit more transparent than another function call on the modelset.

[72squared]

Any progress made on this front? This is almost the only thing missing from redisco. I'd like to be able to do:

foo_list = Foo.objects.get_by_id(ids)

If it pipelined the loading of the objects instead of fetching one by one, that'd be a huge improvement.