reverse lookup prototype

[sinisaos]

Related to #378. Prototype for reverse foreign key lookup. Any suggestions and help are welcome. If I missed the point or it's not a good api design feel free to close PR. Usage of reverse FK lookup:

from piccolo.table import Table
from piccolo.columns import Varchar, ForeignKey
from piccolo.columns.reference import LazyTableReference
from piccolo.columns.reverse_lookup import ReverseLookup

class Manager(Table):
    name = Varchar()
    artists = ReverseLookup(
        LazyTableReference("Artist", module_path=__name__),
    )

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

rows = Manager.select(
        Manager.name, Manager.artists(Artist.id, Artist.name, table=Manager) # or as list Manager.artists(Artist.name, as_list=True, table=Manager)
    ).run_sync()

[codecov-commenter]

Codecov Report

Merging #599 (a7405b7) into master (ff26a88) will decrease coverage by 0.02%. The diff coverage is 88.79%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##           master     #599      +/-   ##
==========================================
- Coverage   91.35%   91.33%   -0.02%     
==========================================
  Files         108      109       +1     
  Lines        7517     7629     +112     
==========================================
+ Hits         6867     6968     +101     
- Misses        650      661      +11     

Impacted Files	Coverage Δ
piccolo/columns/m2m.py	91.19% <ø> (+1.13%)	:arrow_up:
piccolo/query/methods/select.py	96.63% <87.80%> (-1.27%)	:arrow_down:
piccolo/columns/reverse_lookup.py	88.73% <88.73%> (ø)
piccolo/table.py	96.40% <100.00%> (+0.01%)	:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[dantownsend]

Thanks for this! It looks very promising.

[sinisaos]

Thanks. I used a lot of code (with some modifications) from M2M fields. I tried to use the best names of variables and methods. The main obstacle is that the user would have to enter reverse_lookup_name. I will try to make that optional (if table has only one FK default index value for foreign_key_columns will be 0), so that the user has to specify reverse_lookup_name only if the table has multiple FK so ReverseLookup knows which column to use. If you have any ideas, any help is welcome. UPDATE: I removed the reverse_lookup_name and added a table parameter to the ReverseLookupSelect class. With that, Piccolo can identify which table to use for reverse lookup if the table has multiple FKs. I think that is a better option.

[powellnorma]

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

Also I like how in peewee one can just set the backref parameter on the ForeignKeyField, see here Could we have a similar API in piccolo?

[dantownsend]

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

I haven't had chance to review it yet - but I agree that it's a feature we need.

[sinisaos]

@dantownsend It would be great if you could find the time to review this. @powellnorma This code works and passes all tests. If you think that things should be changed, it would be best after Daniel's review. Thank you both.

[powellnorma]

This code works and passes all tests

I haven't looked, but I guess the tests always use a ForeignKey named the same as the referenced Table. As e.g. here:

class Manager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

the sections I pointed lead to errors / wrong queries

[powellnorma]

When I query an M2M field of the referenced table, I get:

AttributeError: type object 'M2MSelect' has no attribute 'value_type'

[powellnorma]

I have fixed the parts that I reviewed on my branch reverse_lookup, see: https://github.com/powellnorma/piccolo/commits/reverse_lookup

However querying an M2M field (of the referenced table) seems to be rather slow at the moment. When I set self.serialisation_safe = False in the case of the above mentioned AttributeError, it seems that all m2m joining keys are being used as arguments to a second query. This is overall (in my case) slower than using a for loop in python (which is located in a transaction) instead of the ReverseLookup. @dantownsend what could we do to make this (querying m2m of reverse-referenced table) faster?

[sinisaos]

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

the sections I pointed lead to errors / wrong queries

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist. Shouldn't it be like this

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

I want to try your branch. In my reverse lookup attempt I use LazyTableReference like this

class Manager(Table):
    name = Varchar()
    bands = ReverseLookup(
        LazyTableReference("Band", module_path=__name__),
    )


class Band(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

# query for all managers bands
query = await Manager.select(
        Manager.name, Manager.bands(Band.name, as_list=True, table=Manager)
    )

Can you show me how to write a table schema and an example query because right now I get TypeError: __init__() missing 1 required positional argument: 'reverse_fk'. Thanks in advance

[powellnorma]

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist.

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

Can you show me how to write a table schema and an example query because right now I get TypeError: init() missing 1 required positional argument: 'reverse_fk'.

Like this:

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

[sinisaos]

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

You are right, it doesn't work if the column is different from the table name. I tried to use foreign_key_columns_index so that if the table has multiple fk columns we know the index of those columns, but it obviously doesn't work well.

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

[powellnorma]

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

Hm, for me that example works. I (already) made changes so that _name should (indeed) be set from the Table Metaclass here. Have you made sure that you checked out my branch correctly? Maybe only parts of my changes got applied?

[sinisaos]

Sorry, that was my mistake. I tried everything from scratch because I probably messed something up and you are right, the tests from my PR pass with your code. The only thing I've noticed is that if the column name is plural the tests fail, but singular name pass. I tried to strip column name if it ends with s and it works in Sqlite but not in Postgres. I have to go now and will be gone for 2 days, but you should continue with this because your version is much better and easier to understand and it works. I'll be happy to delete my PR so you can make a new one. I'll try to help as much as I can.

[powellnorma]

@sinisaos I created a PR on your fork/branch. If you accept it, my changes should be included in this PR.

The only thing I've noticed is that if the column name is plural the tests fail

Do you have an example that demonstrates this problem? How can I reproduce this? Thank you

[powellnorma]

The only thing I've noticed is that if the column name is plural the tests fail

Does it still occur after this commit? https://github.com/sinisaos/piccolo/pull/1/commits/d6c64531246e20c00657e0dbd570268b548668ca

[sinisaos]

@powellnorma I have added your changes. Thanks again for your help.

[lqmanh]

Hi, just wonder what's the status quo of this PR, as it's been around for a long time. Are there any blockers left? I'd really love to see this feature getting merged.

Thanks you guys in advance.

[dantownsend]

@lqmanh Will try and get this merged in soon. It just needs reviewing in detail to make sure we're happy with the API 👍

[powellnorma]

One thing this is currently missing is the ability to use .order_by on the ReverseLookupSelect. Similarly with .where etc. @dantownsend Do you think this should be supported? And if so, do you see a way in which we could reuse the existing implementations for these methods?

[erhuabushuo]

Wow, I'm waiting for this PR.

[py-radicz]

it would be cool to have reverse lookup in the api, cant wait for it

[dantownsend]

OK, seems like a lot of people want this. Getting Piccolo v1 out has taken most of my attention. Will take another look at this once I've released Python 3.12 support. Bear with me, I know this PR has been around for a while 🙏

[kayprogrammer]

You guys are doing well. Still waiting for this PR, including composite unique constraints and check constraints.