How to handle inlines

[sathishantonlinuxs]

Hi, I have two inline models and category model as follows

class CategoryAdmin(ExportActionModelAdmin, PageAdmin):
    fieldsets = category_fieldsets
    formfield_overrides = {ImageField: {"widget": ImageWidget}}
    filter_horizontal = ("options", "products",)

class ProductVariationAdmin(admin.TabularInline):
    verbose_name_plural = _("Current variations")
    model = ProductVariation
    fields = variation_fields
    max_num = variations_max_num
    extra = variations_extra
    formfield_overrides = {MoneyField: {"widget": MoneyWidget}}
    form = ProductVariationAdminForm
    formset = ProductVariationAdminFormset
    ordering = ["option%s" % i for i in settings.SHOP_OPTION_ADMIN_ORDER]

class ProductImageAdmin(TabularDynamicInlineAdmin):
    model = ProductImage
    formfield_overrides = {ImageField: {"widget": ImageWidget}}

class ProductAdmin(ImportExportMixin,DisplayableAdmin):

   class Media:
       js = (static("cartridge/js/admin/product_variations.js"),)
       css = {"all": (static("cartridge/css/admin/product.css"),)}

   list_display = product_list_display
   list_display_links = ("admin_thumb", "title")
   list_editable = product_list_editable
   list_filter = ("status", "available", "categories")
   filter_horizontal = ("categories",) + tuple(other_product_fields)
   search_fields = ("title", "content", "categories__title",
                 "variations__sku")
   inlines = (ProductImageAdmin, ProductVariationAdmin)
   form = ProductAdminForm
   fieldsets = product_fieldsets

How should i write resource for Product model which includes category and two inline models Productvariation and Productimage.

[jakubste]

Apparently nothing has changed since over year ago and there is no way to do this. That's pity, because this would be great feature. Although I realize it can be difficult to store such values in csv or similar, but should be easily done in json…

[sandeepsayone]

Could have been an awesome feature if implemented.

[praveen-sayone]

Yup. Please add this feature.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tochimclaren]

This should needed to be added to Django import export.

[killkelleyr]

This is definitely needed and I am shocked it has not been added yet

[andrewgy8]

Reopening because it seems like a good feature to add. Cheers everyone for the input.

Having said that, Im not sure how the input/output csv data should be structured. How would you represent the inline relationship. Any ideas with a small example?

[killkelleyr]

Thanks @andrewgy8 ! I have a feeling that dealing with inline models might only be possible in a format such as json/xml i may be wrong though. During my search, I also came across #717 and #851 which seem to be related.

Perhaps with a csv type file you could have the non-inline models be included on each line. example

authorFirstName, authorLastName, bookTitle, bookSKU, bookPublishDate
john, doe, myBook1, 12312312, 06/20/2002

Where the authors attribute would be one model and the book information is the inline model contained within the same admin view.

[CARocha]

Hi, this feature is necessary in import and export :+1:

[LOYAC]

Hi, any updates regarding this? I have a Survey model and Question model that is related to the survey, when exporting a survey I would like all the related questions to show, is that possible?

[c4ys]

Export order and items

class OrderResource(resources.ModelResource):
    class Meta:
        model = Order

    def export(self, queryset=None, *args, **kwargs):
        if queryset is None:
            queryset = self.get_queryset()
        ds = tablib.Dataset()
        data = []
        for order in queryset:
            for item in order.items.all():
                row = {
                    'order_id': order.id,
                    "item_id": item.idd,
                }
                data.append(row)
        ds.dict = data
        return ds

[CARocha]

@c4ys correct if you only have 2 model (Order and Items) but if you have N models in inline example 3 as I would do it?

class OrderResource(resources.ModelResource): class Meta: model = Order

def export(self, queryset=None, *args, **kwargs):
    if queryset is None:
        queryset = self.get_queryset()
    ds = tablib.Dataset()
    data = []
    for order in queryset:
        for item in order.items_set.all(): # you need _set.all for working well
            row = {
                'order_id': order.id,
                "item_id": item.idd,
            }
            data.append(row)
       ### repeat this ???
       for item in order.othermodels_set.all():
            row = {

                "othermodel_id": item.id,
               "othermodel_name": item.name,
              "othermodel_description": item.description,
            }
            data.append(row)
    ds.dict = data
    return ds

i have this error InvalidDimensions at /admin/polls/poll/export/ No exception message supplied

[techscss]

Hi, any update on this one? Seems to be a good feature to add.

[CARocha]

@techscss not yet!!!

[pokken-magic]

The 'stock' way to do this is to create the related models separately. You can do this by creating a resource for each related field and exporting those in bulk and importing them. You can also use dumpdata/loaddata.

Design wise, I think a good way to approach this would be to make a ResourceField that can be based off a Resource designed for the related inline model. The challenges here is that (I think as someone else said) you would be restricted to Json/YAML formats to get good quality data, and if there was any more depth (e.g. the related model related to something else) you could have issues. The CSV formats struggle with nested json sometimes I've seen.

I have an issue that would support this -> https://github.com/django-import-export/django-import-export/issues/1375

[santiagoagullar]

hello! i am dealing with the same issue. How i would go through it in a CSV or XLS is that the columns that correspond to the model with admin.StackedInline or any type of inline class should have a row for each value, and the columns that correspond to non-inline classes should be able to be repeated as many times as there are rows corresponding to the inline class columns, as they would be the "parents" of the inline class, as you can tell in the example i've dropped below, the Product and the Price columns would be inlines, and the other ones would be non-inlines, hope you can implement this! EJEMPLO.xlsx

[pokken-magic]

So the way I did it was for inlines that related to the parent class -- any number of ItemSettings (key value pairs) could relate to an Item. Then I made a RelatedField and RelatedWidget that inherited from ManytoManyWidget, and would create them using the relatedManager.

It was a bit tricky and only properly worked with json and yaml, unfortunately, although you could fix that by forcing it to serialize that as json strings (which could live in csv). the CSV parser has some problems with serializing entire models within a field as is (but I think that could also be fixed if I get a chance to look at it again).

Basically, the trick to creating things at import time is to use the Many to Many manager to inherit from and build around that, since the code is hardcoded to save m2m objects later in the import process.

The stock many to many does not work with the Relatedmanager for a couple reasons, one is that you need to serialize all the fields on the object and two is that there's some light syntactical differences between how relatedmanager work and how many to many relationships work.

In order to make the RelatedManager available you have to override the Clean method of Field and pass the parent object down, to be able to create inlines on the newly created object through the related manager, if that makes sense.

Basically for RelatedManager available inlines (and apologies I have not got permission from work to contribute this code)

1. make a field that inherits from fields.Field, override clean and add obj to the arguments of clean. override save so that you can use the .set method of relatedmanager to load your inlines.
2. make a related widget that inherits from m2m widget, and configure it to know how to determine which inlines to leave and which to keep. In my instance, I just deleted all existing inlines for the model and added the ones in the import (because I knew I would include a complete set of inlines in the imported object).

That's a lot but hopefully illustrates some of the complexity :)

Longer term I think we will add some methods of serializing child resources/sets of related resources - there's an issue on it for looking into using Tablib's databooks for that. https://github.com/django-import-export/django-import-export/issues/1375

[danoctua]

Hi everyone. Are there any updates on this issue? I'm also looking for a clean and efficient solution.

The issues I currently have:

• To create related entries without adding custom fields and resources, I'm overwriting the after_import and making multiple queries to get the parent object and all related entries (for comparing and showing the difference between current objects and the imported ones). That all hits performance and the application is making thousands of queries to the database to import 200 rows.
• While the solution proposed by @pokken-magic looks interesting, it still looks a bit hacky. And as the only format I'm using is csv (the end user would like to review data in Excel later), it seems to be not working in my scenario.

[matthewhegarty]

Hi - there's no progress on the issue. The library is maintained by a very small group who can only work on it part time. If you have any suggestions for a suitable approach, you can add to the discussion here, and then consider raising a PR. I would recommend to discuss the approach first to avoid wasted effort.

If submitting a PR is not an option for you, and if you use the library for commercial purposes, you could consider sponsoring development of this feature. We are receiving similar requests and currently looking into a sponsorship programme. If this is an option for you, you can reach us at [email protected] to discuss further.

[pokken-magic]

I delivered a functional prototype you can check out here, but there is really no good way to do this without a resource -- because to know how to import something, import export uses a resource to understand how to import it. We have made some improvements at automatically creating resources for models by introspection, but there are a lot of field types that don't just work automatically.

Here's the example: https://github.com/django-import-export/django-import-export/pull/1430

There are some architecture gaps in import-export that we need to fix with Fields vs. Widgets--Fields don't know enough about what they're saving to know when to save it, etc.

[tned73]

Hi - I am running into this issue. I have some sort of inline models working, but with the introduction of an Arrayfield in a referenced model, it breaks (it stringify's the content of the field, instead of using the ArrayField detection of the resource).

I have time available to help with the solution, so would like as mentioned above to discuss the approach.

Without diving into the internals, as an api I would like to give a resource-class as input for the widget instead of a model. that way we can the benefits of the ModelResource and it's tweaks and don't overload the widget with more tasks

example

class SubProductResource(resources.ModelResource):    
    class Meta:
        model = SubProduct
        import_id_fields = ["uuid"]

class ProductResource(resources.ModelResource):
    sub_product = fields.Field(
        column_name="sub_product",
        attribute="sub_product",
        widget=ManyToManyWidget(resource=SubProductResource),
    )

    class Meta:
        model = Product
        import_id_fields = ["uuid"]

Let me know if this is in the right direction with the goal of having some sort of general roadmap. Thanks for the great product, hope that I can help with this feature

[matthewhegarty]

Thanks for the comment. We've had a few comments over the years for this, so I'm going to push to see if we can get it into v4, which we're hoping to get out by the end of the year. Ryan made a start on this feature here.

I would like to add that if you use the library for commercial purposes, your company could consider making a contribution towards development of this feature, that would likely mean we could assign someone to work on specifically and increase the chance we could get it into the v4 release. We now have a github sponsors programme.

[pokken-magic]

Using a resource is definitely the the best design for this. I have done json serialized versions and it always had problems once the model got complex. You start with a resource that always works and nest that.

The main issues are:

1. Api differences between related manager fields (in lines), m2m fields, and foreign key fields. To handle all of these we need to redesign the field class a little.
2. Afaik it simply does not behave right without transactions. The sub resource items will probably get created.
3. Structuring the nested datasets is really only appropriate with multiple files or json/yaml. My own personal implementation requires the nested fields to be yaml or json. I think for it to work really well we need to commit on that.

I'd love to finish this PR but I need to lock in the architecture. I built it once with the spaghetti approach I had to take for work and it was a lot of trouble to get right and I don't want to be rewriting it forever.

My preference would be for someone to help figure out the Field / Widget problem at least. Right now the widgets are too aware of manager concepts iirc. There was another partially finished PR that cleaned up that relationship.

If we can get to the point where a field can properly save any type of manager without the widget needing to be aware it would make the code a lot easier for importing sub resources since we wouldn't need duplicate widgets for every manager type.

The other major architectural decision is files or nested data structures (json/yaml). Making this support csv and excel is a serious pig. Excel and csv have some support for nesting but I don't know enough about it to work it out easily and our Tablib setup would need some revisions.

If anyone has insights into getting tablib to work well with multiple excel tabs or sectioned csvs I'd be down to check that out.

If we can come to a defined architecture and scope I am down to do some more work on this particularly this winter.

On Tue, Sep 19, 2023 at 4:45 AM Matt Hegarty @.***> wrote:

Thanks for the comment. We've had a few comments over the years for this, so I'm going to push to see if we can get it into v4, which we're hoping to get out by the end of the year. Ryan made a start on this feature here https://github.com/django-import-export/django-import-export/pull/1430.

I would like to add that if you use the library for commercial purposes, your company could consider making a contribution towards development of this feature, that would likely mean we could assign someone to work on specifically and increase the chance we could get it into the v4 release. We now have a github sponsors programme.

— Reply to this email directly, view it on GitHub https://github.com/django-import-export/django-import-export/issues/445#issuecomment-1725078483, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO4F7GVQ543W2EA75YEDP2LX3FLTJANCNFSM4CCWGRQQ . You are receiving this because you were mentioned.Message ID: @.*** com>

[tned73]

@matthewhegarty Yes we do use it in our commercial service, but unfortunately the current business state is that I am more able to invest time then arrange funding. So as @pokken-magic states, if we could define an architecture and scope I am willing to spend time.

I will at least start with trying to implement my proposal, to get some sense where the issues are located and how it is structured. And get a bit more sense how I should place the above mentioned remarks.

[pokken-magic]

👍 I think my draft PR illustrates a lot of the issues. basically the way things work today with how saving behavior works, the widget has to know a lot about saving behavior and managers and stuff.

https://github.com/django-import-export/django-import-export/pull/1430/files

I am not sure how to solve the two major problems with Field/Widget tight coupling, and file structure for nested datasets. coming to alignment on those is probably the first thing to do.

I am not super excited about trying to make zip files of csvs and then figure out which ones import in which order or whatever personally for whatever that's worth :)

[matthewhegarty]

@tned73 thanks for the offer of help. I wonder if you could provide some sample import files it would help to direct our thinking.

[matthewhegarty]

If we can get to the point where a field can properly save any type of manager without the widget needing to be aware it would make the code a lot easier for importing sub resources since we wouldn't need duplicate widgets for every manager type.

Another idea we have spoken about previously is perhaps leveraging DRF to handle this. We'd maybe be looking quite a serious refactoring exercise due to how closely everything is coupled at the moment, i.e. a Resource is basically a row, but also contains the logic for importing an entire dataset. It would be cool if you could define your data in terms of DRF serializers and it would import ok. Probably a significant task.

[pokken-magic]

So I think that what DRF really brings to the table is the ModelSerializer kinda manages the saving of models? At least that's how I recall it working. so in theory we just do the same thing, tablib makes row into dictionary which goes into a serializer (instead of a resource) and the serializer handles the whole nested saving logic. What that means is basically replacing resources with serializers. Which makes a lot of sense, honestly, and solves a lot of interface problems Resources have.

Unfortunately it doesn't really solve the other problems (file handling, nested data representation in CSV/Excel), so you still have to work that out.

The rewrite would definitely position import-export better for use with DRF, since people could use their existing serializers - but it feels like you're going to want to chuck most of the Widget/Field concept since Serializers basically reinvent that.

You'd probably want to just muck around with a Serializer and see what happens when you load the output of a serializer into Tablib.

On Tue, Sep 19, 2023 at 2:10 PM Matt Hegarty @.***> wrote:

If we can get to the point where a field can properly save any type of manager without the widget needing to be aware it would make the code a lot easier for importing sub resources since we wouldn't need duplicate widgets for every manager type.

Another idea we have spoken about previously is perhaps leveraging DRF https://www.django-rest-framework.org/api-guide/serializers/#dealing-with-nested-objects to handle this. We'd maybe be looking quite a serious refactoring exercise due to how closely everything is coupled at the moment, i.e. a Resource is basically a row, but also contains the logic for importing an entire dataset. It would be cool if you could define your data in terms of DRF serializers and it would import ok. Probably a significant task.

— Reply to this email directly, view it on GitHub https://github.com/django-import-export/django-import-export/issues/445#issuecomment-1726249498, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO4F7GWP32E2LAEVEGULORTX3HN2NANCNFSM4CCWGRQQ . You are receiving this because you were mentioned.Message ID: @.*** com>

[matthewhegarty]

Related: #1375 #1507