BaseStorage doesn't handle file inputs with "multiple" attribute

[jt-v]

BaseStorage doens't handle multiple file form fields like this:

attachments = forms.FileField(widget=forms.ClearableFileInput(attrs={'multiple': True}), required=False)

There's a proposed enhancement here: https://github.com/django/django-formtools/issues/58 but it doesn't seem to work.

[clauvt]

I had the same issue with the WizardForm not being able to handle file with attribute 'multiple' when uploading. So I created a new custom class based on the BaseStorage using the suggestion by the #58 above. But, I found out that you also need to modify the "get_step_files" and "reset" functions to work with the enhancement suggestion for "set_step_files".

Here is a copy of my code that works for me. Hope it helps you out with your problem.

class CustomSessionStorage(BaseStorage):
    """
    Customize Session Storage to handle multiple files upload
    """
    def __init__(self, *args, **kwargs):
        super(CustomSessionStorage, self).__init__(*args, **kwargs)
        if self.prefix not in self.request.session:
            self.init_data()

    def _get_data(self):
        self.request.session.modified = True
        return self.request.session[self.prefix]

    def _set_data(self, value):
        self.request.session[self.prefix] = value
        self.request.session.modified = True

    data = property(_get_data, _set_data)

    def reset(self):
        # Store unused temporary file names in order to delete them
        # at the end of the response cycle through a callback attached in
        # `update_response`.
        wizard_files = self.data[self.step_files_key]
        for step_files in six.itervalues(wizard_files):
            for field_dict in six.itervalues(step_files):
                for step_file in field_dict:
                    self._tmp_files.append(step_file['tmp_name'])
        self.init_data()

    def get_step_files(self, step):
        wizard_files = self.data[self.step_files_key].get(step, {})

        if wizard_files and not self.file_storage:
            raise NoFileStorageConfigured(
                "You need to define 'file_storage' in your "
                "wizard view in order to handle file uploads.")

        files = {}
        for key in wizard_files.keys():
            files[key] = {}
            uploaded_file_array = []
            for field_dict in wizard_files.get(key, []):
                field_dict = field_dict.copy()
                tmp_name = field_dict.pop('tmp_name')
                if (step, key, field_dict['name']) not in self._files:
                    self._files[(step, key, field_dict['name'])] = UploadedFile(
                        file=self.file_storage.open(tmp_name), **field_dict)
                uploaded_file_array.append(self._files[(step, key, field_dict['name'])])
            files[key] = uploaded_file_array

        return files or None

    def set_step_files(self, step, files):
        if files and not self.file_storage:
            raise NoFileStorageConfigured(
                "You need to define 'file_storage' in your "
                "wizard view in order to handle file uploads.")

        if step not in self.data[self.step_files_key]:
            self.data[self.step_files_key][step] = {}

        for key in files.keys():
            self.data[self.step_files_key][step][key] = []
            for field_file in files.getlist(key):
                tmp_filename = self.file_storage.save(field_file.name, field_file)
                file_dict = {
                    'tmp_name': tmp_filename,
                    'name': field_file.name,
                    'content_type': field_file.content_type,
                    'size': field_file.size,
                    'charset': field_file.charset
                }
                self.data[self.step_files_key][step][key].append(file_dict)

[jt-v]

Thanks! I tried your code, but when I try get_cleaned_data_for_step('step_with_file_field'), I get None because is_valid() returns False. The error is: [ValidationError(['No file was submitted. Check the encoding type on the form.'].

FileField is the only field on the form for the step 'step_with_file_field'.

[clauvt]

@c-charles1 it sounds like in your html code where you display the form, you didn't set the appropriate encoding type. Did you set the enctype="multipart/form-data" in the

tag? It should look something like this:

<form action="some_action" method="post" enctype="multipart/form-data">
   <input type="file" name="some_files" multiple>
</form>

If you are still having the problem after checking your form's encoding type. Maybe you can try posting some code to your django form and model and the template file.

[euxoa2010]

I have the same issue than c-charles1. I tried your fix clauvt. It should work, but the thing is the code breaks in the end when render_done is called in the WizardView. In the previous steps, forms are directly validated with request.POST, and request.FILES. In render_done they are revalidated by constructing a form_obj from the storage object. At that moment the code breaks as the multiple files are stored in a list, but this will not pass the form validation. Although I was able to pinpoint the issue, I am not able to come up with a solution atm.

[bashu]

@euxoa2010 @jt-v @clauvt still no solution?

[akhilthomas17]

Converting the dict returned by get_step_files to a django.http.request.MultiValueDictwould fix it.

def get_step_files(self, step):
    "unchanged until end"
    files_mval = MultiValueDict(files)
    return files_mval

[HenryMehta]

I'm having the same problem loading multiple files and I'm trying to implement this solution but clearly doing something wrong.

I had been using file_storage = FileSystemStorage(location=os.path.join(settings.MEDIA_ROOT, 'car_images/')).

I replaced with file_storage = CustomSystemStorage(location=os.path.join(settings.MEDIA_ROOT, 'car_images/')).

This gives me an error TypeError: __init__() missing 1 required positional argument: 'request'. How am I meant to implement this solution?

I also tried editing the wizard.storage.base.BaseStorage class by adding the above code as a new class class BaseStorage(OldBaseStorage) and renaming the original OldBaseStorage. This simply resulted in no fields being passed forward correctly.

[atschwarz]

@HenryMehta a bit late, but maybe helpful for others. If you want to use the CustomSessionStorage then you can do the following:

• Add the storage_path to the CustomSessionStorage

class CustomSessionStorage:
    storage_path = '{}.{}'.format(__name__, 'CustomSessionStorage')
    ...

• Use the storage_path variable to specify the storage to use inside your SessionWizardView:

class MySessionWizardView(SessionWizardView):
    storage_name = CustomSessionStorage.storage_path
    file_storage = FileSystemStorage(location=os.path.join(settings.MEDIA_ROOT, 'uploads'))
    ...

[abdul-hamid-achik]

does not work, it says No File was submitted

[atschwarz]

@sicksid I've just created a small django example application which demonstrates a multiple file upload wizard using the view class WizardView and django-crispyforms. Maybe it helps you to fix your errors: https://github.com/astahlhofen/formtools-wizard-multiple-fileupload

The uploaded files of a step inside the wizard are always handled as temporary files. That's caused by the implementation of the function BaseStorage.reset(). If you want to keep the files, you maybe have to overwrite this method or store them at another location using djangos FileSystemStorage.

[abdul-hamid-achik]

hey @astahlhofen i got it working with a session view, any idea how can i implement the same changes with a namedurlsessionview? my problem at the moment is that the first steps always will have the step data empty, but the files will be there, not 100% sure yet why

[ctrlshiftkev]

hi @astahlhofen thank you for the example, it helped me to understand the problem. Now I run into the issue that on the 2nd form all fields are being rendered with errors ('this field is required'). This must have to do with the form validation before being rendered but I'm not sure, I need to look further into it. You wouldn't happen to have an idea?

[atschwarz]

@sicksid - Sorry, but in my use cases I just needed the SessionView. But do you have an example on github? Maybe I can take a look at your code.

@hanfeld - Yes, this sounds like a validation problem. Do you have any code uploaded on github to review your error? I always debug my code using pudb. Maybe it's helpful for you to find the problem.

[japhethmutai]

@astahlhofen, I implemented your code and it works perfectly when I use as it is. However, I need to save the files info to database. I also have other data I am gathering on the other step forms. So, like @hanfeld, I am getting an error in the second step of ('this field is required'), even before I fill it or submit it. Thirdly, the files are not saved to database. Might be there is something I am missing while saving them. I can invite you to collaborate if you need to have a look at the code. Thank you.

[abdalwhabSalahaldeen]

I had the same issue with the WizardForm not being able to handle file with attribute 'multiple' when uploading. So I created a new custom class based on the BaseStorage using the suggestion by the #58 above. But, I found out that you also need to modify the "get_step_files" and "reset" functions to work with the enhancement suggestion for "set_step_files".

Here is a copy of my code that works for me. Hope it helps you out with your problem.

class CustomSessionStorage(BaseStorage):
    """
    Customize Session Storage to handle multiple files upload
    """
    def __init__(self, *args, **kwargs):
        super(CustomSessionStorage, self).__init__(*args, **kwargs)
        if self.prefix not in self.request.session:
            self.init_data()

    def _get_data(self):
        self.request.session.modified = True
        return self.request.session[self.prefix]

    def _set_data(self, value):
        self.request.session[self.prefix] = value
        self.request.session.modified = True

    data = property(_get_data, _set_data)

    def reset(self):
        # Store unused temporary file names in order to delete them
        # at the end of the response cycle through a callback attached in
        # `update_response`.
        wizard_files = self.data[self.step_files_key]
        for step_files in six.itervalues(wizard_files):
            for field_dict in six.itervalues(step_files):
                for step_file in field_dict:
                    self._tmp_files.append(step_file['tmp_name'])
        self.init_data()

    def get_step_files(self, step):
        wizard_files = self.data[self.step_files_key].get(step, {})

        if wizard_files and not self.file_storage:
            raise NoFileStorageConfigured(
                "You need to define 'file_storage' in your "
                "wizard view in order to handle file uploads.")

        files = {}
        for key in wizard_files.keys():
            files[key] = {}
            uploaded_file_array = []
            for field_dict in wizard_files.get(key, []):
                field_dict = field_dict.copy()
                tmp_name = field_dict.pop('tmp_name')
                if (step, key, field_dict['name']) not in self._files:
                    self._files[(step, key, field_dict['name'])] = UploadedFile(
                        file=self.file_storage.open(tmp_name), **field_dict)
                uploaded_file_array.append(self._files[(step, key, field_dict['name'])])
            files[key] = uploaded_file_array

        return files or None

    def set_step_files(self, step, files):
        if files and not self.file_storage:
            raise NoFileStorageConfigured(
                "You need to define 'file_storage' in your "
                "wizard view in order to handle file uploads.")

        if step not in self.data[self.step_files_key]:
            self.data[self.step_files_key][step] = {}

        for key in files.keys():
            self.data[self.step_files_key][step][key] = []
            for field_file in files.getlist(key):
                tmp_filename = self.file_storage.save(field_file.name, field_file)
                file_dict = {
                    'tmp_name': tmp_filename,
                    'name': field_file.name,
                    'content_type': field_file.content_type,
                    'size': field_file.size,
                    'charset': field_file.charset
                }
                self.data[self.step_files_key][step][key].append(file_dict)

it's not work with me can you give me some full example project

[memegauste]

I had the same issue with the WizardForm not being able to handle file with attribute 'multiple' when uploading. So I created a new custom class based on the BaseStorage using the suggestion by the #58 above. But, I found out that you also need to modify the "get_step_files" and "reset" functions to work with the enhancement suggestion for "set_step_files". Here is a copy of my code that works for me. Hope it helps you out with your problem.

class CustomSessionStorage(BaseStorage):
    """
    Customize Session Storage to handle multiple files upload
    """
    def __init__(self, *args, **kwargs):
        super(CustomSessionStorage, self).__init__(*args, **kwargs)
        if self.prefix not in self.request.session:
            self.init_data()

    def _get_data(self):
        self.request.session.modified = True
        return self.request.session[self.prefix]

    def _set_data(self, value):
        self.request.session[self.prefix] = value
        self.request.session.modified = True

    data = property(_get_data, _set_data)

    def reset(self):
        # Store unused temporary file names in order to delete them
        # at the end of the response cycle through a callback attached in
        # `update_response`.
        wizard_files = self.data[self.step_files_key]
        for step_files in six.itervalues(wizard_files):
            for field_dict in six.itervalues(step_files):
                for step_file in field_dict:
                    self._tmp_files.append(step_file['tmp_name'])
        self.init_data()

    def get_step_files(self, step):
        wizard_files = self.data[self.step_files_key].get(step, {})

        if wizard_files and not self.file_storage:
            raise NoFileStorageConfigured(
                "You need to define 'file_storage' in your "
                "wizard view in order to handle file uploads.")

        files = {}
        for key in wizard_files.keys():
            files[key] = {}
            uploaded_file_array = []
            for field_dict in wizard_files.get(key, []):
                field_dict = field_dict.copy()
                tmp_name = field_dict.pop('tmp_name')
                if (step, key, field_dict['name']) not in self._files:
                    self._files[(step, key, field_dict['name'])] = UploadedFile(
                        file=self.file_storage.open(tmp_name), **field_dict)
                uploaded_file_array.append(self._files[(step, key, field_dict['name'])])
            files[key] = uploaded_file_array

        return files or None

    def set_step_files(self, step, files):
        if files and not self.file_storage:
            raise NoFileStorageConfigured(
                "You need to define 'file_storage' in your "
                "wizard view in order to handle file uploads.")

        if step not in self.data[self.step_files_key]:
            self.data[self.step_files_key][step] = {}

        for key in files.keys():
            self.data[self.step_files_key][step][key] = []
            for field_file in files.getlist(key):
                tmp_filename = self.file_storage.save(field_file.name, field_file)
                file_dict = {
                    'tmp_name': tmp_filename,
                    'name': field_file.name,
                    'content_type': field_file.content_type,
                    'size': field_file.size,
                    'charset': field_file.charset
                }
                self.data[self.step_files_key][step][key].append(file_dict)

it's not work with me can you give me some full example project

It doesn't work since Django and stuff changes and it need to get updated. I'm stuck with the same issue.

#EDIT: It works, but you need to clean session cookie though.

[Asday]

Thanks! I tried your code, but when I try get_cleaned_data_for_step('step_with_file_field'), I get None because is_valid() returns False. The error is: [ValidationError(['No file was submitted. Check the encoding type on the form.'].

FileField is the only field on the form for the step 'step_with_file_field'.

Good morning, it's currently 03:00 and I have solved this issue.

For some reason, django-formtools runs the validation for wizards' forms multiple times per request. When files are involved, the second time they're validated, the cursor is at the end of the file(!!!) so the files appear blank.

The correct thing would be for django-formtools to not validate the forms over and over again, but I fixed it in my code with the following. This is part of a larger problem I'm solving which involves uploading multiple files in one field, so there's some cruft here you may not need, but hey. It's 3AM, I'm sick of dealing with this.

class MultiImageField(forms.ImageField):

    def __init__(self, *args, **kwargs):
        kwargs.setdefault('widget', MultipleFileInput())

        super().__init__(*args, **kwargs)

    def clean(self, data, initial=None):
        # Filelikes can end up in this method having been read already,
        # say, if someone decides to force validation many times on the
        # same filelike...  Which means the cursor isn't at the start of
        # the file, and validation will fail confusingly.
        #
        # While the option to be a good citizen and `.seek(0)` the files
        # after validating them, exists, and this lack of politeness
        # causes the issue, expecting the filelikes to be freshly opened
        # is fragile, so defensively `.seek(0)` at the beginning
        # instead.
        def seek0(f):
            f.seek(0)

            return f

        single_file_clean = super().clean
        if isinstance(data, (list, tuple)):
            result = [single_file_clean(seek0(d), initial) for d in data]
        else:
            result = [single_file_clean(seek0(data), initial)]

        return result

[tomasserravento]

It would be amazing if BaseStorage added support for multiple files. Thanks for everything shared here!