UnicodeEncodeError

[atodorov]

It looks like somebody tried to upload a file with a funky name to @kiwitcms/Kiwi and we got this traceback in Sentry:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 57-58: ordinal not in range(128)
  File "django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/http.py", line 40, in inner
    return func(request, *args, **kwargs)
  File "django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "attachments/views.py", line 50, in add_attachment
    form.save(request, obj)
  File "attachments/forms.py", line 29, in save
    super(AttachmentForm, self).save(*args, **kwargs)
  File "django/forms/models.py", line 456, in save
    self.instance.save()
  File "django/db/models/base.py", line 729, in save
    force_update=force_update, update_fields=update_fields)
  File "django/db/models/base.py", line 759, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "django/db/models/base.py", line 842, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "django/db/models/base.py", line 880, in _do_insert
    using=using, raw=raw)
  File "django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "django/db/models/query.py", line 1125, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "django/db/models/sql/compiler.py", line 1283, in execute_sql
    for sql, params in self.as_sql():
  File "django/db/models/sql/compiler.py", line 1236, in as_sql
    for obj in self.query.objs
  File "django/db/models/sql/compiler.py", line 1236, in <listcomp>
    for obj in self.query.objs
  File "django/db/models/sql/compiler.py", line 1235, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "django/db/models/sql/compiler.py", line 1185, in pre_save_val
    return field.pre_save(obj, add=True)
  File "django/db/models/fields/files.py", line 287, in pre_save
    file.save(file.name, file.file, save=False)
  File "django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "django/core/files/storage.py", line 48, in save
    name = self.get_available_name(name, max_length=max_length)
  File "django/core/files/storage.py", line 72, in get_available_name
    while self.exists(name) or (max_length and len(name) > max_length):
  File "django/core/files/storage.py", line 308, in exists
    return os.path.exists(self.path(name))
  File "python3.5/genericpath.py", line 19, in exists
    os.stat(path)

Let me know what other info I can provide because I'm not able to reproduce locally.

[bartTC]

Looking into it, this certainly shouldn't happen, especially not on Py3.

[bartTC]

@atodorov what version do you use on production?

[atodorov]

we have version 1.3 in production which is the latest version

[atodorov]

I think this is the same issue: https://github.com/kiwitcms/Kiwi/issues/350

from what I can tell from the traceback it fails when os.path.exists/os.stat is executed.

Maybe that will fail on a non UTF8 system ?

[bartTC]

Hrm. I poked around and had no issues with Py3.6. I had unicode in the filename and even emojis. I'll add them to the testsuite to see if its a Py3.5 issue (doubt).

Maybe that will fail on a non UTF8 system ?

Yeah, that might be it. But I dont see how I could work around that. See https://stackoverflow.com/a/10986010/1460186

[bartTC]

Side note; I figured that a unicode filename wasn't properly handled by Python2.7: https://travis-ci.org/bartTC/django-attachments/jobs/386620989 so I added unicode literals. That fixes that issue, but I doubt, while looking similar, its related. Py3.5 does do that on its own already.

[atodorov]

Maybe that will fail on a non UTF8 system ?

Yeah, that might be it. But I dont see how I could work around that. See https://stackoverflow.com/a/10986010/1460186

According to the issue in our own project, linked above, the user has a Linux system which is fullyc configured for UTF8. However they do use Python 3.5 for that particular version since we haven't made an official release with 3.6 yet.

Edit, look at https://github.com/kiwitcms/Kiwi/issues/350#issuecomment-393486727 - they provide a file name that reproduces the issue.

[bartTC]

Closing this as unable to reproduce.

[atodorov]

@bartTC a user of ours was able to reproduce. I was also able to reproduce with a file name Screenshot_2021-12-07 Плащане.png so reopening.

This is with django-attachments==1.9.1 on Python 3.8.

Here are all of the ENV variables from the running container

$ export
declare -x HOME="/"
declare -x HOSTNAME="ae608d3d65e3"
declare -x OLDPWD
declare -x PATH="/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin     VIRTUAL_ENV /venv"
declare -x PWD="/"
declare -x SHLVL="1"
declare -x TERM="xterm"
declare -x container="oci"

$ set
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:expand_aliases:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="4" [2]="20" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='4.4.20(1)-release'
COLUMNS=155
DIRSTACK=()
EUID=1001
GROUPS=()
HISTFILE=//.bash_history
HISTFILESIZE=500
HISTSIZE=500
HOME=/
HOSTNAME=ae608d3d65e3
HOSTTYPE=x86_64
IFS=$' \t\n'
LINES=38
MACHTYPE=x86_64-redhat-linux-gnu
MAILCHECK=60
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH='/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin     VIRTUAL_ENV /venv'
PIPESTATUS=([0]="0")
PPID=0
PS1='\s-\v\$ '
PS2='> '
PS4='+ '
PWD=/
SHELL=/bin/sh
SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor
SHLVL=1
TERM=xterm
UID=1001
_=export
container=oci

Notably I don't see LANG="en_US.UTF-8" defined.

When I try attaching the same file while running my app in devel mode it works. On my computer I have LANG="en_US.UTF-8" defined of course.

Could this be the issue ?