django-postgres-copy icon indicating copy to clipboard operation
django-postgres-copy copied to clipboard

No dynamic encoding for adapted params

Open osvill opened this issue 1 year ago • 0 comments

All adapted query parameters in copy_to.py line 45 have the default encoding of latin1. This comes from the psycopg2.extensions.adapt function.

This behavior leads to an implicit error if you try something like MyModel.objects.filter(name="björn").to_csv(export_path) and your database's encoding is set to UTF8, because of the ö in björn. The generated SQL looks like: COPY (SELECT "mymodel"."id", "mymodel"."name", "mymodel"."num", "mymodel"."dt", "mymodel"."parent_id" FROM "mymodel" WHERE "mymodel"."name" = 'bj�rn ') TO STDOUT DELIMITER ',' CSV HEADER and there will be no match with name='bj�rn' for björn.

So my suggestion is to add following code to copy_to.py line 49:

48    with connections[self.using].cursor() as c:
49        # set client encoding to adapted params    # new
50        client_encoding = c.connection.encoding    # new
51        for p in adapted_params:    # new
52            p.encoding = client_encoding if client_encoding else p.encoding    # new

This will generate the right SQL: COPY (SELECT "mymodel"."id", "mymodel"."name", "mymodel"."num", "mymodel"."dt", "mymodel"."parent_id" FROM "mymodel" WHERE "mymodel"."name" = 'björn') TO STDOUT DELIMITER ',' CSV HEADER

osvill avatar Mar 13 '23 14:03 osvill