Multiple DB Connections with DB Alias Does import to correct DB

[sauravmahuri2007]

For Django projects having multiple DB connections something like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'HOST': '127.0.0.1',
        'NAME': 'default_db',
        'USER': 'postgres',
        'PASSWORD': 'xxxxxxx',
        'OPTIONS': {'options': '-c search_path=sys,public'},
        }
    'basf': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'HOST': '127.0.0.1',
        'NAME': 'basf_db',
        'USER': 'postgres',
        'PASSWORD': 'xxxxxxx',
        'OPTIONS': {'options': '-c search_path=sys,public'},
        },
    'bts': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'HOST': '127.0.0.1',
        'NAME': 'bts_db',
        'USER': 'postgres',
        'PASSWORD': 'xxxxxxx',
        'OPTIONS': {'options': '-c search_path=sys,public'},
        }
}

CSV import (from_csv) doesn't seem to be working. It looks like no matter what ever DB alias I give, it always refer to default DB alias.

eg:

1. MyModel.objects.using('basf').from_csv('/csv/file/import_path.csv') - Imports the data to default DB connection instead of basf.
2. MyModel.objects.using('NotExisted').from_csv('/csv/file/import_path.csv') - Still worked and imported to default DB Alias instead of throwing error: DB connection doesn't exist.

Though, exporting (to_csv) functionality worked perfectly with DB aliases:

1. MyModel.objects.using('basf').to_csv('/csv/file/export_path.csv') -- Exported basf DB perfectly.
2. MyModel.objects.using('NotExisted').to_csv('/csv/file/export_path.csv') - Raised django.utils.connection.ConnectionDoesNotExist which was expected.

Below is my specs:

Ubuntu: 20.04 LTS
Python: 3.8.10
Django==3.2.7
django-postgres-copy==2.7.0

Any quick fix to make CSV import for multiple DB aliases?

I tried something from my end but still didn't work:

from django.db import models
from postgres_copy import CopyQuerySet

class CopyQuerySetWithUsing(CopyQuerySet):

    def using(self, alias):
        """Select which database this QuerySet should execute against."""
        clone = self._chain()
        clone._db = alias
        return clone

class CopyManagerWithUsing(models.Manager.from_queryset(CopyQuerySetWithUsing)):
    pass

class MyModel(models.Model):
    id = models.BigAutoField(primary_key=True)
    user_name = models.CharField(max_length=500, blank=True, null=True)
    first_name = models.CharField(max_length=500, blank=True, null=True)
    last_name = models.CharField(max_length=500, blank=True, null=True)
    user_id = models.CharField(max_length=500, blank=True, null=True)
    user_email = models.CharField(unique=True, blank=True, null=True)

    objects = CopyManagerWithUsing()

    class Meta:
        db_table = 'my_users'

[palewire]

Interesting. I thought I had unittests that covered this use case. Your report suggests they are inaccurate. Do the tests pass for you?

[sauravmahuri2007]

Interesting. I thought I had unittests that covered this use case. Your report suggests they are inaccurate. Do the tests pass for you?

I didn't run any tests. Sorry, I don't know how to run the tests. Would be happy to know the steps.

[sauravmahuri2007]

I went through the code copy_from.CopyMapping and found that using has to be passed as a keyword argument to the method from_csv.

Strange! But below is working:

MyModel.objects.from_csv('/csv/file/import_path.csv', using='basf')

Just an FYI, to_csv will not work as mentioned above. Please check my previous comment, how to use DB aliases for exporting.

[palewire]

Hmm. It's been long enough since I wrote that code that I've forgotten the history. I'm glad we found you a workaround for now. I'll give it a look when I find some time. If you have a PR you'd like to suggest, I'd welcome it.

[sauravmahuri2007]

Currently, I do not have any PR. The code changes I made here didn't work so I've to think of a different way to achieve the import by generic implementation of using. I'll go through the code once again as soon as I get free time. Will let you know if I need any help understanding the code :)