CumulusCI

CumulusCI copied to clipboard

Describe the bug

Exception in task load_dataset

Table 'Data_Load__c_rt_target_mapping' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

Reproduction steps

1. "load_dataset" task into an empty Salesforce Org.

Your CumulusCI and Python versions

CumulusCI version: 3.46.0 Python version: 3.9.6 (C:\Users\Nigel.Hughes.local\pipx\venvs\cumulusci\Scripts\python.exe)

Operating System

Windows 10

Windows environment

Command Prompt

CumulusCI installation method

pipx

Error Gist

https://gist.github.com/nigelhughesfairsailcom/413c32f8cbcbdcd4d7034a52a4c0a86c

Additional information

No response

@nigelhughesfairsailcom I believe that what's happened here is this:

1. You ran an extract to create your dataset, using a SQLite database as the target.
2. You ran a load operation.
3. Your database file showed a diff after the load, which you couldn't review because it's a binary file.
4. You may have committed the database file.
5. On any load after that initial load, you observed this issue.

Is that an accurate summary?

@davidmreed

I think it is simpler than that.

1. Extract dataset to SQLite target.
2. Build new scratch org.
3. Deploy the metadata.
4. Load the dataset into the new scratch org.

The load fails. I'm sure that I have missed something important but can't work it out. The odd thing is if I load a few rows from a sql file dataset 1st then load the full dataset from SQLite file everything works fine.

Nigel

Thank you for the update. I'll run a quick repro here.

I can't reproduce the issue using the steps you mentioned, which, in combination with your comment that

The odd thing is if I load a few rows from a sql file dataset 1st then load the full dataset from SQLite file everything works fine.

makes me think that something else is going on here. I cannot think why that behavior might be the case.

Is it possible that during an earlier load from this SQLite database file an exception was thrown (the load failed)?

In any case, the root cause is that a table is created to persist the target org's record type ids in order to support mapping data on the way in. In persistent SQLite database files, these temporary tables apparently do not get disposed properly in at least some situations. You can work around this issue by either:

1. Manually dropping any table whose name ends in _target_mapping in SQLite.
2. Using a SQL script file instead (many teams prefer this strategy because these files are diff-able).

I'll review our backlog to see if we already have a work item to change this behavior and add an item if not. Thanks again for the report.

This is logged now as W-10056853

I can confirm that dropping the "_target_mapping" tables fixes the problem.