sqlmesh
sqlmesh copied to clipboard
TobikoData
ValueError: Duplicate key ... found in UniqueKeyDict<models>. Call dict.update(...) if this is intentional On Plan
I use sqlmesh on a dbt project. Initial plan goes as it should, but all the following plans throw error: ValueError: Duplicate key ... found in UniqueKeyDict<models>. Call dict.update(...) if this is intentional.. The thing is, i dont have duplicate models, otherwise it would throw an error on initial plan also. The problem disappears if if delete the state connection tables. Full log:
2025-04-23 21:48:35,019 - MainThread - sqlmesh.dbt.project - DEBUG - Processing project file 'D:\Documents\VCS Проекты\Работа\Dinord\DWH\sqlmesh\dbt_dwh\dbt_project.yml'. (project.py:53)
2025-04-23 21:48:35,022 - MainThread - sqlmesh.dbt.profile - DEBUG - Processing profile 'D:\Documents\VCS Проекты\Работа\Dinord\DWH\sqlmesh\dbt_dwh\profiles.yml'. (profile.py:89)
2025-04-23 21:48:35,040 - MainThread - sqlmesh.core.console - WARNING - postgres adapter does not currently support 'encoding'. (console.py:3189)
2025-04-23 21:48:35,044 - MainThread - sqlmesh.dbt.package - DEBUG - Loading package at 'D:\Documents\VCS Проекты\Работа\Dinord\DWH\sqlmesh\dbt_dwh'. (package.py:61)
2025-04-23 21:48:35,045 - MainThread - sqlmesh.dbt.package - DEBUG - Processing project file 'D:\Documents\VCS Проекты\Работа\Dinord\DWH\sqlmesh\dbt_dwh\dbt_project.yml'. (package.py:63)
2025-04-23 21:48:35,047 - MainThread - sqlmesh.dbt.package - DEBUG - Processing project variables. (package.py:73)
2025-04-23 21:48:36,426 - MainThread - sqlmesh.dbt.loader - DEBUG - Converting audits to sqlmesh (loader.py:168)
2025-04-23 21:48:38,062 - MainThread - sqlmesh.dbt.loader - DEBUG - Converting models to sqlmesh (loader.py:133)
2025-04-23 21:48:38,191 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Pinging the database to check the connection (base.py:2069)
2025-04-23 21:48:38,226 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Executing SQL: SELECT 1 FROM "information_schema"."tables" WHERE "table_name" = '_versions' AND "table_schema" = 'sqlmesh' (base.py:2132)
2025-04-23 21:48:38,234 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Pinging the database to check the connection (base.py:2069)
2025-04-23 21:48:38,237 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Executing SQL: SELECT * FROM "sqlmesh"."_versions" (base.py:2132)
2025-04-23 21:48:38,240 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Pinging the database to check the connection (base.py:2069)
2025-04-23 21:48:38,242 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Executing SQL: SELECT 1 FROM "information_schema"."tables" WHERE "table_name" = '_versions' AND "table_schema" = 'sqlmesh' (base.py:2132)
2025-04-23 21:48:38,244 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Pinging the database to check the connection (base.py:2069)
2025-04-23 21:48:38,246 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Executing SQL: SELECT * FROM "sqlmesh"."_versions" (base.py:2132)
2025-04-23 21:48:38,248 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Pinging the database to check the connection (base.py:2069)
2025-04-23 21:48:38,250 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Executing SQL: SELECT "start_at", "end_at", "previous_finalized_snapshots", "finalized_ts", "expiration_ts", "requirements", "suffix_target", "normalize_name", "snapshots", "name", "plan_id", "previous_plan_id", "promoted_snapshot_ids", "catalog_name_override", "gateway_managed" FROM "sqlmesh"."_environments" WHERE "name" = 'prod' (base.py:2132)
2025-04-23 21:48:38,303 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Pinging the database to check the connection (base.py:2069)
2025-04-23 21:48:38,306 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Executing SQL: SELECT "name", "identifier", "updated_ts", "unpaused_ts", "unrestorable", "next_auto_restatement_ts" FROM "sqlmesh"."_snapshots" AS "snapshots" LEFT JOIN "sqlmesh"."_auto_restatements" AS "auto_restatements" ON "snapshots"."name" = "auto_restatements"."snapshot_name" AND "snapshots"."version" = "auto_restatements"."snapshot_version" WHERE ("name", "identifier") IN (('"staging"."cash"."stg_cash_flow"', '449041174'), ('"staging"."cash"."stg_counteragents"', '2687969201'), ('"staging"."cash"."stg_pl"', '3790231422'), ('"staging"."diadoc"."stg_documents"', '219651434'), ('"staging"."onec_kz"."stg_analytical_categories"', '2125156955'), ('"staging"."onec_kz"."stg_cash_flow"', '4094880226'), ('"staging"."onec_kz"."stg_cf_articles"', '378605926'), ('"staging"."onec_kz"."stg_departments"', '2667051251'), ('"staging"."onec_kz"."stg_income_articles"', '628664579'), ('"staging"."onec_kz"."stg_pl"', '4044930814'), ('"staging"."onec_kz"."stg_spending_articles"', '4216769708'), ('"staging"."onec_ru"."int_sales"', '1218687972'), ('"staging"."onec_ru"."int_transactions_dinero"', '552179404'), ('"staging"."pbi"."pbi_bank_account_balances"', '3455375432'), ('"staging"."pbi"."pbi_business_calendar"', '2070013139'), ('"staging"."pbi"."pbi_calendar"', '2713360467'), ('"staging"."pbi"."pbi_cash_flow"', '3951942935'), ('"staging"."pbi"."pbi_contracts"', '3863985886'), ('"staging"."pbi"."pbi_counteragents"', '1373647408'), ('"staging"."pbi"."pbi_countries"', '3138165168'), ('"staging"."pbi"."pbi_currency_rates"', '1117247137'), ('"staging"."pbi"."pbi_debit_debt_summary"', '2212454583'), ('"staging"."pbi"."pbi_departments"', '1139563139'), ('"staging"."pbi"."pbi_invoices"', '3603758195'), ('"staging"."pbi"."pbi_payment_calendar"', '1335758489'), ('"staging"."pbi"."pbi_sources"', '1589187556'), ('"staging"."pbi"."pbi_transactions_dinero"', '3011649674'), ('"staging"."report"."report_bank_account_balances"', '1999725615'), ('"staging"."report"."report_cash_flow"', '469560751'), ('"staging"."report"."report_contracts"', '1674638746'), ('"staging"."report"."report_counteragents"', '817253988'), ('"staging"."report"."report_countries"', '2437752633'), ('"staging"."report"."report_currency_rates"', '739138800'), ('"staging"."report"."report_debit_debt_summary"', '1247169109'), ('"staging"."report"."report_departments"', '3301505904'), ('"staging"."report"."report_employees"', '1846657792'), ('"staging"."report"."report_organizations"', '3482371810'), ('"staging"."report"."report_payment_calendar"', '1179830437'), ('"staging"."report"."report_sources"', '2965442135'), ('"staging"."report"."report_transactions_dinero"', '1767384763'), ('"staging"."tests"."test_employees_without_onec_entry"', '3221790907'), ('"staging"."timetta"."stg_users"', '2257751839')) (base.py:2132)
2025-04-23 21:48:38,314 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Pinging the database to check the connection (base.py:2069)
2025-04-23 21:48:38,317 - MainThread - sqlmesh.core.engine_adapter.base - DEBUG - Executing SQL: SELECT "id", "intervals"."name", "intervals"."identifier", "intervals"."version", "intervals"."dev_version", "start_ts", "end_ts", "is_dev", "is_removed", "is_pending_restatement" FROM "sqlmesh"."_intervals" AS "intervals" WHERE ("intervals"."name", "intervals"."version") IN (('"staging"."cash"."stg_cash_flow"', '3588804247'), ('"staging"."cash"."stg_counteragents"', '3278901946'), ('"staging"."cash"."stg_pl"', '3588804247'), ('"staging"."diadoc"."stg_documents"', '1608405497'), ('"staging"."onec_kz"."stg_analytical_categories"', '1305280249'), ('"staging"."onec_kz"."stg_cash_flow"', '4124639161'), ('"staging"."onec_kz"."stg_cf_articles"', '4002374067'), ('"staging"."onec_kz"."stg_departments"', '3122757087'), ('"staging"."onec_kz"."stg_income_articles"', '487984595'), ('"staging"."onec_kz"."stg_pl"', '1777488801'), ('"staging"."onec_kz"."stg_spending_articles"', '3753973999'), ('"staging"."onec_ru"."int_sales"', '2352619944'), ('"staging"."onec_ru"."int_transactions_dinero"', '3614335392'), ('"staging"."pbi"."pbi_bank_account_balances"', '2379638957'), ('"staging"."pbi"."pbi_business_calendar"', '4292254580'), ('"staging"."pbi"."pbi_calendar"', '635726587'), ('"staging"."pbi"."pbi_cash_flow"', '4076521035'), ('"staging"."pbi"."pbi_contracts"', '450647744'), ('"staging"."pbi"."pbi_counteragents"', '3905572083'), ('"staging"."pbi"."pbi_countries"', '2629740576'), ('"staging"."pbi"."pbi_currency_rates"', '3792714464'), ('"staging"."pbi"."pbi_debit_debt_summary"', '890431951'), ('"staging"."pbi"."pbi_departments"', '424023697'), ('"staging"."pbi"."pbi_invoices"', '464937208'), ('"staging"."pbi"."pbi_payment_calendar"', '3829260041'), ('"staging"."pbi"."pbi_sources"', '3447347528'), ('"staging"."pbi"."pbi_transactions_dinero"', '3825813391'), ('"staging"."report"."report_bank_account_balances"', '1565506289'), ('"staging"."report"."report_cash_flow"', '180128330'), ('"staging"."report"."report_contracts"', '4121848129'), ('"staging"."report"."report_counteragents"', '1118034529'), ('"staging"."report"."report_countries"', '1666639245'), ('"staging"."report"."report_currency_rates"', '3394162509'), ('"staging"."report"."report_debit_debt_summary"', '447757496'), ('"staging"."report"."report_departments"', '1387050710'), ('"staging"."report"."report_employees"', '1709218548'), ('"staging"."report"."report_organizations"', '1156380656'), ('"staging"."report"."report_payment_calendar"', '1878102547'), ('"staging"."report"."report_sources"', '1245636440'), ('"staging"."report"."report_transactions_dinero"', '2586918596'), ('"staging"."tests"."test_employees_without_onec_entry"', '2028594992'), ('"staging"."timetta"."stg_users"', '3559671745')) ORDER BY "intervals"."name", "intervals"."version", "created_ts" NULLS FIRST, "is_removed" NULLS FIRST, "is_pending_restatement" NULLS FIRST (base.py:2132)
2025-04-23 21:48:38,328 - MainThread - sqlmesh.cli - ERROR - Unhandled exception (__init__.py:53)
Traceback (most recent call last):
File "d:\Documents\VCS Проекты\Работа\Dinord\DWH\.venv\Lib\site-packages\sqlmesh\cli\__init__.py", line 51, in _debug_exception_handler
return func()
^^^^^^
File "d:\Documents\VCS Проекты\Работа\Dinord\DWH\.venv\Lib\site-packages\sqlmesh\cli\__init__.py", line 29, in <lambda>
return handler(sqlmesh_context, lambda: func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "d:\Documents\VCS Проекты\Работа\Dinord\DWH\.venv\Lib\site-packages\sqlmesh\cli\main.py", line 108, in cli
context = Context(
^^^^^^^^
File "d:\Documents\VCS Проекты\Работа\Dinord\DWH\.venv\Lib\site-packages\sqlmesh\core\context.py", line 445, in __init__
self.load()
File "d:\Documents\VCS Проекты\Работа\Dinord\DWH\.venv\Lib\site-packages\sqlmesh\core\context.py", line 632, in load
store[snapshot.name] = snapshot.node # type: ignore
~~~~~^^^^^^^^^^^^^^^
File "d:\Documents\VCS Проекты\Работа\Dinord\DWH\.venv\Lib\site-packages\sqlmesh\utils\__init__.py", line 78, in __setitem__
raise ValueError(
ValueError: Duplicate key '"staging"."cash"."stg_pl"' found in UniqueKeyDict<models>. Call dict.update(...) if this is intentional.
2025-04-23 21:48:38,337 - MainThread - root - INFO - Shutting down the event dispatcher (dispatcher.py:159)
So looks like you have 2 models that result in the same FQN: "staging"."cash"."stg_pl". Did you change anything between the initial plan and the subsequent one? Or just ran the same command two times in a row without any changes?
Have you tried clearing the cache (sqlmesh clean command) between the plan commands?
Yes, i ran the same comand in a row without any changes. I tried sqlmesh clean, but it results in the same error.
Got it, thank you! Any chance you can come up with a minimal reproducible example for this?
Got the same thing, for a sqlmesh-only project, when I added a project to my Config object. Error goes away when I remove it. I'm learning the tool, I just figured adding a project name would be a good idea.
Got the same thing, for a sqlmesh-only project, when I added a
projectto myConfigobject. Error goes away when I remove it. I'm learning the tool, I just figured adding aprojectname would be a good idea.
I think your particular problem is similar to this one raised over here https://github.com/TobikoData/sqlmesh/issues/4497 I'm not sure if it's exactly the same problem as the one raised in this issue, albeit the error message is the same.
In summary, adding a project name is a great idea but it needs to be done before any plan is promoted. We ended up destroying everything so that we could implement it and start using the multi-repo capabilities.
