dms icon indicating copy to clipboard operation
dms copied to clipboard
verified publisher icon OCA

[12 -> 13.0] muk_dms: OpenUpgrade: dms: error in migration script /opt/odoo/lib/addons/oca-mirror-dms/dms/migrations/13.0.8.1.0/post-migration.py: ('dms.file(3249,).content', None) #4881

Open olaf-wagner opened this issue 9 months ago • 12 comments

When I run the migration in 13.0, it fails in the postmigration step for the dms. Im am using the latest versions of the openupgradelib (3.9.3) and the 13.0 branch from openupgrade itself.

Module

dms

Describe the bug

Post-migration script fails with this error:

2025-03-24 16:20:03,420 2372 INFO elego_v13 odoo.modules.migration: module dms: Running migration [13.0.8.1.0>] post-migration
2025-03-24 16:20:03,422 2372 INFO elego_v13 OpenUpgrade: dms: post-migration script called with version 12.0.2.0.1
2025-03-24 16:20:03,579 2372 ERROR elego_v13 OpenUpgrade: dms: error in migration script /opt/odoo/lib/addons/oca-mirror-dms/dms/migrations/13.0.8.1.0/post-migration.py: ('dms.file(3249,).content', None)
2025-03-24 16:20:03,579 2372 ERROR elego_v13 OpenUpgrade: ('dms.file(3249,).content', None)
Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/api.py", line 748, in get
    value = self._data[field][record._ids[0]]
KeyError: 3249

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/fields.py", line 1038, in __get__
    value = env.cache.get(record, self)
  File "/opt/odoo/lib/odoo/odoo/api.py", line 754, in get
    raise CacheMiss(record, field)
odoo.exceptions.CacheMiss: ('dms.file(3249,).content', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/api.py", line 748, in get
    value = self._data[field][record._ids[0]]
KeyError: 3249

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/openupgradelib/openupgrade.py", line 2331, in wrapped_function
    version,
  File "/opt/odoo/lib/addons/oca-mirror-dms/dms/migrations/13.0.8.1.0/post-migration.py", line 9, in migrate
    env["dms.file"].search([])._compute_mimetype()
  File "/opt/odoo/lib/addons/oca-mirror-dms/dms/models/dms_file.py", line 379, in _compute_mimetype
    record.res_mimetype = guess_mimetype(base64.b64decode(record.content or ""))
  File "/opt/odoo/lib/odoo/odoo/fields.py", line 1065, in __get__
    value = env.cache.get(record, self)
  File "/opt/odoo/lib/odoo/odoo/api.py", line 754, in get
    raise CacheMiss(record, field)
odoo.exceptions.CacheMiss: ('dms.file(3249,).content', None)
2025-03-24 16:20:03,584 2372 ERROR elego_v13 odoo.modules.loading: Error executing post migration script for module dms
: ('dms.file(3249,).content', None)
2025-03-24 16:20:03,596 2372 ERROR elego_v13 odoo.modules.registry: Failed to load registry
Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/api.py", line 748, in get
    value = self._data[field][record._ids[0]]
KeyError: 3249

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/fields.py", line 1038, in __get__
    value = env.cache.get(record, self)
  File "/opt/odoo/lib/odoo/odoo/api.py", line 754, in get
    raise CacheMiss(record, field)
odoo.exceptions.CacheMiss: ('dms.file(3249,).content', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/api.py", line 748, in get
    value = self._data[field][record._ids[0]]
KeyError: 3249

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/modules/registry.py", line 87, in new
    odoo.modules.load_modules(registry._db, force_demo, status, update_module)
  File "/opt/odoo/lib/odoo/odoo/modules/loading.py", line 489, in load_modules
    force, status, report, loaded_modules, update_module, models_to_check, upg_registry)
  File "/opt/odoo/lib/odoo/odoo/modules/loading.py", line 368, in load_marked_modules
    upg_registry=upg_registry,
  File "/opt/odoo/lib/odoo/odoo/modules/loading.py", line 256, in load_module_graph
    migrations.migrate_module(package, 'post')
  File "/opt/odoo/lib/odoo/odoo/modules/migration.py", line 191, in migrate_module
    migrate(self.cr, installed_version)
  File "/usr/local/lib/python3.7/dist-packages/openupgradelib/openupgrade.py", line 2331, in wrapped_function
    version,
  File "/opt/odoo/lib/addons/oca-mirror-dms/dms/migrations/13.0.8.1.0/post-migration.py", line 9, in migrate
    env["dms.file"].search([])._compute_mimetype()
  File "/opt/odoo/lib/addons/oca-mirror-dms/dms/models/dms_file.py", line 379, in _compute_mimetype
    record.res_mimetype = guess_mimetype(base64.b64decode(record.content or ""))
  File "/opt/odoo/lib/odoo/odoo/fields.py", line 1065, in __get__
    value = env.cache.get(record, self)
  File "/opt/odoo/lib/odoo/odoo/api.py", line 754, in get
    raise CacheMiss(record, field)
odoo.exceptions.CacheMiss: ('dms.file(3249,).content', None)
2025-03-24 16:20:03,597 2372 CRITICAL elego_v13 odoo.service.server: Failed to initialize database `elego_v13`.
Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/api.py", line 748, in get
    value = self._data[field][record._ids[0]]
KeyError: 3249

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/fields.py", line 1038, in __get__
    value = env.cache.get(record, self)
  File "/opt/odoo/lib/odoo/odoo/api.py", line 754, in get
    raise CacheMiss(record, field)
odoo.exceptions.CacheMiss: ('dms.file(3249,).content', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/api.py", line 748, in get
    value = self._data[field][record._ids[0]]
KeyError: 3249

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/odoo/lib/odoo/odoo/service/server.py", line 1194, in preload_registries
    registry = Registry.new(dbname, update_module=update_module)
  File "/opt/odoo/lib/odoo/odoo/modules/registry.py", line 87, in new
    odoo.modules.load_modules(registry._db, force_demo, status, update_module)
  File "/opt/odoo/lib/odoo/odoo/modules/loading.py", line 489, in load_modules
    force, status, report, loaded_modules, update_module, models_to_check, upg_registry)
  File "/opt/odoo/lib/odoo/odoo/modules/loading.py", line 368, in load_marked_modules
    upg_registry=upg_registry,
  File "/opt/odoo/lib/odoo/odoo/modules/loading.py", line 256, in load_module_graph
    migrations.migrate_module(package, 'post')
  File "/opt/odoo/lib/odoo/odoo/modules/migration.py", line 191, in migrate_module
    migrate(self.cr, installed_version)
  File "/usr/local/lib/python3.7/dist-packages/openupgradelib/openupgrade.py", line 2331, in wrapped_function
    version,
  File "/opt/odoo/lib/addons/oca-mirror-dms/dms/migrations/13.0.8.1.0/post-migration.py", line 9, in migrate
    env["dms.file"].search([])._compute_mimetype()
  File "/opt/odoo/lib/addons/oca-mirror-dms/dms/models/dms_file.py", line 379, in _compute_mimetype
    record.res_mimetype = guess_mimetype(base64.b64decode(record.content or ""))
  File "/opt/odoo/lib/odoo/odoo/fields.py", line 1065, in __get__
    value = env.cache.get(record, self)
  File "/opt/odoo/lib/odoo/odoo/api.py", line 754, in get
    raise CacheMiss(record, field)
odoo.exceptions.CacheMiss: ('dms.file(3249,).content', None)
2025-03-24 16:20:03,599 2372 INFO elego_v13 odoo.service.server: Stopping gracefully
openupgrade failed

This may be a subsequent error, as I see some of these error logs before:

2025-03-24 16:20:03,383 2372 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0

I haven’t been able to isolate the problem further yet. The actual data — dms.file(3249) — is there and should be correct (at least in the MuK tables); content is one of the columns that are new or migrated.

To Reproduce

  • not clear how to reproduce without the database in question yet

Expected behavior Migration from muk_dms to dms should not fail.

Additional context

OpenUpgrade is run in a docker-container with OpenUpgrade 13 mounted as Odoo repository. Some version information:

odoo@38ee1fa96a81:/$ cat /etc/debian_version 
10.3
odoo@38ee1fa96a81:/$ python3 --version
Python 3.7.3
odoo@38ee1fa96a81:/$ pip3 list | grep openupgrade
openupgradelib        3.9.3

All errors from upgrade log:

2025-03-24 17:03:49,533 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:02,655 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:05,534 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:22,461 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:26,977 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:28,358 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:29,458 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:33,662 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:04:57,047 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:05:07,801 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:05:23,135 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:05:47,220 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:05:47,278 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:05:52,302 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:05:57,235 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:06:22,008 3915 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-24 17:06:22,198 3915 ERROR elego_v13 OpenUpgrade: dms: error in migration script /opt/odoo/lib/addons/oca-mirror-dms/dms/migrations/13.0.8.1.0/post-migration.py: ('dms.file(3249,).content', None) 
2025-03-24 17:06:22,199 3915 ERROR elego_v13 OpenUpgrade: ('dms.file(3249,).content', None) 
2025-03-24 17:06:22,207 3915 ERROR elego_v13 odoo.modules.loading: Error executing post migration script for module dms
2025-03-24 17:06:22,225 3915 ERROR elego_v13 odoo.modules.registry: Failed to load registry

Complete log attached.

elego-mig-v13-ou.log

olaf-wagner avatar Mar 26 '25 13:03 olaf-wagner

Moved here from https://github.com/OCA/OpenUpgrade/issues/4881 as requested by @victoralmau

olaf-wagner avatar Mar 26 '25 13:03 olaf-wagner

Repeat comment by @victoralmau with explanation:

The problem is related to the dms code and/or the migration scripts included there.

I explain what happens:

Running https://github.com/OCA/dms/blob/13.0/dms/migrations/13.0.8.1.0/post-migration.py#L9 produces the error you indicate. The error indicates that there is no value for the content field in file 3249. The value of the content field in the files is defined with a compute according to different possibilities https://github.com/OCA/dms/blob/ca641ce37fc156b20144cde79b78c99579c4a7c5/dms/models/dms_file.py#L395 The reason why the error is displayed is because none of the 3 conditions are met, i.e. the file does not have content_file, content_binary or attachment_id defined. What is the reason why none of these values are defined in this (maybe other) file(s)? The solution will be to define one of those values to all files. Another possibility would be to modify in dms 13.0 to make it possible to have the content field empty (this option would not be the most appropriate).

olaf-wagner avatar Mar 26 '25 13:03 olaf-wagner

Here is the content of the original row in table muk_dms_file. AFAICS all values that are set in other entries are there:

id locked_by custom_thumbnail custom_thumbnail_medium custom_thumbnail_small name active directory storage company color category extension mimetype size checksum content_binary create_uid create_date write_uid write_date reference_name reference_model reference_field reference_id content_file automatic_thumbnail automatic_thumbnail_medium automatic_thumbnail_small message_main_attachment_id partner_id invoice_number date_invoice user_id import_type invoice_type
3249 ~$sign_Briefing_ASERVO_v01.docx t 681 2 1 0 docx application/vnd.openxmlformats-officedocument.wordprocessingml.document 162 1432a4d25b5a44945dfa11beaeff557fef09d2ad 7 2021-01-2317:44:28.646817 7 2021-01-2317:44:28.646817 /var/lib/odoo/files/elego_erp_v12/14/1432a4d25b5a44945dfa11beaeff557fef09d2ad
(1row)

The dms is working fine in v12 for that file, though it seems to be a bogus and unneeded entry. If I remove 3249 from the table, we fail with another file:

odoo.exceptions.CacheMiss: ('dms.file(9835,).content', None)

This again has a strange name starting with ~$, so it may be related to certain characters in the filename.

A search shows only those two files with such strange name, so I delete them both for test purposes before the migration.

The next failure then is related to 000240.pdf with index 8615, so I assume these are just the first three files to be processed in that method.

id locked_by custom_thumbnail custom_thumbnail_medium custom_thumbnail_small name active directory storage company color category extension mimetype size checksum content_binary create_uid create_date write_uid write_date reference_name reference_model reference_field reference_id content_file automatic_thumbnail automatic_thumbnail_medium automatic_thumbnail_small message_main_attachment_id partner_id invoice_number date_invoice user_id import_type invoice_type
8615 000240.pdf t 1671 2 1 0 pdf application/pdf 27098 9f127ea69ba96dd62530bd17160ffe579574ad0a 7 2021-01-2320:26:32.335222 7 2021-01-2320:26:32.335222 /var/lib/odoo/files/elego_erp_v12/9f/9f127ea69ba96dd62530bd17160ffe579574ad0a
(1row)

I've also checked that the content for the files is actually available in the volumes of the docker container:

wagner@odoo-mig-1:/opt/dconf/odoo-v13-ou$ docker-compose exec odoo /bin/bash -c 'ls -l /var/lib/odoo/files/elego_erp_v12/14/1432a4d25b5a44945dfa11beaeff557fef09d2ad'
-rw-r--r-- 1 odoo odoo 162 Jan 23  2021 /var/lib/odoo/files/elego_erp_v12/14/1432a4d25b5a44945dfa11beaeff557fef09d2ad
wagner@odoo-mig-1:/opt/dconf/odoo-v13-ou$ docker-compose exec odoo /bin/bash -c 'ls -l /var/lib/odoo/files/elego_erp_v12/9f/9f127ea69ba96dd62530bd17160ffe579574ad0a'
-rw-r--r-- 1 odoo odoo 27098 Jan 23  2021 /var/lib/odoo/files/elego_erp_v12/9f/9f127ea69ba96dd62530bd17160ffe579574ad0a

It might be something simple I'm missing and overlooking all the time.

I'd still guess it is related to the errors previously reported from the openupgradelib:

2025-03-26 14:04:43,789 2598 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
2025-03-26 14:04:56,092 2598 ERROR elego_v13 OpenUpgrade: Pass argument 'env_or_cr' as Cursor parameter until 16.0 
...

olaf-wagner avatar Mar 26 '25 14:03 olaf-wagner

odoo.exceptions.CacheMiss: ('dms.file(9835,).content', None)

The problem is that the value of the content field is not set, maybe it is related to have access to the file indicated in content_file. @pedrobaeza do you have any idea what could be the reason?

victoralmau avatar Mar 26 '25 17:03 victoralmau

I think the problem is because the DMS files are stored in a non compatible storage. MuK had other storages that are not supported by OCA.

pedrobaeza avatar Mar 26 '25 17:03 pedrobaeza

It is correct that MuK DMS uses separate fielstores, though they have the same structure as the ones used by Odoo for the attachments. I have copied that filestore (below /var/lib/odoo/files) manually and mounted in the correct location into the docker image. If the existing migration scripts do not support that, could it be a solution to just copy all the files into the regular Odoo filestore? I haven't tried that yet but it could work wrt. the structure used, I think.

olaf-wagner avatar Mar 26 '25 17:03 olaf-wagner

I think that's the problem indeed.

pedrobaeza avatar Mar 26 '25 17:03 pedrobaeza

I have tried

sudo rsync -au /var/lib/odoo/files/elego.erp_v12/elego_erp_v12/ /var/lib/odoo/filestore/odoo-v13/elego_v13/
sudo chown -R odoo:odoo /var/lib/odoo/filestore/odoo-v13/elego_v13/

now but the migration still aborts with the cache miss :-| I'll check and try more later or tomorrow. Perhaps we also need to adapt the paths stored in the content_file field. I'm not sure what the migration scripts actually except, but would have thought somebody has tried it before.

olaf-wagner avatar Mar 26 '25 18:03 olaf-wagner

We did the migration from MuK DMS 12 to 13 with these scripts and worked, but it was a long time ago and don't remember the details of the used storage, sorry.

About the warning Pass argument 'env_or_cr' as Cursor parameter until 16.0, it's not the culprit of the problem.

pedrobaeza avatar Mar 26 '25 18:03 pedrobaeza

Thank you, that's valuable information, Pedro. And it's really good to know that is has worked once before :-) I think I'll figure out the storage problems.

olaf-wagner avatar Mar 26 '25 18:03 olaf-wagner

I just want to post some updates here in case anybody else is interested. Things are progressing slowly. I was able to continue with the overall migration by commenting out the recomputation of the mime type from the post migration steps. There were lots of other problems of course not related to DMS, but the overall structure got converted and is usable in version 14 with two exceptions:

  • file content from the filestore is not heeded (we can reattach that later, if necessary, but will be looking into the code to do it properly)
  • all users from the access groups are lost, because the corresponding relation table is not migrated

The latter can easily be fixed by

alter table muk_security_access_groups_users_rel rename to dms_access_group_users_rel

but we should probably also add a proper table renaming at the right position in the python code. My colleague Mike is looking into the python issues but he's rather busy with other things.

I'll report here with some patches as soon as we have them.

olaf-wagner avatar Mar 29 '25 13:03 olaf-wagner

I think this is the proper patch for the missing rename:

diff --git a/dms/migrations/13.0.1/pre-migration.py b/dms/migrations/13.0.1/pre-migration.py
index 0e739fae..f655daae 100644
--- a/dms/migrations/13.0.1/pre-migration.py
+++ b/dms/migrations/13.0.1/pre-migration.py
@@ -23,6 +23,7 @@ _table_renames = [
     ("muk_dms_directory_tag_rel", "dms_directory_tag_rel"),
     ("muk_dms_directory_star_rel", "dms_directory_star_rel"),
     ("muk_security_access_groups", "dms_access_group"),
+    ("muk_security_access_groups_users_rel", "dms_access_group_users_rel"),
     ("muk_security_groups_explicit_users_rel", "dms_access_group_explicit_users_rel"),
     # Security Mixin relation tables
     ("muk_dms_directory_groups_rel", "dms_directory_groups_rel"),

olaf-wagner avatar Mar 29 '25 14:03 olaf-wagner

There hasn't been any activity on this issue in the past 6 months, so it has been marked as stale and it will be closed automatically if no further activity occurs in the next 30 days. If you want this issue to never become stale, please ask a PSC member to apply the "no stale" label.

github-actions[bot] avatar Sep 28 '25 12:09 github-actions[bot]