File paths not updated when using 'Package current QGIS project' option

[raherin]

When using the Mergin Maps plugin to upload an existing project using the 'Package current QGIS project' option, the paths to the newly created project's file paths are not updated in the new .qgz and .gpkg files. Instead, the project will still point to the original file paths resulting in unavailable layers when opening the project in the app for the first time or downloading it via the Mergin Maps Plugin on a separate machine. To reproduce this issue:

1. Download the attached .zip file containing a simple survey project
2. Extract the project from the .zip file
3. Open the project_create.qgz file in QGIS (in the example, I used 3.34 LTR)
4. Press the '+' button in on the Mergin Maps Plugin menu
5. In the project creation dialogue, select the 'Package current QGIS project' option to create the project
6. Click 'Next'
7. In the next dialogue window, select a different directory for the 'Create Mergin Maps Project in:' than the current project directory and press 'Finish'
8. Once the project creation is finished, download and open the project in the Mergin Maps mobile app and an error will appear stating that there are missing layers.

project_create.zip

There is a workaround if this error occurs after creating the project. After finishing step 7 in the above sequence, go to the new project directory and open the project in QGIS. Then follow the steps provided in the QGIS documentation to fix the paths to the layer data sources for each of the layers in your project and synchronise the changes in the plugin.

[drekseimertutje]

Any update on this issue? We have about 100 employees which have this issue. Some of them also have this issue when you synchronize an existing project.

[tomasMizera]

I just tried to reproduce the issue but could not. Maybe a silly question, but did you select the option "package" for gpkgs?

[drekseimertutje]

Hi Tomas,

Yes we do.

The strange thing is, my computer with the same QGIS and Plugin version does not have this issue. Also, this issue only appeared for my colleagues after they got new laptops with fresh installs. Even with QGIS 3.24 and the previous version of the plugin does the issue arrise so to me, it means it is Windows (configuration) related.

Can you perhabs tell which environment variables (related to the layer file paths) are used by the plugin?

[raherin]

Another user has reported this problem while using QGIS 3.40.0 and Mergin Maps plugin version 2024.2.1

[ValentinBuira]

Hi, I tried to reproduce it again, and unfortunately I could not reproduce the issue in the latest versions either.

Could you share the start path of the project and the destination path of the mergin project ? in order to see:

• is there any specials characters ?
• is the path longer than 256 characters (it's a limitation of windows)

[raherin]

@ValentinBuira The path of the original on my machine is: C:\Users\raahe\Documents\Projects\project_create

The path of the packaged project is: C:\Users\raahe\Documents\Projects\package_test/1_package_test

Here is a screenshot of the packaged project pointing to the incorrect source path for the survey layer:

The project will still load if you open it on the same machine where the original is stored because the path still exists on that machine. However, if you open the project in the mobile app or download it on a different machine you get an error that the layer is unavailable.

[ValentinBuira]

Mmmm, I see nothing out of the ordinary about these paths, and in my tests it does point to the correct geopackage in the packaged project... This is quite infuriating

[rvanmaaren]

Hi, It works fine on my desktop. But saving it to a server causes the bug to appear. I've mailed a screen video of the bug with all the paths. Could the bug appear due to a different OS?

[rvanmaaren]

Hi, I've been testing and discovered that it works as usual if I use the 'absolute' absolute path of the server. For my server I use a network drive that is mapped with a letter, like 'G:'. Now when I create a mergin project and save the project in the drive mapping G:/mergin_projects/my_bugged_project than the bug appears. But when I use the absolute server path like //server/user/mergin_projects/my_bugged_project than the project is made as it should. I also found out if the project is once exported the right way (with the server path written out) the bug disappears and exporting this project with the first method, using G:/Mergin_projects/my_bugged_project, also works again. This makes it hard to replicate the bug. You'll need a project where the bug appears to replicate. I hope this helps.

Workaround: Use absolute drive path when exporting: //server/user/mergin_projects/my_bugged_projects

[rvanmaaren]

Also found out; when you copy and past a file where there is already a connection to Mergin Maps, edit the project, upload it again to Mergin, this bug appears. To recreate the bug:

1. create a Mergin project like normal and upload it
2. go to the path and copy the file (with the project, .mergin, geopackages) to another destination
3. open the project from the copied file
4. edit and create a new Mergin project with it (package all layers)

[tomasMizera]

Could it be because of the project path (https://github.com/MerginMaps/qgis-plugin/issues/427) value? (@ValentinBuira)

[ValentinBuira]

I'll try to reproduce with the PR associated to it applied. See if it's still an issue

[rvanmaaren]

Could it be because of the project path (#427) value? (@ValentinBuira)

'project home' and 'project title' in the project properties are empty in my case.

[tomasMizera]

'project home' and 'project title' in the project properties are empty in my case.

Thanks for the quick feedback @rvanmaaren!

[rvanmaaren]

Update on this issue.

This bug is still not fixed. Actually it got worse. I don't know if it became worse with the new update or it just happened to be with this specific project.

I got a project where the datasource was set to the wrong path when using 'package current QGIS project' (like the bug mentioned above). Normally I would just change the data source and the layers were visable in the app. I did this and in QGIS the datasource was going to the right path. But when opening the app it gave the missing layers error where the datasource was still going to the wrong path. Weird because in QGIS the right datasource is stored.

I have tried the change datasource function (when you right click the layer), changing the datasource in the layer properties, changing the layer in missing layers window (when opening QGIS) and drag and dropping the right geopackages. All these would not resolve the issue. All would just sync the .qgz file.

The last thing I've tried was to save the styling as default in the datasource database. This made geopackage to sync with undetected changes instead of the project. This fixed it for me. The layers are now visable in the app. This bug is still very frustrating.

[tomasMizera]

Hi @rvanmaaren, we'll do further investigations on our side. Sorry for the trouble

[rvanmaaren]

@tomasMizera Thank you! no worries, if you need help let me know.

I've tested some more. I have found out that the geopackages in the .mergin does not change datasource when I change it in the layers of the project. That is why the above methods did not work (the .qgz sync). So I think the path is stored inside the geopackage and Mergin Maps uses the .mergin file in the app. Could this be true? When changing the datasource styling it renews the geopackage and thus changing the .mergin geopackage as well.

So I think it has to do with the geopackage styling or stored credentials that are not synced with the synchronisation.

If you re-upload a project would the geopackage styling or credentials be renewed or reused? This may be the cause of the path linking to the old path. Like the bug mentioned before.

[tomasMizera]

Thanks for the details @rvanmaaren, I'll forward this to our dev team :)

[JanCaha]

@rvanmaaren Would it be possible to share what the layers that do not work have in Path under Layer Properties -> Information prior to running package current QGIS project?

There is some custom logic that handles transfer that seems to not be working in some cases but it is very hard to replicate.

[JanCaha]

I believe that the issue will be in: https://github.com/MerginMaps/qgis-plugin/blob/7da905e6de997cdc7830b239535442624e21193a/Mergin/utils.py#L566C1-L591C53 where some non-expected value comes as dataSourceUri() which results in returning None and thus not editing the layer properly. But since it is hard to replicate we need some good example.

[rvanmaaren]

project_with_bug_before_uploading.zip

@JanCaha while testing, after deleting some layers (because of privacy reasons) I found out that the layers were still beeing uploaded. I deleted it from the project and the file. But the layers were still copied to the mergin file.

I've placed a .zip of the project that has the bug. We can also video call and show you how it works.

The path is nothing crazy. Just the path where the geopackage is located. It does not matter where this is on the computer (server space or local.

path: C:\Users\Robin\Desktop\projects\project_with_bug_before_uploading\knip_lijn.gpkg

[JanCaha]

@rvanmaaren

Thanks for this. It helps a bit but still do not quite see why files are not copied properly.

If you would have some time today for a quick chat and demo (likely 10-15 minutes), we can take a closer look. I have a full schedule till about 10:30 CET but should be available after that. Feel free to reach out to my email [email protected]

[rvanmaaren]

@JanCaha important things to know about this project: project_with_bug_before_uploading.zip

The layers that did not work came from another already packaged Mergin Project. So I packaged a already packaged existing Mergin Project to another location. Then added the layers that did work in the demo.

Only saving the style to the geopackage (datasource) forces mergin to sync the geopackage. Only then the layer will be visible on the phone (refers to the right path).

[JanCaha]

@rvanmaaren could you test a workflow?

• create completely new project
• add the layers to the project, preferably stored in location outside of existing MM project
• pack the project using MM plugin

And let me know how it behaves ?

[rvanmaaren]

@JanCaha create folder on desktop 'test_mergin_bug' create folder in 'test_mergin_bug' as 'start_project' create folder in 'test_mergin_bug' as '1st_packaged_mergin_project' create folder in 'test_mergin_bug' as '2nd_packaged_mergin_project' create folder in 'test_mergin_bug' as 'copied_packaged_mergin_project'

open QGIS sign up to mergin maps (with mergin maps plugin) save QGIS file to 'start_project' as 'start_project.qgz' create new geopackage point layer (save to 'start_project') to 'created_layers.gpkg' as 'created_layers' create new geopackage polygon layer (save to 'start_project') to 'created_layers.gpkg' as 'polygon' open wfs layer from pdok ('Provincies') select and export selected to 'pdok_layer.gpkg as 'Provincies' Saving project current layers in the project: - pdok_layer - Provincies - polygon - created_layers create mergin project package current QGIS project

• all layers selected as package
• workspace = BGZN
• project name = test_1st_packaged_project
• save to '1st_packaged_mergin_project'
• finish
• close project

go to '1st_packaged_mergin_project copy test_1st_packaged_project paste to 'copied_packaged_mergin_project' go to 'copied_packaged_mergin_project' open 'test_1st_packaged_project.qgz' package current QGIS project

• all layers selected as package
• workspace = BGZN
• project name = 2nd_packaged_mergin_project
• save to '1st_packaged_mergin_project'
• finish
• conclusion: layers do not go to correct path
• close project

go to '1st_packaged_mergin_project/test_1st_packaged_project open 'test_1st_packaged_project.qgz' package current QGIS project

• all layers selected as package
• workspace = BGZN
• project name = test_2nd_packaged_project
• save to '2nd_packaged_mergin_project'
• finish
• conclusion: layers do not go to correct path
• close project

I think this creates the bug consistent

test_mergin_bug.zip

[JanCaha]

@rvanmaaren I do not quite get the complex logic here. It is way to complicated and some of the steps ale likely creating the issue.

Likely source of the issue are these steps

project name = 2nd_packaged_mergin_project
save to '1st_packaged_mergin_project'

copy test_1st_packaged_project
paste to 'copied_packaged_mergin_project'
go to 'copied_packaged_mergin_project'
open 'test_1st_packaged_project.qgz'

You are saving to the location where the files already exist and the paths are thus not updated. Also replacing QGIS project file is not a good idea.

The tool should be used in really simple way. You have current project with layers that are possible scattered around your PC (or external drivers etc.). You select to pack it and store it in empty directory. MM plugin will take care of gathering all the layers into that folder and copying QGIS project file there, while doing this QGIS project is also updated with paths pointing to the selected folder. This created structure is then uploaded to MM server.

If try package project into folder that already contains some of your data or you modify the QGIS project file from outstide of QGIS (by copying it from different location etc.) then things are likely to start behave unpredictably.

[rvanmaaren]

@JanCaha

Sometimes I or coworkers want to use the project with styling and layers for another project. By copy pasting it or creating a new Mergin project is the quickest way. I could test if it still happens in a new project and only adding the layers. But then you have to readjust the settings from the project and maybe the bug will still apear.

I haven't replaced the .qgz file. Just created new folders

desktop\test_mergin_bug\1st_packaged_mergin_project\test_1st_packaged_project <-- first packaged desktop\test_mergin_bug\1st_packaged_mergin_project\2nd_packaged_mergin_project <-- second time desktop\test_mergin_bug\2nd_packaged_mergin_project\test_2nd_packaged_project <-- third (project name is decieving) a bit further from the first

In this test I put everything at the same location. But the bug would also apear if I package current project to documents or to a NAS location.

[JanCaha]

@rvanmaaren Yeh, I see but then using this packaging tool from MM plugin might not be a good fit to such workflow. It has completely different purpose.

desktop\test_mergin_bug\2nd_packaged_mergin_project\test_2nd_packaged_project

Are those subdirectories of another mergin project? That could also lead to issues.

[rvanmaaren]

@JanCaha I just created emty folders to package mergin projects to. So if you follow the workproces then you will get the results I got (I hope). this is the structure: Desktop/ └── test_mergin_bug/ ├── start_project/ │ ├── start_project.qgz │ ├── created_layers.gpkg │ └── pdok_layer.gpkg │ ├── 1st_packaged_mergin_project/ │ ├── test_1st_packaged_project/ │ │ ├── test_1st_packaged_project.qgz │ │ ├── created_layers.gpkg │ │ └── pdok_layer.gpkg │ │ │ └── 2nd_packaged_mergin_project/ │ ├── test_2nd_packaged_project.qgz │ ├── created_layers.gpkg │ └── pdok_layer.gpkg │ └── copied_packaged_mergin_project/ └── test_1st_packaged_project/ ├── test_1st_packaged_project.qgz ├── created_layers.gpkg └── pdok_layer.gpkg

What is the purpose of package current QGIS project?

[JanCaha]

@rvanmaaren The purpose of the "Package current QGIS Project" is to pack QGIS project that is not yet a MM project and turn it into MM project. If you need a copy of a MM project then the correct way to do it would be to do a "Clone". Take a look at https://merginmaps.com/docs/manage/create-project/ for more details.

Be sure to select empty folder for packaging the project.