Configuring secrets in self-hosted QFieldCloud instance

[Forstrevier]

Hi. I have deployed a qfieldcloud instance in production environment on my local server. I have a problem understanding this and unfortunately did not find a solution when searching for answers. The local instance runs flawlessly. I log in via /admin and can maintain almost all data as I know it on qfield.cloud. But I can't find anything about "secrets" in the admin frontend. I have quite a few layers in the project whose data is in a separate Postgres database. On qfield.cloud, access can be easily set up via secrets. In my local qfieldcloud instance I see a menu item "geodbs". There I can specify username, database name, hostname and port. But no password for the external database server.

Is "geodbs" actually the analog to "secrets"? If I want to create a pg_service.conf manually, where would I put this file? Just add it to the project files? Or in one of the Docker directories (e.g. volume) on my local qfieldcloud server?

Currently, the sync from QGis to my local qfieldcloud server is working. But from QField on my tablet I get the following error messages when syncing exactly to the PostGis layers (example): Log from qfieldcloud-Instance (admin frontend) 08:18:21.324 QGSMSGLOG WARNING Connection to database failed definition of service "forstdb" not found

Display QField on Android-Tablet: QFieldCloud had troubles packaging your project Some layers have not been packaged correctly on QFieldCloud. These layers might be misconfigured or their data source is not accessible from the QFieldCloud server. ..... ...Project "projectid" Packaged layer "layername" is not valid. Error code invalid_dataprovider, error message: Unable to connect to service "servicename"

It would be nice if someone could help me understand it better how to map "secrets" in a local qfieldcloud instance.

[suricactus]

Hey @Forstrevier , thanks for the question, you did a good and correct investigation.

There is no UI way to configure secrets using QFieldCloud Admin. The only available UI client is by using qfield.cloud.

It is relatively easy and fast to develop such admin interface, but unfortunately is not currently on our radar and you can imagine there are ton other things to do before a release.

If you want such feature developed and supporting the broader QFieldCloud community, drop us an email [email protected] . Altetrnatively community PRs are very well welcomed.

[Forstrevier]

Hi @suricactus, many thanks for your reply. Regarding community relations, I'll send another email. But how could an adhoc solution look like on a manual basis as server admin? To tell qfieldcloud what my service (for example, "forstdb") looks like, I guess I need to provide it with a well-defined file called pg_service.conf. The content of the file is clearly specified. But where must the file go so that qfieldcloud recognizes and uses it when assembling the package?

[suricactus]

The proper way would be to open run docker compose exec app manage.py shell and execute command similar to this one:

Secret.objects.create(
            name=f"PG_SERVICE_{suffix}",
            type=Secret.Type.PGSERVICE,
            value=value,
            project=project,
            created_by=user,
        )

Of course you need to add proper imports and set project and user to make it work.

Simply storing the .pg_service file cannot do it, since QFieldCloud has to encrypt the secrets in the database. If you really insist doing it with a file you have to modify the code in this repository.

[Forstrevier]

Hi @suricactus, thank you very much, this is very helpful. Yes, the pg_config file is basically on my separate database server and provides the service for external access. And the secure encryption when accessing such a service is an exemplary measure that qfieldcloud offers. That would already tempt me to program a corresponding extension for the admin frontend :-) Thanks again for your help.

[Forstrevier]

Documentation of an interim solution for generating secrets in self-hosted QFieldCloud instance

Goal setting: Create a valid record in the "core_secret" table in the "qfieldcloud_db" database.

1. Preparing a python script

——- snip ——-

from qfieldcloud.core.models import Secret
from qfieldcloud.core.models import Project
from qfieldcloud.core.models import User
        
Secret.objects.create(
            name=„PG_SERVICE_YourQFieldCloudServiceName",
            type=Secret.Type.PGSERVICE,
            project=Project.objects.all().filter(name= "YourProjectName").first(),
            created_by=User.objects.all().filter(username= "YourUserName").first(),
            value=(
                "[YourDBserverServiceName]\n"
                "dbname=YourDBname\n"
                „host=YourHost\n“
                "port=YourPort\n"
                "user=YourDBuser\n"
                "password=YourPassword\n"
                "sslmode=disable\n"
            ),
  )

——- snap ——-

Note: The value block must conform to the specification of pg_service.conf. So maybe the ssl-mode must be enabled, etc. Also note that the secret name has a naming convention. The name must start with "PG_SERVICE_" before you can add your own information to the name.

2. Launch the Python shell

It is assumed that you have successfully deployed a selv-hosted instance of QFieldCloud to your server and that all containers are running properly. Change to the "qfieldcloud" directory and start the Python shell in the app container with the following command line: docker compose exec app python manage.py shell

3. Run the Python snippet

Of course, you replaced all the "Your..." placeholders in the Python snippet above with your own data. Then all you have to do is copy and paste the snippet into your open Python shell and run it. You can see that everything went well by the fact that there is no error message but a confirmation (see image below) that the record was created.

The number indicates the current record number and can of course be different for you. A look into the table "core_secret" also shows you that your secret is now created. The column "value", a bytea type, contains the encrypted data that is needed for the packet handling of (possibly external) PostGis layers. You are done and can exit the Python shell with CTRL-D or the command exit(0)

I have tested the above interim solution myself, and both the synchronization from QGis to the self-hosted QFieldCloud and the synchronization between the self-hosted QFieldCloud and QField on an Android tablet with a test project and over 30 PostGis layers work flawlessly.

I hope my post here is also useful to many in the QFieldCloud community. Good luck!

[suricactus]

Hey, @Forstrevier this is great!

Since your locale uses double quotes which are different from ", hopefully ' is safer choice. Here is a slightly modified snippet that minimizes the number of quotes and copy-paste-abilitty of the snippet:

Secret.objects.create(
    name='PG_SERVICE_YourQFieldCloudServiceName',
    type=Secret.Type.PGSERVICE,
    project=Project.objects.get(name='YourProjectName'),
    created_by=User.objects.get(username='YourUserName'),
    # NOTE keep the indentation within the next string as you see it
    value='''
[YourDBserverServiceName]
dbname=YourDBname
host=YourHost
port=YourPort
user=YourDBuser
password=YourPassword
sslmode=disable
'''
    ),
)

Great work! On our side we will think where we can document such community snippets in the future.

[EgidijusViko]

@Forstrevier Hello, can explain more this step: 3. Run the Python snippet

Of course, you replaced all the "Your..." placeholders in the Python snippet above with your own data. Then all you have to do is copy and paste the snippet into your open Python shell and run it. You can see that everything went well by the fact that there is no error message but a confirmation (see image below) that the record was created.

Because I am not sure that work, because I got invalid syntax.

[boardend]

This is implemented with Add Secrets to the admin page #939 and available in QFieldCloud >= v0.27.0