papermerge Upgrade issue 3.2 to 3.3

Discussed in https://github.com/ciur/papermerge/discussions/640

^{Originally posted by thndrbck January 7, 2025} I did an in place upgrade of PM version 3.2 to 3.3. None of the documents I had uploaded, nor the folders I had made in 3.2 were visible in 3.3. I downgraded back to 3.2 and everything was back to normal.

I'm using Portainer to manage my containers. I followed the procedure for upgrade that I've used in the past to go from 3.0 to 3.1 to 3.2. I have that procedure documented and it's worked in the past.

Any advice on upgrading will be appreciated. Need the field feature.

Jan 07 '25 19:01 thndrbck

The problem is that i renamed all underlying database tables in 3.3.

Basically table "core_user" is now "users", "core_basetreenodes", is now "nodes". "core_folders" is now "folders" etc.

So, yes, it won't work just to change the docker image.

Jan 08 '25 17:01 ciur

I see also that 3.2 uses SQLite database and 3.3 doesn't. So what's the best way to bring my instance up to 3.3? Will a backup of 3.2 restore to 3.3? If so, Is there anything to be done with the persistent folders?

Thanks.

Jan 08 '25 19:01 thndrbck

No. But I will fix the import into 3.3 part; it is not that complicated. Currently I am busy with updating landing page and adding an online demo instance. Once I am done with that, I will deal with import from 3.2 into 3.3 issue.

Jan 08 '25 19:01 ciur

Thank you. I'll keep checking back.

Jan 08 '25 22:01 thndrbck

Nudge.

Feb 21 '25 20:02 thndrbck

@thndrbck this is work in progress. In couple of weeks I will release a separate tool (pmdump, a binary) with which you will be able to export all data from 3.3 into a tar.gz and then import that tar.gz into 3.4. Actually tool is being designed to be able to export data also from and 2.0, 2.1, 3.3.

The tool I am talking about is: https://github.com/papermerge/pmdump It is in very early stage of development i.e. work in progress. Still experimenting.

Feb 22 '25 04:02 ciur

Quick update.

Utility for upgrading from 3.2 to 3.4 will be out in couple of days.

Mar 07 '25 06:03 ciur

@thndrbck what database do you use ? SQLite or PostgreSQL ?

Mar 08 '25 05:03 ciur

Sqlite

On Sat, Mar 8, 2025, 12:45 AM Eugen Ciur @.***> wrote:

@thndrbck https://github.com/thndrbck what database do you use ? SQLite or PostgreSQL ?

— Reply to this email directly, view it on GitHub https://github.com/ciur/papermerge/issues/641#issuecomment-2708042933, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEADQQQS6ZMEXI2IMNNSZED2TJ7ZJAVCNFSM6AAAAABUYNJLCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBYGA2DEOJTGM . You are receiving this because you were mentioned.Message ID: @.***> [image: ciur]ciur left a comment (ciur/papermerge#641) https://github.com/ciur/papermerge/issues/641#issuecomment-2708042933

@thndrbck https://github.com/thndrbck what database do you use ? SQLite or PostgreSQL ?

— Reply to this email directly, view it on GitHub https://github.com/ciur/papermerge/issues/641#issuecomment-2708042933, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEADQQQS6ZMEXI2IMNNSZED2TJ7ZJAVCNFSM6AAAAABUYNJLCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBYGA2DEOJTGM . You are receiving this because you were mentioned.Message ID: @.***>

Mar 08 '25 21:03 thndrbck

Could you give a try to pmdump v0.2 ?

With it you can export data from Papermerge DMS 3.2 (running on sqlite database) and import it into Papermerge DMS 3.4 - either sqlite or postgres database.

At this point documentation for it is only here: https://github.com/papermerge/pmdump

Mar 09 '25 07:03 ciur

I will be happy to. Tomorrow afternoon.

On Sun, Mar 9, 2025, 3:17 AM Eugen Ciur @.***> wrote:

Could you give a try to pmdump v0.2 https://github.com/papermerge/pmdump/releases ?

At this point documentation for it is only here: https://github.com/papermerge/pmdump

— Reply to this email directly, view it on GitHub https://github.com/ciur/papermerge/issues/641#issuecomment-2708709449, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEADQQVZUUNBYYDJSDOYFGT2TPTHPAVCNFSM6AAAAABUYNJLCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBYG4YDSNBUHE . You are receiving this because you were mentioned.Message ID: @.***> [image: ciur]ciur left a comment (ciur/papermerge#641) https://github.com/ciur/papermerge/issues/641#issuecomment-2708709449

Could you give a try to pmdump v0.2 https://github.com/papermerge/pmdump/releases ?

At this point documentation for it is only here: https://github.com/papermerge/pmdump

— Reply to this email directly, view it on GitHub https://github.com/ciur/papermerge/issues/641#issuecomment-2708709449, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEADQQVZUUNBYYDJSDOYFGT2TPTHPAVCNFSM6AAAAABUYNJLCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBYG4YDSNBUHE . You are receiving this because you were mentioned.Message ID: @.***>

Mar 09 '25 18:03 thndrbck

I have Papermerge working in Docker containers. The export appears to be working. I'm keeping notes of the entire procedure. I have a second instance to perform when done. I'll post the notes when done.

Is the best place to post that under discussions? Under Papermerge or under pmdump? or both? Which do you recommend?

Mar 28 '25 22:03 thndrbck

Under Papermerge

Mar 29 '25 04:03 ciur

Trying to get 3.4 to run under docker. Using the docker-compose file in the documentation for pm 3.4

One issue I had with it was getting the postgres database container to run. Found i had to remove the trailing / from this line:

volumes:
  - pgdata:/var/lib/postgresql/data/

Change to:

volumes:
  - pgdata:/var/lib/postgresql/data

Now the database container will start

Next issue

I'm getting an NGINX 500 server error

I've redirected port 7001->80, not 12000-> as in the example. I've shut down my prior stack of PM so I know that's not a conflict. I've checked that the port is being redirected with docker ps -a. That's ok. I'm not running nginx on my server, so I know the instance is within the container.

I'm trying to troubleshoot the issue, I've shelled into the running container. The /var/log/nginx/error.log is an empty file.

Checking the /etc/nginx directory, I see there is no sites-available subdirectory. Could this be the problem I'm having?

Thanks Mark

Apr 04 '25 18:04 thndrbck

I can help you only if you share your docker compose and (error) logs

Apr 05 '25 06:04 ciur

Here's my docker-compose file (password sanitized):

services: webapp: image: papermerge/papermerge:3.4.1 environment: PAPERMERGE__SECURITY__SECRET_KEY: admin PAPERMERGE__AUTH__USERNAME: admin PAPERMERGE__AUTH__PASSWORD: admin PAPERMERGE__DATABASE__URL: postgresql://admin:admin@db:5432/pmgdb PAPERMERGE__SEARCH__URL: solr://solr:8983/pmg PAPERMERGE__MAIN__MEDIA_ROOT: /var/media/pmg PAPERMERGE__REDIS__URL: redis://redis:6379/0 PAPERMERGE__OCR__LANG_CODES: "eng" PAPERMERGE__OCR__DEFAULT_LANG_CODE: "eng" volumes: - media_root:/var/media/pmg ports: - "7001:80" depends_on: - db - redis - solr path_template_worker: image: papermerge/path-tmpl-worker:0.3.2 command: worker environment: PAPERMERGE__DATABASE__URL: postgresql://admin:admin@db:5432/pmgdb PAPERMERGE__REDIS__URL: redis://redis:6379/0 PATH_TMPL_WORKER_ARGS: "-Q path_tmpl -c 2" depends_on: - redis ocr_worker: image: papermerge/ocrworker:0.3.1 command: worker environment: PAPERMERGE__DATABASE__URL: postgresql://admin:admin@db:5432/pmgdb PAPERMERGE__REDIS__URL: redis://redis:6379/0 PAPERMERGE__MAIN__MEDIA_ROOT: /var/media/pmg OCR_WORKER_ARGS: "-Q ocr -c 2" depends_on: - redis - db volumes: - media_root:/var/media/pmg i3worker: image: papermerge/i3worker:0.3 command: worker environment: PAPERMERGE__DATABASE__URL: postgresql://admin:admin@db:5432/pmgdb PAPERMERGE__SEARCH__URL: solr://solr:8983/pmg PAPERMERGE__REDIS__URL: redis://redis:6379/0 I3_WORKER_ARGS: "-Q i3 -c 2" depends_on: - redis - db - solr
db: image: postgres:16.1 volumes: - pgdata:/var/lib/postgresql/data environment: POSTGRES_PASSWORD: admin POSTGRES_DB: pmgdb_pat POSTGRES_USER: admin healthcheck: test: pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB interval: 5s timeout: 10s retries: 5 start_period: 10s redis: image: bitnami/redis:7.2 ports: - "6379:6379" environment: ALLOW_EMPTY_PASSWORD: "yes"
solr: image: solr:9.7 ports: - "8983:8983" volumes: - solr_data:/var/solr command: - solr-precreate - pmg volumes: pgdata: media_root: solr_data:

my error log is empty /var/log/nginx # cat error.log /var/log/nginx # ls -al total 12 drwxr-xr-x 1 nginx nginx 4096 Apr 3 23:00 . drwxr-xr-x 1 root root 4096 Mar 15 10:18 .. -rw-r--r-- 1 root root 0 Apr 3 23:00 error.log /var/log/nginx #

Apr 07 '25 15:04 thndrbck

Fixed it! Can't change the PAPERMERGE__SECURITY__SECRET_KEY: 12345

I changed it to alphabetical characters. When I changed it back to it's default value PM started up.

Changed it to another 5 digit key and it worked, so apparently the secret key must be numeric.

Apr 07 '25 23:04 thndrbck

However, I can't log in with the auth username and auth password I set in the webapp section of the docker-compose file.

Apr 07 '25 23:04 thndrbck

getting Error: Status Code 500 at the Papermerge 3.4 login screen? Using the same credentials I used as parameters in the docker-compose file

I'm using a 7 letter user name. No capitals. I've replaced them with * here

If I use admin: admin to build it, as per the documentation, I can log in. Is there a username or password length limit?

Here are logs:

2025-04-09 20:49:16.234 UTC [263269] FATAL: password authentication failed for user "*******"

2025-04-09 20:49:16.234 UTC [263269] DETAIL: Role "*******" does not exist.

Connection matched file "/var/lib/postgresql/data/pg_hba.conf" line 128: "host all all all scram-sha-256"

2025-04-09 20:49:22.804 UTC [263286] FATAL: password authentication failed for user "*******"

2025-04-09 20:49:22.804 UTC [263286] DETAIL: Role "*******" does not exist.

Connection matched file "/var/lib/postgresql/data/pg_hba.conf" line 128: "host all all all scram-sha-256"

2025-04-09 21:07:11.713 UTC [265006] FATAL: password authentication failed for user "*******"

2025-04-09 21:07:11.713 UTC [265006] DETAIL: Role "*******" does not exist.

Connection matched file "/var/lib/postgresql/data/pg_hba.conf" line 128: "host all all all scram-sha-256"

Apr 09 '25 21:04 thndrbck

Can't run pmdump import command in a docker container per the instructions. keep getting pmdump not found. Have execute permissions. Only way to not get an error is to precede it with #!/bin/sh but still get 500 Internal Server Error on the login page.

I'm getting frustrated.

May 27 '25 22:05 thndrbck