papermerge icon indicating copy to clipboard operation
papermerge copied to clipboard
verified publisher icon ciur

setup.py is missing out package 'admin'

Open Lucki opened this issue 5 years ago • 14 comments

I've installed papermerge through setup.py but the resulting package is missing out 'admin' and isn't starting.

A workaround to for this is to copy papermerge/contrib into build/lib/papermerge before running python setup.py build and python setup.py install --root="$pkgdir" --optimize=1 --skip-build later.

Lucki avatar Sep 23 '20 13:09 Lucki

Hi @Lucki,

thank you for trying Papermerge.

In theory it sounded to me as if it is a great idea to have a pypi package for papermerge. But in practice - I never use papermerge pypi package. I always end-up cloning from the repo - even in production environments.

This explains why "admin" is missing (papermerge.contrib.admin was introduced since about 2 months ago).

Thank you for bringing up this issue, I will take care of it!

ciur avatar Sep 23 '20 17:09 ciur

I actually did create all necessary PKGBUILDs for Arch Linux but currently I'm not happy with Papermerge after some fiddling around so I think I'm not going to push them into the AUR.

Lucki avatar Sep 23 '20 17:09 Lucki

I'm not happy with Papermerge after some fiddling around

Maybe you can list points which you didn't like about Papermerge here. I would gladly consider your feedback and improve application in future.

ciur avatar Sep 23 '20 18:09 ciur

I actually did create all necessary PKGBUILDs for Arch Linux but currently I'm not happy with Papermerge after some fiddling around so I think I'm not going to push them into the AUR.

I was just looking for a nice PKGBUILD in the AUR, so I can only encourage you to publish it there and maybe really state the issues you are having. Sometimes, solutions can be found unexpectedly easily ;)

amo13 avatar Sep 24 '20 15:09 amo13

Maybe you can list points which you didn't like about Papermerge here.

I'll do but I need to figure out the exact points first. In general it's that the whole usage feels really clunky and I doubt they aren't already mentioned somewhere around here.

I was just looking for a nice PKGBUILD in the AUR, so I can only encourage you to publish it there

I don't want to maintain these without using them so I won't publish them there. But I drop them here in case someone wants to pick them up. The main PKGBUILD with the associated sysusers, service unit and tmpfiles needs some general polish. I was in the process of hooking gunicorn between django and nginx but my OCR broke while doing that. That's the current state.

papermerge 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

pkgname=papermerge
pkgver=1.4.3
pkgrel=2
pkgdesc="An open source document management system (DMS) primarily designed for archiving and retrieving your digital documents."
arch=('any')
url="https://github.com/ciur/papermerge"
license=('Apache')
backup=("etc/$pkgname.conf.py" "usr/lib/python3.8/site-packages/config/settings/production.py")
depends=('imagemagick' 'pdftk' 'poppler' 'python-celery' 'python-django' 'python-django-allauth' 'python-django-celery-results' 'python-django-dynamic-preferences'
      'python-django-modelcluster' 'python-django-mptt' 'python-django-polymorphic-tree' 'python-django-rest-framework' 'python-django-rest-knox' 'python-django-taggit'
      'python-imapclient' 'python-lxml' 'python-mgclipboard' 'python-mglib' 'python-psycopg2' 'python-pyaml' 'tesseract')
optdepends=('gunicorn: run papermerge in a production environment with nginx'
          'nginx: run papermerge through gunicorn in a production environment')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/ciur/$pkgname/archive/v$pkgver.tar.gz"
      "$pkgname.sysusers"
      "$pkgname.tmpfiles"
      "$pkgname-gunicorn.service"
      "$pkgname-worker.service"
      "$pkgname.sh")
sha512sums=('1665873c666424c456655f91a5e980a519945d937c2ae351306add4b44d3618e4a8700f20d08fb52639c1b37b90f1dfe4565dcfbb8452897133880d83daeb1ea'
          'eb408d6d6c0072b4883cae9b6aa591935b910585e93aa6fdacbb543dbc4233588678d396c06d0f10b7c74b4d393af19b8be526d094821de23792c29df593d4a5'
          '07726d47deb2438dbeb3f5dc6ebc2218a4d5f25dacb7e0304102deddfb914d4d6d649643e26cb844ac4d02c7f21b58835edd4026ff91a6481fee59f786b90925'
          'c99103b8a160dde851e169d4249da8ade66377f428df0784401309a1c181ce4c5011c31b28d26981456e4490f670ff161bf0ebc0c9daf47cdf761b4d5cb9cb32'
          'fcdd49e0b6b93ade84ddafd77111f28bdea75a4f12829ec249d414d46a14399198be7bf29d0ce5166de6dba1d6386eda4ab9b39a1f5376b7c7758c8966a53ce3'
          'b838d54dca56e45f99690ddaae1b058e3f72b314adf9d689869f14bb141a5bbf209df974eea20cb2305b127d5119d0b8d11293494d47350f6974edc6fffc374f')

prepare() {
  {
      echo 'DBDIR = "/var/lib/papermerge/database"';
      echo 'MEDIA_DIR = "/var/lib/papermerge/media"';
      echo 'STATIC_DIR = "/var/lib/papermerge/static"';
      echo 'TASK_QUEUE_DIR = "/var/tmp/papermerge/queue"'
  } >> "$pkgname-$pkgver/$pkgname.conf.py.example"

  # https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html#step-1-install-gunicorn
  {
      echo 'from .base import *  # noqa';
      echo 'DEBUG = False';
      echo "ALLOWED_HOSTS = ['*']"
  } > "$pkgname-$pkgver/config/settings/production.py"

  # The admin package is missing somehow so let's move it plain stupid directly in place
  mkdir -p "$pkgname-$pkgver/build/lib/$pkgname/"
  cp -dpr --no-preserve=ownership "$pkgname-$pkgver/$pkgname/contrib" "$pkgname-$pkgver/build/lib/$pkgname/"
}

build() {
  cd "$pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$pkgname-$pkgver" || exit
  ./run_tests.sh
}

package() {
  # tmpfiles.d
  install -Dm 644 "$pkgname.tmpfiles" \
      "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf"

  # sysusers.d
  install -Dm 644 "$pkgname.sysusers" \
      "$pkgdir/usr/lib/sysusers.d/$pkgname.conf"

  # service files
  install -Dm 644 "$pkgname-gunicorn.service" \
      "$pkgdir/usr/lib/systemd/system/$pkgname-gunicorn.service"
  install -Dm 644 "$pkgname-worker.service" \
      "$pkgdir/usr/lib/systemd/system/$pkgname-worker.service"

  # wrapper
  install -Dm 755 "$pkgname.sh" \
      "$pkgdir/usr/bin/$pkgname-manage.py"

  cd "$pkgname-$pkgver" || exit

  # config
  install -Dm 644 "$pkgname.conf.py.example" \
      "$pkgdir/etc/$pkgname.conf.py"

  export PYTHONHASHSEED=0
  python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}
papermerge-worker.service 
[Unit]
Description=Papermerge Worker
Documentation=https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html
After=network.target
PartOf=papermerge-gunicorn.service

[Service]
Type=simple
User=papermerge
Group=papermerge
WorkingDirectory=/tmp
Environment="DJANGO_SETTINGS_MODULE=config.settings.production"
ExecStart=/usr/bin/django-admin worker --pidfile /var/run/papermerge/papermerge-worker.pid
Restart=on-failure

[Install]
WantedBy=multi-user.target
papermerge-gunicorn.service 
[Unit]
Description=Gunicorn providing papermerge
Documentation=https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html,https://docs.gunicorn.org/en/latest/deploy.html#systemd
After=network.target
Wants=papermerge-worker.service

[Service]
Type=notify
User=papermerge
Group=papermerge
RuntimeDirectory=papermerge
WorkingDirectory=/var/lib/papermerge
Environment="DJANGO_SETTINGS_MODULE=config.settings.production"
ExecStart=/usr/bin/gunicorn config.wsgi:application --config /etc/gunicorn.conf.py
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true
Restart=on-failure

[Install]
WantedBy=multi-user.target
papermerge.sh 
#!/bin/sh
export DJANGO_SETTINGS_MODULE=config.settings.dev
cd /tmp || exit
exec /usr/bin/django-admin "$@"
papermerge.sysusers 
u papermerge - "open source document management system" -
papermerge.tmpfiles 
d /var/lib/papermerge/media - papermerge papermerge -
d /var/lib/papermerge/static - papermerge papermerge -
d /var/lib/papermerge/database - papermerge papermerge -
d /run/papermerge - papermerge papermerge -
d /var/log/papermerge - papermerge papermerge -
python-django-celery-results 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-celery-results
pkgname=python-django-celery-results
pkgver=1.2.1
pkgrel=1
pkgdesc="Store Celery task results using the Django ORM."
arch=('any')
url="https://github.com/celery/django-celery-results"
license=('BSD')
depends=('python-celery')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/celery/django-celery-results/archive/$pkgver.tar.gz")
sha512sums=('9f25d8ecf9716dca04be36f608d682ec138f0d6ceeaff2acdbe3f2c617e0eab8b0f8748dd35eead9fd7c40cdfbfd2ed9663a586a41f70f3a4bebecd3aba90d3d')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# check() {
#     # TODO
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}
python-django-modelcluster 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-modelcluster
pkgname=python-django-modelcluster
pkgver=5.1
pkgrel=1
pkgdesc="Django extension to allow working with 'clusters' of models as a single unit, independently of the database."
arch=('any')
url="https://github.com/wagtail/django-modelcluster"
license=('BSD')
depends=('python-pytz')
optdepends=('python-django-taggit')
makedepends=('python-setuptools')
checkdepends=('python-django' 'python-django-taggit')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/wagtail/django-modelcluster/archive/v$pkgver.tar.gz")
sha512sums=('bb3d8640a04cabfdebe7f60b571a2b2c4d71bbf5d4ea0ff0e7db5fe9d120e6eab7ec3f1b6992d21aff377045f8ed3e11d1b351ce2f460b33dd4c0f3a8dd749fb')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$_pkgname-$pkgver" || exit
  python runtests.py
}

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}
python-django-mptt 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-mptt
pkgname=python-django-mptt
pkgver=0.11.0
pkgrel=1
pkgdesc="Utilities for implementing a modified pre-order traversal tree in django."
arch=('any')
url="https://github.com/django-mptt/django-mptt"
license=('custom')
depends=('python-django' 'python-django-js-asset')
makedepends=('python-setuptools')
checkdepends=('python-coverage' 'python-mock')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/django-mptt/django-mptt/archive/$pkgver.tar.gz")
sha512sums=('0302fbdc759fe3e69c7adc3e785025c4c28931904c2b8904d43a3fdf1471ef1a83f18170f1db05dc3d4360e946c35db06900cf9c40b87562fba03240f9608da7')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# TODO: AssertionError: False is not true : Couldn't find 'javascript" src="/static/mptt/draggable-admin.js" data-context="{&quot;' in response
# check() {
#     cd "$_pkgname-$pkgver/tests" || exit
#     ./runtests.sh
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}
python-django-polymorphic-tree 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-polymorphic-tree
pkgname=python-django-polymorphic-tree
pkgver=1.5.1
pkgrel=1
pkgdesc="Polymorphic MPTT tree support for models"
arch=('any')
url="https://github.com/django-polymorphic/django-polymorphic-tree"
license=('Apache')
depends=('python-django' 'python-django-mptt' 'python-django-polymorphic' 'python-django-tag-parser' 'python-future')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/django-polymorphic/django-polymorphic-tree/archive/v$pkgver.tar.gz")
sha512sums=('291ed73400c3a5073cbfe00fdd0dab16ac7dbb4fa96fc1797f526ef7235111162806c4be3fddb0bc6cd61560bdb6d1ac806c2f83380a1d45735d876c4f79ac67')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$_pkgname-$pkgver" || exit
  python runtests.py
}

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"
}
python-django-rest-knox 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-rest-knox
pkgname=python-django-rest-knox
pkgver=4.1.0
pkgrel=1
pkgdesc="Authentication Module for django rest auth"
arch=('any')
url="https://github.com/James1345/django-rest-knox"
license=('MIT')
depends=('python-cryptography' 'python-django' 'python-django-rest-framework')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/James1345/django-rest-knox/archive/$pkgver.tar.gz")
sha512sums=('ff7252e35ed3998d882ba5f857ba7278340065ab849475b0b3d46a04842eac43f38f089fc138ebbe1495a06c4fd4ddb7bd1025a816ae4ad2c646752e74f85336')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# check() {
#     https://wiki.archlinux.org/index.php/Python_package_guidelines#Check
#     > Avoid using tox to run testsuites
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}
python-django-tag-parser 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-tag-parser
pkgname=python-django-tag-parser
pkgver=3.2
pkgrel=1
pkgdesc="Micro-library to easily write custom Django template tags "
arch=('any')
url="https://github.com/edoburu/django-tag-parser"
license=('Apache')
depends=('python')
makedepends=('python-setuptools')
checkdepends=('python-django')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/edoburu/django-tag-parser/archive/v$pkgver.tar.gz")
sha512sums=('ab39dcf847b1da473ded83655bd8ba7a6e376de68668cd9dc7b2019d8666864ed5665ab162f4a7df56cb7a3eb32e83c8fad3fdecf5774140effc643853997879')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$_pkgname-$pkgver" || exit
  python runtests.py
}

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"
}
python-mgclipboard 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

pkgname=python-mgclipboard
_pkgname=mg-clipboard
pkgver=0.3.0
pkgrel=1
pkgdesc="Modules shared across all Papermerge project."
arch=('any')
url="https://github.com/papermerge/mg-clipboard"
license=('Apache')
depends=('python')
makedepends=('python-setuptools')
_commit="f5a4e9370820c3d8bbfbc3bed08131d32bc141fc"
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/papermerge/mg-clipboard/archive/$_commit.tar.gz")
sha512sums=('25f7787ca3aec9d0f56ddf4dc65a530a39c4238a91040af592b8f5c686f539ddcf734101307aa9ee86f6850825c7eb33ea66e2cfd19c27a96358e755bf1b78a9')

build() {
  cd "$_pkgname-$_commit" || exit
  python setup.py build
}

package() {
  cd "$_pkgname-$_commit" || exit

  export PYTHONHASHSEED=0
  python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}
python-mglib 
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

pkgname=python-mglib
_pkgname=mglib
pkgver=1.2.8
pkgrel=1
pkgdesc="Modules shared across all Papermerge project."
arch=('any')
url="https://github.com/papermerge/mglib"
license=('Apache')
depends=('python')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/papermerge/mglib/archive/v$pkgver.tar.gz")
sha512sums=('edfdc8d7ef861a055cfeba574e15d024053e8d6da2fc0e9696df0055bb91eb90be255b1570dd6d67697b9bfd1f205baf53e53d51a54c68354c44aed7eea2557f')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# TODO: Fails with ModuleNotFoundError: No module named 'mglib'
# check(){
#     cd "$_pkgname-$pkgver" || exit
#     python test/run.py
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}

Lucki avatar Sep 24 '20 19:09 Lucki

@Lucki, thank you for your hard work! I will leave this ticket open maybe someone volunteers to continue your effort.

ciur avatar Sep 26 '20 03:09 ciur

I might continue this but don't count on it just yet, since I have no experience with packaging yet and I also need to try papermerge first to see if I'm actually going to use it Anyway, thank you for leaving this here!

amo13 avatar Sep 26 '20 07:09 amo13

I'm coming from Paperwork and while I'm still happy with it I wanted to look into similar project with multi user capability.

So here's a list with things which aren't ideal IMHO:

  • In contrast to Paperwork or Docspell this uses a folder base approach and is really bad at it:
    • It's missing drag and drop or a context menu which adds five extra steps to every process. Automation helps here but you can't automate all of them in advance.
      • While cutting and moving pages also has the same problem it's nice to have it and I don't see another good way to do this. You don't see this feature that often.
    • Modal popup input fields don't get automatic text focus.
  • Having a multi user setup is pointless if only the superuser inbox is usable.
    • Fetching emails has to be a user setting.
    • Maybe watch subdirs for users?
  • Lack of feedback everywhere
    • Nothing happens when uploading finished.
    • Reloading the page and looking in the document is missing selectable text. Has processing failed or is it still on it? Do I have to retry? How do I let it process the document again? (It's possible to get some of the information in the log section but only if it was somewhat successful)
  • Scrolling documents to the bottom is a pain. The site has 4 (!) vertical and 3 horizontal scroll bars which activate one after another until you're finally at the bottom.
  • No fuzzy search. OCR often falsely recognizes characters or words or both and having a fuzzy search is crucial. A bit less than for Paperwork or Docspell because Papermerge uses folders but still important. (Search you demo for "Gesamtbetrag", which it indexed as "Gesantbetrag")
  • The right side menu (meta) is ugly compared to the other elements.
  • Automation based on pages will probably bite me later by ripping documents apart. Haven't tried that feature yet though.
  • Minor thing: No dark theme.

Lucki avatar Sep 27 '20 01:09 Lucki

Hi @Lucki,

thank you very much for your feedback. I really appreciate it, because I heavily rely on this sort of feedback to improve the application. I am sort of "blindly" in love with Papermerge, which makes me unable to see many of its problems. I will open a separate enhancement ticket to keep track of your remarks.

The only thing that I want comment is regarding this point:

In contrast to Paperwork or Docspell this uses a folder base approach

this is not true anymore (well, at least in master branch :) ). In couple of weeks I will release newer version which will introduce tag management (I realized just very recently how useful tags really are!).

ciur avatar Sep 27 '20 08:09 ciur

The apparent contra for paperwork ist that is appears to be only a desktop and no web based application. In that manner, it is simply not comparable with papermerge. Docspell though could actually inspire further ideas for papermerge I guess.

amo13 avatar Sep 27 '20 10:09 amo13

thanks @amo13, I will have a look at docspell project for inspiration.

ciur avatar Sep 29 '20 15:09 ciur

thanks @amo13, I will have a look at docspell project for inspiration.

Awesome! I am still trying to figure out which software I want to go for, considering papermerge, docspell and teedy, which appear to be the web-based candidates that are open-source and self-hostable.

About docspell: I like a lot the idea of having automatic correspondent identification and automatic tagging according to the content of a document. These would be awesome features in papermerge too. But I think I'm not going to install docspell because of the reliance on apache solr for full text search. I don't feel like spinning up and running this colossus just for full text search. About teedy: what draws me to teedy is the simple fact that the project is already 7 years old, which makes me hope that it will not get abandoned in a year or so. Unfortunately, the drawback of young and innovative projects is that a lot of them end up abandoned before having the chance to actually flourish...

amo13 avatar Sep 29 '20 15:09 amo13

Unfortunately, the drawback of young and innovative projects is that a lot of them end up abandoned before having the chance to actually flourish...

My plans with papermerge span over several decades. It takes time to build a good product.

ciur avatar Sep 30 '20 15:09 ciur

Though I don't know exactly how, the admin package seems to be better integrated now and does not need to be moved in place any more. Therefore, I guess, this issue can be closed.

amo13 avatar Mar 31 '21 11:03 amo13