marcel marcel startup fails

On Ubuntu 22.04 with Python 3.11.6 in a virtual environment:

python3 -mvenv venv
. ./venv/bin/activate
pip install marcel

$ marcel

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 317, in <module>
    main()
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 293, in main
    MAIN = Main(None, same_process=False, old_namespace=old_namespace)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 107, in __init__
    self.initialize_reader()  # Sets self.reader
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 177, in initialize_reader
    self.reader = Reader(self.env, self.env.locations.history_path())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 58, in __init__
    super().__init__(history_file=history_file)
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/multilinereader.py", line 46, in __init__
    self._fix_history()
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/multilinereader.py", line 115, in _fix_history
    readline.read_history_file(self.history_file)
OSError: [Errno 22] Invalid argument

Oct 29 '23 11:10 nkabir

I think I understand what's going on here. Starting marcel for the first time, the history file is not present. This file is stored $XDG_DATA_HOME/.local/share/marcel/marcel/history. If $XDG_DATA_HOME is not defined, ~ is used insted. marcel should try to create .local/share if they don't exist. That code appears to have a bug.

However, the failure I'm seeing is different from the one you are seeing. Another oddity is that the line numbers in main.py don't match my source.

So there are a few oddities. I'm going to fix the path creation bug. Let's see if that gets you going.

Oct 29 '23 15:10 geophile

I think I have a fix. I think I also understand why I'm seeing a different failure than you are, and it has to do with the source discrepancy. It looks like you have a version of marcel from Sept. 7 or earlier. Can you please look at marcel/version.py and tell me what the value of VERSION is?

Oct 29 '23 16:10 geophile

More research: It appears that the code in PyPi is an odd mixture of current (VERSION=0.18.3) and older versions, e.g. main.py and history.py. Investigating ...

Oct 29 '23 16:10 geophile

I think I have a fix. I think I also understand why I'm seeing a different failure than you are, and it has to do with the source discrepancy. It looks like you have a version of marcel from Sept. 7 or earlier. Can you please look at marcel/version.py and tell me what the value of VERSION is?

# This file is part of Marcel.
# 
# Marcel is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or at your
# option) any later version.
# 
# Marcel is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
# 
# You should have received a copy of the GNU General Public License
# along with Marcel.  If not, see <https://www.gnu.org/licenses/>.

VERSION = '0.18.3'

Oct 29 '23 16:10 nkabir

OK, it looks like there is a problem in installing to PyPi. My code and dist directory are up to date, but the code in PyPi is broken, I just downloaded and verified that. Working on it ....

Oct 29 '23 16:10 geophile

I have installed 0.18.4, which looks like it has current versions of main.py and history.py. And this should have a fix for the issue you reported. At least, it fixed my version of that problem. Can you please uninstall marcel 0.18.3 and try 0.18.4?

Oct 29 '23 17:10 geophile

Thanks for investigating! It generates a similar (same?) error with 0.18.4

$ ./venv/bin/pip install marcel==0.18.4
Looking in indexes: http://10.1.200.100:8081/repository/pypi-all/simple
Collecting marcel==0.18.4
  Downloading http://10.1.200.100:8081/repository/pypi-all/packages/marcel/0.18.4/marcel-0.18.4-py3-none-any.whl (292 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.8/292.8 kB 10.7 MB/s eta 0:00:00
Requirement already satisfied: dill in ./venv/lib/python3.11/site-packages (from marcel==0.18.4) (0.3.7)
Requirement already satisfied: psutil in ./venv/lib/python3.11/site-packages (from marcel==0.18.4) (5.9.6)
Installing collected packages: marcel
Successfully installed marcel-0.18.4

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
nkabirlabkeyx-dev~$ marcel
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 314, in <module>
    main()
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 290, in main
    MAIN = Main(None, same_process=False, old_namespace=old_namespace)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 107, in __init__
    self.initialize_reader()  # Sets self.reader
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 177, in initialize_reader
    self.reader = Reader(self.env, self.env.locations.history_path())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/main.py", line 58, in __init__
    super().__init__(history_file=history_file)
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/multilinereader.py", line 46, in __init__
    self._fix_history()
  File "/home/nkabir/venv/lib/python3.11/site-packages/marcel/multilinereader.py", line 115, in _fix_history
    readline.read_history_file(self.history_file)
OSError: [Errno 22] Invalid argument

Oct 29 '23 21:10 nkabir

Hmm. This is not happening for me. I tried using the venv I set up previously, and with a brand new user, and in both cases, marcel 0.18.4 started without error and seemed to be working correctly.

Can you please show me the output you get for this:

cd ~
ls -ld .local .local/share .local/share/marcel .local/share/marcel/history

Oct 29 '23 22:10 geophile

Note that I'm running in a Docker container with linux homebrew. It work with the default Python installation.

/home/linuxbrew/.linuxbrew/bin/python3
$ python3 --version
Python 3.11.6

$ ls -ld .local .local/share .local/share/marcel .local/share/marcel/history
drwxr-xr-x 1 nkabir nkabir 4096 Oct 22 11:33 .local
drwxr-xr-x 1 nkabir nkabir 4096 Oct 29 19:36 .local/share
drwxr-xr-x 2 nkabir nkabir 4096 Oct 29 19:36 .local/share/marcel
-rw-r--r-- 1 nkabir nkabir    0 Oct 29 19:36 .local/share/marcel/history

Oct 30 '23 00:10 nkabir

I can't think of a reason that that container would make a difference.

Can you try an experiment? What happens when you start python3 from the console, and do this (from your home directory):

import readline readline.read_history_file('.local/share/marcel/history')

I'm wondering what this happens in your venv, and with your default python installation. I'm guessing that from venv, you will see OSError, and that from default python it will return without error.

Oct 30 '23 01:10 geophile

$ ls -ld .local .local/share .local/share/marcel .local/share/marcel/history
drwxr-xr-x 1 nkabir nkabir 4096 Oct 22 11:33 .local
drwxr-xr-x 1 nkabir nkabir 4096 Oct 29 20:28 .local/share
drwxr-xr-x 2 nkabir nkabir 4096 Oct 29 20:28 .local/share/marcel
-rw-r--r-- 1 nkabir nkabir    0 Oct 29 20:28 .local/share/marcel/history

$ python3
Python 3.11.6 (main, Oct  2 2023, 13:45:54) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> readline.read_history_file('.local/share/marcel/history')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument

```

Oct 30 '23 01:10 nkabir

That's with venv, right?

I don't think it's a permission problem, because permissions look ok, and if you chmod 000, for example, you'd get a different error.

We could go back and forth here, but I'm wondering if this is happening in a place that I could login and debug.

For now, other things to try:

Dows the read_history_file call work on an empty file on your home directory?
What about a non-empty file (try a short file containing text).
what version of the readline library do you have? (There is a .so underlying the Python readline module.)

Oct 30 '23 01:10 geophile

Thinking about this some more: It's pretty clear that this is a subtle problem with libreadline.so. I'm not able to reproduce this problem. My OS is Pop OS 22.04 (an Ubuntu derivative), and my readline is this:

jao@loon:~/.local/share/marcel$ ls -l /usr/lib/x86_64-linux-gnu/libreadline.so.8.1
-rw-r--r-- 1 root root 335936 Jan  6  2022 /usr/lib/x86_64-linux-gnu/libreadline.so.8.1

I do remember having a very different problem with readline once. Line wrapping was broken, and I eventually traced the problem to libreadline.so. I upgraded it and that solved the problem.

In the environments that you have run marcel, what is the version of libreadline.so do you have? I am wondering if you have the SAME version in both environments, producing different results (OSError with errno 22 in one case, working correctly in the other).

Oct 30 '23 13:10 geophile

Let me put together a docker container with my setup for you to check.


$ ls -l /usr/lib/x86_64-linux-gnu/libreadline.so.8.1 
-rw-r--r-- 1 root root 335936 Jan  6  2022 /usr/lib/x86_64-linux-gnu/libreadline.so.8.1

Package: libreadline8
Architecture: amd64
Version: 8.1.2-1
Multi-Arch: same
Priority: important
Section: libs
Source: readline
Origin: Ubuntu
Maintainer: Ubuntu Developers <[email protected]>
Original-Maintainer: Matthias Klose <[email protected]>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 461
Depends: readline-common, libc6 (>= 2.33), libtinfo6 (>= 6)
Filename: pool/main/r/readline/libreadline8_8.1.2-1_amd64.deb
Size: 153210
MD5sum: 5c6ea4b0f2a30fc0054fb8e6b2b65b82
SHA1: 674a564c77e3c831875121d938b5ce704d11681b
SHA256: 51fd16f608efa710c987e4b071ffba0bec63a9ac8e55eddd5770f5e5252edc09
SHA512: fa7cd79731d535294ec3e0e03f8589d6ae434498c9fb76e2ca7fb9bb19f15b9a9c2717747f410580720e7d596aeb75c90b3a08c45e7c018a8e27be6915530d4b
Description: GNU readline and history libraries, run-time libraries
Description-md5: 02109af5819248ca867891090079f329
Task: minimal, server-minimal

Oct 30 '23 16:10 nkabir