TagStudioDev
Add poetry dependency manager and package builder
What is this?
This pull request adds poetry support. Poetry provides an easy way to install dependencies and to publish. This change was first suggested by #365 but I actually started working on it just before that issue was created.
As was stated in that issue:
Switching to poetry would simplify dependency and venv management, allowing new developers to get started with just two commands regardless of platform. Ideally be able to run
poetry installthen runpoetry run python tagstudio\tag_studio.pyand have the project just work.
I improved it to the point where you can get started with just this:
poetry install
poetry run tagstudio
That tagstudio run script also sets up a console_script. That basically allows you to start tagstudio by just calling tagstudio in a shell after installing with pip.
Using poetry you can also easily publish the package to pypi so that one can actually install using pip with:
poetry publish --username __token__ --password <api_token_generated_on_pypi>
Setting up publishing infrastructure
On this discord it was suggested to make the publishing infrastructure part of the same pr. Currently if you want to publish just adjust the version number and issue the comand above.
However, I think its nice to make a github action so that it publishes a new release when you make a github release. I found a pretty good looking github workflow here
I saw that on pypi that you can add github as a Trusted Publisher here. I assume this works by adding the workflow and then you can simply point pypi to that yml file.
But honestly I have not set that up before. For this reason I think this should just be a separate pr. Also because do we even want to use poetry for this? I think its a good idea because then dependency management and publishing is just one tool.
I suggest to just merge this and setup the automatic release to pypi (and apt) upload when you actually want to do a release.
Also do not forget that pypi has a test version where you can test out uploads without wasting a version number on the official one.
Things to know
- The CliDriver currently has many errors in it. For this reason I decided to not include it in the published package. That means the code is not there when you install it. To make that work I made each driver only be imported when you actually use it using the --ui option. So that means that --ui cli will not work because the code is not there. But it was not going to work anyways because the cli code currently does not work in the first place.
- To make the publishing work I had to make
tagstudio/resourcesan importable package otherwise I could not include it in the wheel. - The maintainers field in pyproject.toml is currently empty. But at some point that should probably be filled with the people that contributed code here
Let me know if you want me to change anything :)
Formatting diff:
diff --git a/tagstudio/__init__.py b/tagstudio/__init__.py
index df6a834..feefebb 100644
--- a/tagstudio/__init__.py
+++ b/tagstudio/__init__.py
@@ -1,3 +1,3 @@
-__version__ = '9.1.1'
+__version__ = "9.1.1"
__all__ = ("__version__",)
diff --git a/tagstudio/tag_studio.py b/tagstudio/tag_studio.py
index 7213914..d557d79 100644
--- a/tagstudio/tag_studio.py
+++ b/tagstudio/tag_studio.py
@@ -5,7 +5,10 @@
"""TagStudio launcher."""
import os, sys
-sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__)))) # add this so that `poetry run tagstudio` works
+
+sys.path.insert(
+ 0, os.path.abspath(os.path.join(os.path.dirname(__file__)))
+) # add this so that `poetry run tagstudio` works
from src.core.ts_core import TagStudioCore
import argparse
@@ -68,14 +71,17 @@ def main():
# Driver selection based on parameters.
if args.ui and args.ui == "qt":
from src.qt.ts_qt import QtDriver
+
driver = QtDriver(core, args)
ui_name = "Qt"
elif args.ui and args.ui == "cli":
from src.cli.ts_cli import CliDriver
+
driver = CliDriver(core, args)
ui_name = "CLI"
else:
from src.qt.ts_qt import QtDriver
+
driver = QtDriver(core, args)
ui_name = "Qt"
The version should match the current version of the program: 9.3.2 instead of 9.1.0/9.1.1. This incorrect version is also listed on the PyPI page.
I would also suggest updating the CONTRIBUTING.md with the new instructions for installing Poetry and launching TagStudio via it. I've also been meaning to update the CONTRIBUTING.md to list Python 3.11 as the minimum supported version, which would also need to be set here in [tool.poetry.dependencies].
@CyanVoxel Ok I will add instructions to CONTRIBUTING.md.
At the moment it says in pyproject.toml that we only support 3.12. Do you mean you actually (want to) support 3.11 as well?
Do you want me to also publish a version to pypi with this version number?
At the moment it says in pyproject.toml that we only support 3.12. Do you mean you actually (want to) support 3.11 as well?
Yes, the intention is to support 3.11 as well.
Do you want me to also publish a version to pypi with this version number?
I'm honestly not sure about publishing a new version to PyPI since it wasn't my intention to have the program on there in the first place at this time. I suppose if it's going to be there though then it might as well have the correct version number.
While this is also a high priority to get this squared away, I'll be prioritizing #332 over this as that adds additional dependencies and is a preexisting high priority PR
@CyanVoxel It is up to you but I think it is probably easier to just merge this and not connect it to #332. This pull request is for the main branch and not sql-bump like #332. After merging this you could merge main into sql bump when ready and just do poetry add <dependency> in sql-bump. So merging this does not affect #332. Additionally, it is not like the virtual environments you already have and use now are broken by merging this pull request you can just keep using them and switch over when you are ready. But switching over is not hard. If you want to try this out before merging it just install poetry and run poetry install --with dev and poetry run tagstudio. If it works I think this can just be merged. You can actually change your workflow when ready.
The point of this pull request is not to update the dependencies it is to add poetry, a system to make adding managing dependencies simpler and easier.
@CyanVoxel It is up to you but I think it is probably easier to just merge this and not connect it to #332. This pull request is for the main branch and not sql-bump like #332. After merging this you could merge main into sql bump when ready and just do
poetry add <dependency>in sql-bump. So merging this does not affect #332.
#332 is also targeting main; sql-bump is the name of yed's branch. #332 has already needed to continuously update for other changes going on in main and has been my priority to get shipped before this PR was opened.
Additionally, it is not like the virtual environments you already have and use now are broken by merging this pull request you can just keep using them and switch over when you are ready. But switching over is not hard. If you want to try this out before merging it just install poetry and run
poetry install --with devandpoetry run tagstudio. If it works I think this can just be merged. You can actually change your workflow when ready.
While it won't break the existing venvs we're using, it would still put these new Poetry responsibilities on #332 since it's updating the requirements.txt for main.
Besides the pre-existing priority for #332, merging this first would involve needing @yedpodtrzitko to switch over to the Poetry system mid-PR to update the changes and additions made to the requirements.txt and redo them the "Poetry way" for the sake of Poetry. If he chimed in here and said he doesn't mind then I'd be fine pulling this PR first, but for now I'd rather put it on this PR to adapt to the changed reqs over there once #332 gets pulled.
The point of this pull request is not to update the dependencies it is to add poetry, a system to make adding managing dependencies simpler and easier.
I personally don't find Poetry necessarily easier easier to work with compared to raw venvs, especially when I've had to install new package managers for a new package just to do what the venv and requirements.txt was already doing. I understand the simplicity if you're already used to the system and already have Poetry, but coming from it fresh it's just changed the way that things were being done without necessarily making them easier. That's my two cents, at least. Since this is here though and there's interest in it, plus it's not like creating venvs manually is something anyone likes, I still intend on pulling this in time.
#332 and it's predecessors, #190 and #187, have been constantly thwarted by the rest of the codebase updating quicker than they could keep up. The last thing I want to do is give more work to yed right now with a different dependency management system when it can be avoided by just waiting to pull this. Again if he's fine with it then it's all good, but that's my view on it as it stands.
Alright I understand. Thanks for expanding more on this for me.
I checked out the latest commit of #332 and if @yedpodtrzitko is fine with merging this pr earlier then here are the commands you would need to issue if #332 were to be merged with the dependencies it has right now:
poetry add structlog==24.2.0 SQLAlchemy==2.0.32
poetry add ruff==0.6.2 mypy==1.11.1 pytest-qt==4.4.0 --group dev
This will update poetry.lock and pyproject.toml correctly. Besides this and installing poetry you can just ignore it. Of course, wait if you want to rather wait!
tl;dr, I have just a few questions
-
why Poetry specifically? I'm not that much involved in things, but I'd assume there will be some discussion upfront to make a consensus what is the best/right tool to use. I'm not gonna just randomly bring up other packaging like "you should use pdm/hatch instead!" now, because they're moreless equal with Poetry, but uv feels like a way better choice for multiple reasons (speed, fix for #293, easy of installation, virtualenv handling etc. )
-
if we're doing Poetry (or any tool in general), could you please open parallel PR to my branch #332, to have it merged there as well instead of having to deal with the conflicts and "you just need to run this command and this command"? Thanks.
I looked at uv and it looks quite nice I have not heard of it before. I do not see the publishing feature that poetry has however. The rest of the things you mention, easy of installation, virtualenv handling etc I think are all the same. But uv claims to be a bit faster indeed. As for ease of install Poetry just recommends to install with pipx but there are also separate install scripts or you can install it with pip so I think that is the same as well. As for #293 I don't see how poetry would also not fix that. Given the similarities I think it matters more that there is a pull request already to add poetry and not one to add uv. But I do agree there could have been more discussion about this. But I saw multiple people talking about poetry and I was familiar with it so I thought I would add it #365. I will make a pr to your branch.
I looked at uv and it looks quite nice I have not heard of it before. I do not see the publishing feature that poetry has however.
I dont want to disregard the effort you put into this, but why does PyPI publishing matter? Normal users will download the executable package. I dont see the use-case for installing python, setting up virtualenv just so I can install it from PyPI and run in awkward way from shell... afaik nobody runs (nor publish) desktop apps this way.
So if the reason to choose Poetry (over uv) is because it has a feature which is not necessary / relevant for the project (as @CyanVoxel mentioned his stance above), then is that still the best option?
I don't think PyPi publishing is particularly important in this case I just mentioned it as something uv doesn't have. I understand that most normal people will just download an executable.
I did a rough test using the time command.
Installing dependencies for the first time.
🯆 poetry install
real 0m18,636s
user 0m18,127s
sys 0m1,521s
🯆 time uv pip sync requirements.txt
real 0m18,684s
user 0m5,166s
sys 0m2,593s
Its basically the same. I gave uv leeway here because creating the venv is a seperate command in uv (not included in the time) but with poetry that was also included in the time.
Running poetry install again uv when nothing needs to happen:
🯆 time uv pip sync requirements.txt
real 0m0,039s
user 0m0,032s
sys 0m0,046s
$ time poetry instal
real 0m0,995s
user 0m0,917s
sys 0m0,078s
Adding a dependency:
🯆 time poetry add tqdm
real 0m2,273s
user 0m1,929s
sys 0m0,114s
🯆 time uv add tqdm
real 0m0,215s
user 0m0,101s
sys 0m0,036s
The initial install which is the most important has no difference. Besides that uv is unsurprisingly a bit faster. Fast enough to disregard the work here and redo it? That is for @CyanVoxel to decide.
I personally think poetry is good and I also like the cli interface more. But I am of course biased. Also do not forget that other people also mentioned poetry independently before I made this. But of course they might not have known about uv either.
Looking forward to this, Poetry is also currently the best tool for packaging Python into Nix packages. In my attempts I also came across a couple of things and I have also gone over this pull request a bit. Maybe I can share one or two things here.
The PySide6 Python package officially states: "Please note: this wheel is an alias to other two wheels PySide6_Essentials and PySide6_Addons, which contains a predefined list of Qt Modules."
I saw something similar again in an info about the package split into Essentials and Addons, but I can't find it anymore.
PySide6 = "6.7.2"
# PySide6 includes PySide6_Essentials and PySide6_Addons
# PySide6_Addons= "6.7.2"
# PySide6_Essentials = "6.7.2"
A single tagstudio/__init__.py should be enough to continue supporting the previous python ./tagstudio/tag_studio.py and poetry run tagstudio at the same time.
The following content in tagstudio/__init__.py should allow importing local modules. See the reference in the comment for an explanation.
# allow local module files as import
# https://github.com/python-poetry/poetry/issues/3928#issuecomment-1399313433
import os, sys; sys.path.append(os.path.dirname(os.path.realpath(__file__)))
No need for multiple new __init__.py spread across the whole project.
An alternative would be using explicit (via . and ..) or absolute imports.
Explicit imports example for tagstudio/src/qt/ts_qt.py
from ..core.enums import SettingItems, SearchMode
from ..core.library import ItemType
from ..core.ts_core import TagStudioCore
from ..core.constants import (
COLLAGE_FOLDER_NAME,
BACKUP_FOLDER_NAME,
TS_FOLDER_NAME,
VERSION_BRANCH,
VERSION,
TAG_FAVORITE,
TAG_ARCHIVED,
)
from ..core.utils.web import strip_web_protocol
from .flowlayout import FlowLayout
from .main_window import Ui_MainWindow
from .helpers.function_iterator import FunctionIterator
from .helpers.custom_runnable import CustomRunnable
from .resource_manager import ResourceManager
from .widgets.collage_icon import CollageIconRenderer
from .widgets.panel import PanelModal
from .widgets.thumb_renderer import ThumbRenderer
from .widgets.progress import ProgressWidget
from .widgets.preview_panel import PreviewPanel
from .widgets.item_thumb import ItemThumb
from .modals.build_tag import BuildTagPanel
from .modals.tag_database import TagDatabasePanel
from .modals.file_extension import FileExtensionModal
from .modals.fix_unlinked import FixUnlinkedEntriesModal
from .modals.fix_dupes import FixDupeFilesModal
from .modals.folders_to_tags import FoldersToTagsModal
Absolute imports example for tagstudio/src/qt/ts_qt.py
from tagstudio.src.core.enums import SettingItems, SearchMode
from tagstudio.src.core.library import ItemType
from tagstudio.src.core.ts_core import TagStudioCore
from tagstudio.src.core.constants import (
COLLAGE_FOLDER_NAME,
BACKUP_FOLDER_NAME,
TS_FOLDER_NAME,
VERSION_BRANCH,
VERSION,
TAG_FAVORITE,
TAG_ARCHIVED,
)
from tagstudio.src.core.utils.web import strip_web_protocol
from tagstudio.src.qt.flowlayout import FlowLayout
from tagstudio.src.qt.main_window import Ui_MainWindow
from tagstudio.src.qt.helpers.function_iterator import FunctionIterator
from tagstudio.src.qt.helpers.custom_runnable import CustomRunnable
from tagstudio.src.qt.resource_manager import ResourceManager
from tagstudio.src.qt.widgets.collage_icon import CollageIconRenderer
from tagstudio.src.qt.widgets.panel import PanelModal
from tagstudio.src.qt.widgets.thumb_renderer import ThumbRenderer
from tagstudio.src.qt.widgets.progress import ProgressWidget
from tagstudio.src.qt.widgets.preview_panel import PreviewPanel
from tagstudio.src.qt.widgets.item_thumb import ItemThumb
from tagstudio.src.qt.modals.build_tag import BuildTagPanel
from tagstudio.src.qt.modals.tag_database import TagDatabasePanel
from tagstudio.src.qt.modals.file_extension import FileExtensionModal
from tagstudio.src.qt.modals.fix_unlinked import FixUnlinkedEntriesModal
from tagstudio.src.qt.modals.fix_dupes import FixDupeFilesModal
from tagstudio.src.qt.modals.folders_to_tags import FoldersToTagsModal
After changing the imports, the above code in __init__.py would no longer be necessary.
Additional installed modules would not require any changes to the import lines, only those from the local project.
Changing the imports in the long run might be a good idea anyway, without having to rely on the workaround via __init__.py for current Python versions and tooling in the long term.
Calling the program via Python would then change from python ./tagstudio/tag_studio.py to python -m tagstudio.tag_studio, calling the application as a module.
nix run and poetry run tagstudio would be called like before.
And at least for packaging in Nix, the following packages block in the pyproject.toml would have been enough for me. It even works completely without it, which is why I left it deactivated again.
# packages = [
# { include = "tagstudio", from = "." },
# ]
Since it is only this one module and the pyproject.toml is located directly in the project, it should work largely out of the box.
Nix Flakes are (usually) pure by nature.
If I run the application in a devShell, it starts fine. However, if I run it directly from GitHub with nix run github:zierf/TagStudio/poetry2nix (and I am no longer in a devShell!) or let it install directly into my system, I get the following error.
warning: Git tree '/home/zierf/workspaces/Python/TagStudio' is dirty
Traceback (most recent call last):
File "/nix/store/k98sinf1gjvbyld65nmqm5zly4lyggrw-python3.12-tagstudio-9.3.2/bin/..tagstudio-wrapped-wrapped", line 6, in <module>
from tagstudio.tag_studio import main
File "/nix/store/k98sinf1gjvbyld65nmqm5zly4lyggrw-python3.12-tagstudio-9.3.2/lib/python3.12/site-packages/tagstudio/tag_studio.py", line 8, in <module>
from .src.cli.ts_cli import CliDriver # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/k98sinf1gjvbyld65nmqm5zly4lyggrw-python3.12-tagstudio-9.3.2/lib/python3.12/site-packages/tagstudio/src/cli/ts_cli.py", line 32, in <module>
from ..qt.helpers.file_opener import open_file
File "/nix/store/k98sinf1gjvbyld65nmqm5zly4lyggrw-python3.12-tagstudio-9.3.2/lib/python3.12/site-packages/tagstudio/src/qt/helpers/file_opener.py", line 13, in <module>
from PySide6.QtWidgets import QLabel
ModuleNotFoundError: No module named 'PySide6.QtWidgets'
It is therefore worth trying out whether the packaged application also works on another system without a QT environment.
Thanks for your comments!
As for specific versions, I created this pr to add poetry not to change versions of dependencies. I suggest you make a new pr to change the PySide6 dependency. But probably after this has merged.
I don't really see the problem of having multiple init.py. But moving the code that adds the import path there is a good idea!
Iirc @CyanVoxel mentioned he did not want to do local imports or prepend the package name. It also says it here: https://github.com/TagStudioDev/TagStudio/blob/main/CONTRIBUTING.md#implementations
I agree though having dots or module name imports would be better. I would suggest using the dots. Al tough it does look a little weird if you see it for the first time. But in my opinion: yes it would be better to change the imports. But again, that is not the point of this pr. I just want to get poetry working. Another pr can try to change the imports. I didn't want to do this because it would mean that this pr would touch much more files.
It is therefore worth trying out whether the packaged application also works on another system without a QT environment.
But without QT it won't work anyways right?
Just wanted to chime in as a python developer that has built and maintained several python packages and applications. A pyproject.toml should definitely be the place to list dependencies, as it's the current standard for all python packages. This is preferred over a requirements.txt and manual invocation of pip. requirements.txt has fallen out of favor for all but the simplest of scripts and one-off code. For a full python application, all dependencies and versions, as well as the definitive application version, and all package metadata should be in the pyproject.toml.
Even if end users download the binary releases as a Windows desktop application and this is never published to pypi, for developers it's good to have a python package definition.
Poetry can be useful, but setuptools is also a perfectly capable build system, and can be simpler and easier to use for some projects. It also has the advantage that developers aren't forced to use a specific tool. They can choose to pip install -e . into any virtual environment of their choice (managed by their editor/ide, uv, venv, virtualenv, pipx, etc, etc).
Here is the guide on setuptools https://setuptools.pypa.io/en/latest/userguide/quickstart.html
I would also opt for not having a requirement.txt at all if that was agreed upon. I like poetry because who likes making virtual enviroments.
I would also opt for not having a requirement.txt at all if that was agreed upon. But I like poetry because who likes making virtual enviroments.
I would prefer to keep the requirement*.txt files. I personally don't see the need for an external dependency manager, and would rather keep managing my own virtual envs as it works better for my workflows.
https://xkcd.com/1172/
But yeah no reason to get rid of the requirements.txt, but a dependency manager that makes the venv and everything for you is also nice.
Why keep the requirements.txt around? You don't need it anymore even for setuptools, which has support for using pyproject.toml for all dependencies.
Will this change be implemented? This pull request actually seemed to be finished already. According to the checks, there is only something left in the linter (the logs have already been cleaned up).
If this is merged, Issue #200 could also be implemented. At least after I have incorporated the more than 238 commits that have been added in the meantime into my fork. 😁
Without this PR and Poetry, poetry2nix will not be able to create an executable version for NixOS.
At the time there was a bit too much bike sheading and people also wanting to update dependencies in this pr so I just gave up.
I can rebase and run the linter.
Ok I merged main and updated the dependencies to the current ones. I fixed the formatting thing. poetry run tagstudio works for me so it can now be merged. The test pass, the type checker doesn't find problems.
I've been reading this thread up and down, and there doesn't seem to be a compelling reason to use Poetry. PyPi publishing does not matter for us as an application, and using it for the sake of Nix is not enough to warrant an overhaul that affects everyone. Even with that aside, poetry2nix has been in an unmaintained status since November 2024. There are some people doing the occasional pull, but it is not a project I would rely upon; it seems to be treated as something receiving support for the sake of eventually transitioning away.
If we are to switch dependency manager in the future, it will most likely be to uv, stated above, in which I have done some tinkering with uv2nix as an experiment (even though Nix support shouldn't be the reason, it is being considered). Please, do not take this as guidance to make a PR adding in uv. Simply put, there are better things to address at the moment for the project, and a change in package manager would be done at a “quieter” time. When the time comes, I won't mind putting motion into it.
Last note about Nix: I have been revamping the flake and modules as of recently, currently with a shell using a virtualenv, redone with some old dependencies trimmed away. It also accomplishes only wrapping the Python interpreter with LD_LIBRARY_PATH, something I failed to do with the old devenv implemtation (yes, we are switching away from devenv, it made sense at the time). My current task now is packaging TagStudio and the relevant Python dependencies. I will make an issue to track all of this in the morning, so some scattered Nix issues can be “bundled” in one place.
I dont know or really care about nix. I fail to see how merging this pr would affect everyone. If you want to keep using requirements.txt then you can. By merging this pr you just give devs the option to use poetry.
Also I think pypi publising does matter. Why wouldn't you make tagstudio pip installable? It's already setup and it would make tagstudio available to everyone with pip installed which is a lot of people. The name on pypi has been claimed as I was the one who orginally claimed it for CyanVoxel.
#844, which has just been merged, has just significantly refactored the project layout to better align with the "src-layout" structure that was intended from the beginning of this project. Along with this layout change has come several other project-level changes that have been made easier thanks to this corrected layout.
This includes moving all dependencies to the pyproject.toml file and removing the requirements.txt and requirements-dev.txt files. This was done in a dependency manager agnostic way, which was one of my biggest issues with the approach here. I did not want the overhead of having to maintain separate ways of installing packages that included the old method plus a poetry-specific method with all of the numerous [tool.poetry] entries in the pyproject.toml. I did not wish to use Poetry as I found it more cumbersome to setup and interface with than a bare venv or an alternative dependency manager such as uv. #844 however circumvents all of this by allowing the dependencies to be declared once and are made installable with either a classic venv + pip approach, uv, or Poetry 2.0 since that release fixes a lot of my gripes with the large amount of specialized configuration. Now that Poetry supports PEP 621 I believe everyone should be able to use whichever dependency management solution they prefer without any undue specialized maintenance required on my part.
I believe these changes satisfy the original request in #365 and render this PR as no longer necessary. Most of, if not all the goals set out by this PR should now be accomplishable via the changes in #844. For reasons mentioned previously and reiterated now, I do not plan on supporting Poetry 1.x as it would be too cumbersome and increasingly unnecessary as people upgrade to 2.0. PyPI may be explored in the future, but is not something we're sure of or eager to do at the moment.
Lastly, the installation documentation has been greatly revamped and the setup instructions that were previously in the CONTRIBUTING.md have now been now mostly transferred over to the docs site. This includes instructions for various dependency managers, including Poetry.
Please let me know if everything has been addressed and this PR can be safely closed. If you have any requests that weren't addressed by either #844 or related to my comments about increased maintenance overhead, then they are probably best served in a new feature request. If you notice any mistakes in our updated documentation, especially related to Poetry, then you are more than welcome to submit a new PR with any corrections.
Sounds good 👍
By now I have actually also moved to uv after learning about it from this pr.
I would not forget about pypi publishing. Having tagstudio pip installable with pip install tagstudio is nice. I made #851 to remember.