`pythonista>=3.4`/`py>=3.10`: Refactoring, removing deprecated requirements, updating for

[o-murphy]

[!IMPORTANT] This PR does not offer backward compatibility with py2! However, it resolves problems for many commands. Possibly we have to remove py2 support

Install to test using this command

import requests as r; exec(r.get('https://gist.githubusercontent.com/o-murphy/ad7bcee38551d24ec172032792a2bf8f/raw/158346c958601cc29a9cc8070d6e501111562e52/get-stash-py3.10-beta.py').content)

WIP: updating to py3, removing deprecated requirements

• [x] Completely removed Python 2 support. StaSh now uses only Python 3 and compatible with Pythonista 3.4 or higher.
  • [x] Removed all Python 2 patches
  • [x] Removed deprecated wheels support. StaSh now uses a common pip approach for installing Python wheels
  • [x] The StaSh python alias has been set to python3 since Python 2 is no longer supported.
  • [x] The man page_5.txt file has been updated to reflect the end of Python 2 support.
  • [x] removed dependencies from six
  • [x] removed dependencies from imp (replaced with importlib)

build-system

• [x] switched to pyproject.toml
• [ ] make it installable from pip ?
• [x] move entry to __main__.py https://github.com/o-murphy/stash/pull/1

features

• [x] register editor action https://github.com/o-murphy/stash/pull/2
• [x] add support of site-packages/bin console-scripts https://github.com/o-murphy/stash/pull/3

Updating core

Module	State	Tested	Comment
core	:green_circle: updated
launch_stash	:green_circle: updated
getstash	:green_circle: actual		just a wrapper around __main__ https://github.com/o-murphy/stash/pull/1
__main__.py	:green_circle: new		main entry point

Updating system/

Module	State	Tested	Comment
dummyconsole	:green_circle: actual
dummyobjc_util	:green_circle: actual
shcommon	:green_circle: updated
shhistory	:green_circle: actual
shio	:green_circle: actual
shiowrapper	:green_circle: updated
shparsers	:green_circle: updated
shruntime	:green_circle: updated
shscreens	:green_circle: updated
shstreams	:green_circle: updated
shthreads	:green_circle: actual
shureactionproxy	:green_circle: actual
shui/__init__	:green_circle: updated
shui/base	:green_circle: updated
shui/dummyui	:green_circle: actual
shui/pythonistaui	:green_circle: actual
shui/stubui	:green_circle: actual
shui/tkui	:green_circle: updated

Updating lib/

• [ ] git
• [x] mlpatches (not-tested)
• [x] stashutils (not-tested)
  • [x] fsi
  • [x] wheels
  • [x] wakeonlan
• [x] libcompleter
• [x] libcore
• [x] libdist
• [x] libversion
• [x] pythonista_add_action

Updating bin/

Command	State	Tested	Comment
alias	:green_circle: actual	:green_circle: (manually)	Can be refactored
cat	:green_circle: actual	:green_circle: (manually)	Can be refactored
cd	:green_circle: actual	:green_circle: (manually)	Can be refactored
clear	:green_circle: actual	:green_circle: (manually)	Can be refactored
cowsay	:green_circle: actual	:green_circle: (manually)	Can be refactored
cp	:green_circle: actual	:green_circle: (manually)	Can be refactored
crypt	:green_circle: updated	:green_circle: (manually)	using pyaes-whl
curl	:green_circle: updated	:green_circle: (manually)
cut	:green_circle: actual	:green_circle: (manually)
diff	:green_circle: updated	:green_circle: (manually)
dropbox_setup	:green_circle: actual	:red_circle:
du	:green_circle: actual	:green_circle: (manually)
easy_config	:green_circle: updated	:green_circle: (manually)	Fixed incomplete shortcut creation with pinstash command
echo	:green_circle: actual	:green_circle: (manually)
edit	:green_circle: actual	:green_circle: (manually)
exit	:green_circle: actual	:yellow_circle: (manually)	Seems like working
fg	:green_circle: actual	:red_circle:
find	:green_circle: actual	:green_circle: (manually)
ftpserver	:green_circle: actual	:red_circle:
gci	:green_circle: actual	:red_circle:
gh	:red_circle: unsupported	:red_circle:	lots of changes needed
git	:red_circle: unsupported	:red_circle:	lots of changes needed, possibly should use dulwich from pypi
grep	:green_circle: actual	:green_circle: (manually)
head	:green_circle: actual	:green_circle: (manually)
httpserver	:green_circle: updated	:green_circle: (manually)
jobs	:green_circle: undone	:red_circle:
kill	:green_circle: actual	:red_circle:
latte	:green_circle: actual	:red_circle:
ln	:green_circle: updated	:green_circle: (manually)
logout	:green_circle: actual	:red_circle:
ls	:green_circle: updated	:green_circle: (manually)
mail	:green_circle: updated	:red_circle:	Not works
man	:green_circle: actual	:green_circle: (manually)
mc	:green_circle: updated	:red_circle:
md5sum	:green_circle: updated	:green_circle: (manually/pytest)
mkdir	:green_circle: actual	:green_circle: (manually)
monkeylord	:green_circle: updated	:green_circle: (manually)
more	:green_circle: actual	:green_circle: (manually)
mount	:green_circle: updated	:red_circle:
mv	:green_circle: actual	:green_circle: (manually)
openin	:green_circle: actual	:green_circle: (manually)
pbcopy	:green_circle: actual	:green_circle: (manually)
pbpaste	:green_circle: updated	:green_circle: (manually)
ping	:green_circle: updated	:green_circle: (manually)
pinstash	:green_circle: new	:green_circle: (manually)
pip	:green_circle: updated	:green_circle: (manually)	uses cmmon pip installation from bootstrap.pypa.io
printenv	:green_circle: actual	:green_circle: (manually)
printhex	:green_circle: updated	:green_circle: (manually)
ptinstaller	:green_circle: actual	:red_circle:
pwd	:green_circle: actual	:green_circle: (manually)
python	:x: removed	:x:	Replaced with alias to python3
python3	:green_circle: updated	:green_circle: (manually)
quicklook	:green_circle: actual	:green_circle: (manually)
rm	:green_circle: updated	:green_circle: (manually)
rmdir	:green_circle: actual	:green_circle: (manually)
scp	:green_circle: updated	:red_circle:
selfupdate	:green_circle: actual	:green_circle: (manually)
sha1sum	:green_circle: updated	:green_circle: (manually/pytest)
sha256sum	:green_circle: updated	:green_circle: (manually/pytest)
sort	:green_circle: actual	:red_circle:
source	:green_circle: actual	:red_circle:
sqlite	:green_circle: actual	:red_circle:
ssh	:green_circle: updated	:red_circle:	resolved with monkeypatch
ssh-keygen	:green_circle: updated	:green_circle: (manually)	resolved with monkeypatch
stashconf	:green_circle: actual	:green_circle: (manually)
tail	:green_circle: updated	:green_circle: (manually)
tar	:green_circle: actual	:green_circle: (manually)
telnet	:green_circle: updated	:red_circle:	replacing telnetlib with standard-telnetlib
touch	:green_circle: actual	:green_circle: (manually)
tree	:green_circle: new	:green_circle: (manually)
umount	:green_circle: actual	:red_circle:
uniq	:green_circle: actual	:green_circle: (manually)
unzip	:green_circle: actual	:green_circle: (manually)
version	:green_circle: updated	:green_circle: (manually)
wc	:green_circle: actual	:green_circle: (manually)
webviewer	:green_circle: actual	:green_circle: (manually)
wget	:green_circle: updated	:green_circle: (manually)
whatis	:green_circle: updated	:green_circle: (manually)
which	:green_circle: actual	:green_circle: (manually)
wol	:green_circle: actual	:red_circle:
xargs	:green_circle: actual	:red_circle:
zip	:green_circle: actual	:green_circle: (manually)

Known issues:

• ImportError pwd not in Pythonista builtins
• AttributeError os.system()
• fsi require dropbox installed and it is not optional
• bin/telnet should be reimplemented with telnetlib3 or Exscript and asyncio in future
• It happen once in CI: FAILED tests/system/test_threads.py::ThreadsTests::test_101 - AssertionError: output not identical assert '[stash]$ sleeping ... 0\n[stash]$ sleeping ... 1\n' == '[stash]$ [stash]$ sleeping ... 0\nsleeping ... 1\n'
• Crypto is outdated #544
• Common pip can not build packages on installation, even if it pure-python. It allows only .whl. The possible fix for it is implementing subprocess patches for Py3
• Common pip can't remove entry-points from ~/Documents/bin on package uninstalling, it's due to absent possibility to add --prefix path to PYTHONPATH globally, so the packages tree is not match PEP-370

[cclauss]

Please rebase on the current master branch.

• #525
• #526

[o-murphy]

Please rebase on the current master branch.

• Add pre-commit to run linters, formatters, etc. on code changes #525
• ruff format #526

Done

[cclauss]

Can you please pull in the changes to

1. .github/workflows/check-code-and-run-tests.yml ("3.13") and
2. setup.py (standard-imghdr) in

• #527

[o-murphy]

Can you please pull in the changes to

1. .github/workflows/check-code-and-run-tests.yml ("3.13") and
2. setup.py (standard-imghdr) in

• Test on Python 3.13 #527

1. which changes do you want?
2. standard-imghdr - is not neccesary, i already removed all deps to it, i use mimetypes instead

[cclauss]

Please add import sys to bin/whatis.py

Error: bin/whatis.py:23:5: F821 Undefined name sys

[cclauss]

Do you use the tools uv or pipx or brew?

• #525

[o-murphy]

Do you use the tools uv or pipx or brew?

• Add pre-commit to run linters, formatters, etc. on code changes #525

I use uv + ruff + pytest

[o-murphy]

Please add import sys to bin/whatis.py

Error: bin/whatis.py:23:5: F821 Undefined name sys

Done

[cclauss]

uv is awesome! In the project root directory, please run:

uv tool install pre-commit
pre-commit install
pre-commit run --all-files

[o-murphy]

@cclauss for some reason stacks on tests/pbcopy_pbpaste/test_pbcopy_pbpaste.py::CopyPasteTests::test_pbpaste_help

but pbtest -h on pytonista works

[cclauss]

test_pbpaste_help passes.

I think it is getting hung up on the next file which is tests/pip/test_pip.py

EDIT: Confirmed at https://github.com/ywangd/stash/pull/524/commits

[o-murphy]

@cclauss Use pyproject.toml pytest addoptions instead of specifying it directly in CI

[o-murphy]

@cclauss please help me with this

FAILED tests/system/test_termemu.py::TermemuTests::test_203 - AssertionError: assert '[stash]$ The...             ' == '[stash]$ The...             '

[o-murphy]

@cclauss please help me with this

FAILED tests/system/test_termemu.py::TermemuTests::test_203 - AssertionError: assert '[stash]$ The...             ' == '[stash]$ The...             '

As I understand this test will pass depends on how some stream handle the "K" (erase_in_line) event., does it work on pythonista?

[o-murphy]

There is no pwd module in Pythonista 3.5, so I'm looking for stashutils.fsi.base fix approach https://github.com/omz/Pythonista-Issues/issues/782

[o-murphy]

Help wanted! 🙏

https://github.com/ywangd/stash/pull/524#issuecomment-3155227665

[cclauss]

https://github.com/ywangd/stash/blob/7e25b57ac2e5204dff91587bca56e1e7bf3ee60b/tests/stashtest.py#L21 You might try:

- ON_TRAVIS = "TRAVIS" in os.environ
+ ON_TRAVIS = "CI" in os.environ

• https://docs.travis-ci.com/user/environment-variables/#default-environment-variables
• https://docs.github.com/en/actions/reference/workflows-and-actions/variables#default-environment-variables

[bennr01]

Reviewing this PR will take some time (which is unfortunately rather limited on my end). Here are a couple short thoughts based on a first glance:

• Such a big PR should generally go into the dev branch first, in case there are any significant bugs. That way, we maintain a stable version while we can properly test out the changes.
• I am really skeptical about dropping pip from StaSh. The current version is well integrated into StaSh. Can the proper pip actually be executed from StaSh?
• If we remove pip (again, very skeptical about that), wheels.py should be removed too. It's only used for pip.
• launch_stash.py shouldn't have a if __name__ == "__main__" guard. It's generally a good idea to do this, but as that file does nothing but launch StaSh in the first place, there really isn't a feasible situation in which that file is executed and main() shouldn't be called.

It generally looks good, but as I said, I haven't had time for a full review. Thank you for taking your time and modernizing StaSh.

[cclauss]

The dev branch is a bad idea.

It is way behind master and its tests fail, and it is unintuitive to contribute to a branch other than the default branch.

If people are looking for a stable branch, they should look at the most current release.

[bennr01]

he dev branch is a bad idea. It is way behind master and it’s tests fail, and it is unintuitive to contribute to a branch other than the default branch.

It actually has a couple of fixes and features that still need to be ported into the master branch. Until a couple of days ago, master wasn't notable ahead. Still, if you think that we can not fix dev, then we can always nuke it and re-init from master.

Think about it in another way: there's a reason that dev became so bad. If we hadn't had the dev branch, then the contributions that broke that branch would have gone to the master branch and broken that one instead ;)

and it is unintuitive to contribute to a branch other than the default branch.

It's literally one of only two points under the contributing guide for StaSh... I sincerely hope that anyone that contributes something to any project at the very least skims through the README and looks if there are any headings about contributing the project.

I hope writing this didn't come across as harsh, which really wasn't my intent. I can understand forgetting this or missing that part of the README and making a couple of such small mistakes isn't anything bad. But I also think that we shouldn't call requiring contributors to skim through the README and then read the three lines about contributing as "unintuitive".

If people are looking for a stable branch, they should look at the most current release.

While that is a great idea for GitHub/GitLab/... projects in general, for this repository specifically it's not a good idea. Please keep in mind that the StaSh update system as well as the install procedure rely solely on branches. The releases are - in the context of StaSh - meaningless. They aren't used for anything but keeping track of progress. When a new user installs StaSh the intended and advertised way, the installer will install from the master branch. And when updating, the update system is also branch-based.

You are absolutely right that normally releases would be the way to go. But that is simply not how the current StaSh architecture is designed. If you want to rewrite the install and update system so that we default to releases, then please go ahead. A couple of years later, when we can be reasonably sure that anyone has updated to the new update system and will no longer rely on the master branch being stable, we can finally fix this behavior and change the contribution system to what you wrote.

But I hope that you agree that it is a bad idea to distribute unstable changes in such a way that our update and install system would wrongly believe them to be stable updates.

[cclauss]

It is unintuitive to install dev or master instead of installing releases like the vast majority of GitHub and Gitlab repos.

Tools like Dependabot, etc., work on the default branch, not the branch that is written in the README.

When people read the codebase to propose fixes, they read the default branch, not the branch in the README.

there's a reason that dev became so bad.

Yes. Probably because all of its GitHub Actions fail, and no one is seeing that on the non-default branch.

I will make the following updates to dev, and let's see if we can get it closer to master.

• #519
  • #535
• #525
  • #537
• #526
  • #536

[o-murphy]

Reviewing this PR will take some time (which is unfortunately rather limited on my end). Here are a couple short thoughts based on a first glance:

• Such a big PR should generally go into the dev branch first, in case there are any significant bugs. That way, we maintain a stable version while we can properly test out the changes.
• I am really skeptical about dropping pip from StaSh. The current version is well integrated into StaSh. Can the proper pip actually be executed from StaSh?
• If we remove pip (again, very skeptical about that), wheels.py should be removed too. It's only used for pip.
• launch_stash.py shouldn't have a if __name__ == "__main__" guard. It's generally a good idea to do this, but as that file does nothing but launch StaSh in the first place, there really isn't a feasible situation in which that file is executed and main() shouldn't be called.

It generally looks good, but as I said, I haven't had time for a full review. Thank you for taking your time and modernizing StaSh.

• stash installation via pip is just proposal, can be never implemented
• yes replacing stash's pip with the pypa/pip works much better, this is one of the first things I added to the PR, I testing it all time for few days and it's such good, and I haven't encountered any critical problems, and everything is easy to restore if things go wrong, if needed i will remove wheel cause pypa/pip handles it
• using if __name__ == "__main__" guard is common thing for executable python scripts, even if it never be used like this

[o-murphy]

@cclauss @bennr01

Starting from this point I propose you to merge it as dev or beta branch

The reasons is:

• PR such massive
• exiting tests passing (exept one that's not critical, which i ask you to help)
• most commands are manually tested
• updating some features / commands should be in other PR's
• adding new tests should be in other PR's
• fixing/updating features / commands that have deprecated/unsupported dependencies should be in other PR's

In the future, I will create a separate PR for all thematically related changes. I'm having a hard time keeping track of the individual changes here.

[cclauss]

I am reluctant to merge code with failing tests, so we should mark failing tests with @unittest.skip, skipIf, or skipUnless, or pytest xfail, etc. We should mark these tests with a TODO that points to an open issue to fix them.

• https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures
• https://docs.pytest.org/en/stable/how-to/skipping.html

[o-murphy]

I am reluctant to merge code with failing tests, so we should mark failing tests with @unittest.skip, skipIf, or skipUnless, or pytest xfail, etc. We should mark these tests with a TODO that points to an open issue to fix them.

Why don't you review this? I think the problem is in the test itself, the function seems to work as expected.

[o-murphy]

@cclauss Should I rebase it onto the dev?

[o-murphy]

I wanna completely cleanup project from py2 support (still have some refs to it). Can I add it to current PR or have I to wait for review?

[o-murphy]

@cclauss Should I rebase it onto the dev?

@bennr01 what do you think?

[o-murphy]

I wanna completely cleanup project from py2 support (still have some refs to it). Can I add it to current PR or have I to wait for review?

@cclauss I just created a new Removing deprecated py2 compatibility PR to the current PR branch that should completely remove still existing compatibility references to py2. I can merge it or have I wait

[bennr01]

I wanna completely cleanup project from py2 support (still have some refs to it). Can I add it to current PR or have I to wait for review?

Adding it to this PR should be fine. It's fits the intention of this PR.

@cclauss Should I rebase it onto the dev?

@bennr01 what do you think?

I'd personally prefer if these changes were to affect dev, but as @cclauss mentioned, our current dev branch is in a rather suboptimal state.

---

@cclaus wrote:

It is unintuitive to install dev or master instead of installing releases like the vast majority of GitHub and Gitlab repos.

Tools like Dependabot, etc., work on the default branch, not the branch that is written in the README.

Yeah, I can see where you are coming from. I'd change the default branch, but alas, it is not within my power...

Yes. Probably because all of its GitHub Actions fail, and no one is seeing that on the non-default branch.

I will make the following updates to dev, and let's see if we can get it closer to master.

Please do, that would be great. Thanks.