cosmopolitan
cosmopolitan copied to clipboard
jart
Compiling Python
~~https://github.com/ahgamut/python27~~
https://github.com/ahgamut/cpython/tree/cosmo_py27
The assert macro needs to be changed in cosmopolitan.h to enable compilation (see #138).
Afterwards, just clone the repo and run superconfigure.
Python 2.7.18 compiled seamlessly once I figured out how autoconf worked, and what flags were being fed to the source files when running make. I'm pretty sure we can compile any C-based extensions into python.exe -- they just need to compiled/linked with Cosmopolitan, with necessary glue code added to the Python source. For example, I was able to compile SQLite into python.exe to enable the internal _sqlite module.
The compiled APE is about 4.1MB with MODE=tiny (without any of the standard modules, the interpreter alone is around 1.6MB). Most of the modules in the stdlib compile without error. The _socketmodule (required for Python's simple HTTP server) doesn't compile, as it requires the structs from netdb.h.
On Windows, the APE exits immediately because the intertpreter is unable to find the platform-specific files.
Module/getpath.c and Lib/site.py in the Python source try to use absolute paths from the prefixes provided during compilation; Editing those files to search the right locations (possibly with some zipos magic) ought to fix this.
This is really exciting. Could you rebase your python27 repo on top of https://github.com/python/cpython so provenance is clearer and I can git diff exactly what you did? Alternatively, some kind of minimal build script showing the steps that are needed would be super helpful.
https://github.com/ahgamut/cpython/tree/cosmo_py27
Clone the repo and run superconfigure (superconfigure calls configure with the right params, then make and objcopy).
There are some minor details in the commit messages regarding what I tried to compile, etc.
Here's the sqlite fork that compiles with Cosmopolitan: https://github.com/ahgamut/sqlite/tree/cosmopolitan
Clone the repo and run superconfigure. It requires libtool and tcl8.6.
Changing the build process for SQLite was as follows:
- Added header stubs
- Created a
superconfigurescript to callconfigurewith the right parameters/flags - Changed
AC_CHECK_FUNC's implementation inconfigureso that it doesn't error when using Cosmopolitan - Changed Makefile target requirements to link Cosmopolitan at the end of necessary build steps
- Fixed name clashes - only one,
hiddenclashed with a char array - Fixed other compilation errors - errno constants (#134) was the only error, changed the switch to an if-else
sqlite compiles without any errors (only 1 warning). I haven't figured out how to run the tests yet.
Adding sqlite to the Python build requires the above compiled sqlite and adding the below recipe to Modules/Setup.local:
# example recipe with SQLite3
# set variables to be used in Makefile
*static*
# location of compiled https://github.com/ahgamut/sqlite
SQLITE3_DIR=../sqlite
# if there are compile-time flags with an equals sign
# set them within a string, otherwise written wrongly into the Makefile
SQLITE3_OMIT_EXTFLAG='SQLITE_OMIT_LOAD_EXTENSION=1'
SQLITE3_MOD='MODULE_NAME="sqlite3"'
# order is (module, sources, includes/defines, link locations, linked libs)
# read Modules/Setup.dist for more details
_sqlite3 _sqlite/util.c _sqlite/connection.c _sqlite/cursor.c \
_sqlite/microprotocols.c _sqlite/cache.c _sqlite/prepare_protocol.c \
_sqlite/row.c _sqlite/statement.c _sqlite/module.c \
-D$(SQLITE3_OMIT_EXTFLAG) -D$(SQLITE3_MOD) \
-IModules/_sqlite -I$(SQLITE3_DIR) \
-L$(SQLITE3_DIR)/.libs -lsqlite3
The python.com APE now opens on Windows!
The interpreter couldn't find the standard library because the paths were coded in as absolute paths at compile time. I changed that to use relative paths (i.e. Lib in the same directory as the interpreter). Now one can just copy python.com and the Lib folder to the same directory in a Windows machine, and the APE will find site.py and start up properly. Later it might be nice to move some of the core modules as .pyc files into a ZIP as part the APE.
Right now, running python.com yourfile.py works on Windows, but the interpreter keeps throwing syntax errors in interactive mode. It may be related to this.
Question: does Cosmopolitan handle paths (forward slash on Linux, backslash on Windows) and environment variables (separated by : on Linux, ';' on Windows) correctly?
Your syntax error in interactive mode might be similar to the errors I experienced in trying to get various shells to run under Windows. Specifically inside Windows console when you hit enter, it will send CRLF, but the interpreter is expecting only LF for end of line, so it interprets the CR as part of the statement. You can test this by hitting Ctrl-J instead of enter. If the syntax error goes away but your cursor ends up in a funny position, then that's the problem.
For path conversion, this is done inside mkntpath.c which should be called through the standard libc functions like stat and open, but you might find you are having a different problem with PYTHONPATH, which might parse the variable using colon as separator when compiled with Cosmopolitan. For libc functions that use PATH (e.g. execlp) this is handled in commandv.c, but that won't help for PYTHONPATH, or if Python includes its own path search logic. I'm not sure if Python dynamically sets the directory and path separators at runtime or if it's compiled in, but the ideal situation would be for it to determine these at runtime the same way that Cosmopolitan does, then everything should just work.
I haven't had much time to look at this project over the past few weeks, but if I do get back to it and find any Windows-specific quirks, I'll probably post over on #117 or open a PR. I expect any solutions will be similar for shells, Python interpreter etc.
The problem is with CRLF: I just tested statements terminating with Ctrl-J on Windows, and those are accepted (I still have to press Enter to run the statement, but at least the statement runs before showing invalid syntax).
PYTHONPATH isn't an issue because I unset it before running python.com.
Directory and path separators are set at compile time (DELIM and SEP in Include/osdefs.h).
Python needs to set sys.path (locations of import-able things), sys.prefix (location of platform-independent .py/.pyc files), and sys.exec_prefix (for shared libraries) to be able to import modules correctly. There are two separate sets of functions for the path search logic:
-
Modules/getpath.ccontains the Unix-related stuff, and -
PC/getpathp.ccontains the functions for DOS/Windows (but this is not compiled).
Both proceed similarly: check argv[0] to set the local directory, check environment variables (PATH, PYTHONPATH, PYTHONHOME, and some Windows registry stuff), try to find common locations for libraries from the directory of the executable, or finally fall back to the locations provided at compile time.
Right now, I've just changed the compile-time absolute path locations to relative paths, and it seems to work ok. Maybe after some reading I can customize Modules/getpath.c to have a IsWindows() check and change everything accordingly.
It's really annoying that POSIX doesn't define a function to search PATH, it seems every shell just reimplements it for itself, and then libc has its own way again for execlp etc. I think something that might make our lives easier is publishing a Cosmopolitan-blessed path searcher function, so given an environment variable name (or buf with the value already in it), search each path for a file underneath it in a platform-agnostic way. Something like the SearchPath function in commandv.c, but which works for other variables, and where you can toggle search for executables or just search for any file. That might be able to replace some of the functions that Python, ash etc are trying to use to find the right file to load (or autocomplete, or whatever).
The CRLF thing is a trickier problem. Something you can try is using mintty (easiest way is from Git Bash) as your "terminal" instead of Windows console. I'm not sure, but that might avoid the generation of a CRLF when hitting enter, which would at least be a temporary workaround. Solving the problem inside console is more challenging. Personally I don't see it as very useful that carriage return is parsed as a non-whitespace token, even on UNIX. It seems to me the cleanest solution would be for Python to ignore CR, or treat it the same as a trailing space. That's the approach I went with trying to get ash to work, but I'm not sure if it will have unintended consequences for files with binary data in them.
Maybe we could "polyfill" this one for UNIX: https://docs.microsoft.com/en-us/windows/win32/api/processenv/nf-processenv-searchpatha It could perhaps be used to handle finding things in the APE's ZIP filesystem too.
A quick list of the internal modules that can't be compiled yet (full list in the repo README):
- [x]
syslognow compiles with the latest commit - [x]
_sqlite3compiles if I havelibsqlite3.a - [x]
bz2compiles if I havelibbz2.a - [x]
mmap--mremaprequired an additionalvoid *parameter (new_address), currently passingNULL - [x]
_ctypes-- requiredlibffi.awhich compiles if you disable allpthreadandmntent-related stuff - [x]
readlinecompiles with some modifications tolibreadline.aandlibtermcap.a, but not particularly useful - [x]
_locale-- linker error searching forstrxfrm - [x]
_socket-- requireshostent,protoentand otherstructs fromnetdb.h - [ ]
crypt-- requires-lcrypt - [x]
_hashliband_sslrequire linking with OpenSSL (which requiresnetdb.hto compile)
_multiprocessing requires _save, which is a variable that is part of Py_BEGIN_ALLOW_THREADS.
Of the remaining modules, maybe dbm/gdbm would be useful to have.
The Python tokenizer now ignores CR when reading input. No more syntax errors when the APE runs in interactive mode on Windows!
We could generalize the commandv API but I suspect one of the reasons why POSIX doesn't do that already is that PATH searching is such an expensive operation (in terms of system calls and disk seeks) and shells usually implement it on their own because they're able to perform local optimizations (like memoization) that the C library isn't able to do. For example, sometimes when using the bash shell, I'll need to occasionally run hash -r to let it know to recompute PATH after it's been changed.
Maybe just skip looking through environment variables when starting up python.com? The interpreter looks through PATH (and also PYTHONPATH, PYTHONHOME) only because it needs to find the necessary standard library modules and .so/.dll shared libs. But that can be changed entirely or skipped: in this commit I've commented out the search for PYTHONPATH and PYTHONHOME, and I'm already using relative paths for the directories.
As a workaround for Cosmopolitan Python, you can build the APE version of the latest Wasm3 and run Rust Python WebAssembly interpreter in it:
$ ./wasm3.com --stack-size 1000000 rustpython.wasm
Welcome to the magnificent Rust Python 0.1.1 interpreter 😱 🖖
>>>>>
I think getting the _socket module to build is the only major thing left. It would enable testing the stdlib, and I could get started on a PR for third_party/python2.
I had a look at the netdb.h implementation in the musl source code:
- [x]
getaddrinfo,freeaddrinfo, andgai_strerrorare available in cosmopolitan - [x] using
musl'sgetnameinforequires many internal functions - [x]
hostent/netentand their related functions are stubs -- can implement or use frommusl - [x]
protoentdepends only onstrlen/strcmp-- can implement or use frommusl - [x]
serventdepends ongetnameinfoand an internal function__lookup_serv - [x]
hostentdepends ongetnameinfoand an internal function__lookup_name
@jart do you (plan to) have an implementation of the above functions in cosmopolitan? I tried to add just getnameinfo to third_party/musl, but then its internal dependencies added a bunch of other files, so I thought I'd check.
Contributions are welcome on getnameinfo. I'd write it from scratch rather than using the Musl code. We already have getaddrinfo so implementing getnameinfo would be almost the same thing, except you send the DNS request to aaa.bbb.ccc.ddd.in-addr.arpa. and parse the returned PTR record.
Until now, python.com required the standard library to be in a nearby folder.
-
Modules/getpath.cin the Python source buildssys.path, which is used by Python to find importable modules. - Cosmopolitan libc allows the APE to be a ZIP; I can add files using the
zipcommand (#166) - the Python interpreter loads
zipimportbefore any other module, to be able to load modules from ZIP files
I added the standard library to the internal ZIP store, and added the location of the APE as the first entry in sys.path. The python.com APE is now self-contained! (tested on Debian Linux and Windows 10)
- The self-contained
python.comis 13MB, because the ZIP store contains all.pyfiles. I expect that picking only the necessary parts of the stdlib and using.pycfiles will reduce the size - Currently
.pycfiles in the ZIP store causezipimportto give abad mtimeerror - Some issues with the APE being on
$PATH, but that's to be examined later.
Now all the functions related to the _socket module have been implemented (#172, #196, #200, #204, #207, and #209 -- thanks @jart for guidance!), I can:
- run
python.com -m testfor regression tests on the APE: 191 tests pass, but a lot of others fail, with weird side effects - serve static HTML pages in a local directory via
python.com -m SimpleHTTPServer - use
pipto install a local.whlfile to a given directory (see here, I haven't figured out SSL support for downloading wheels from the internet). - Add C extensions to the APE during compilation if you take the time: I got it to work with
greenlet, so I expect other simple C extensions to be similar
(Edit Not yet there on windows because _socket has some complaints)
https://github.com/ahgamut/cpython/tree/cosmo_py36 also works. Python 3.6.14 has another 5 months before EOL though.
@jart I was trying to python.com -m SimpleHTTPServer and it was failing with an Errno 10042 [ENOPROTOOPT] on Windows
- the function call is
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, buf, sizeof(buf)) - on Linux the above call becomes
setsockopt(fd, 1, 2, buf, sizeof(buf))and it works without error - on Windows the above call becomes
setsockopt(fd, 0xffff, 0, buf, sizeof(buf))
SO_REUSEADDR is defined as 0 for Windows in libc/sysv/consts.sh, should it be 1 instead?
The Win32 API docs say that:
SO_REUSEADDR: BOOL Allows the socket to be bound to an address that is already in use. For more information, see bind. Not applicable on ATM sockets.
I changed the setsockopt call to setsockopt(fd, SOL_SOCKET, IsWindows() ? 1 : SO_REUSEADDR, buf, sizeof(buf)) and I am able to run SimpleHTTPServer without error.
With the above SO_REUSEADDR fix, it is possible to serve static pages locally on Windows, using python.com -m SimpleHTTPServer.
It is also possible to serve dynamic pages: just download Flask and its pure-python dependencies as wheels, and unzip the wheels into the APE at python.com/Lib/site-packages. Here's a GIF of a simple Flask webapp that runs with such a python.com:
Tested on Windows 10 and Debian Linux. I wrote a summary of the changes made for the Python APE here.
@jart @pkulchenko I would like to know if it possible to use MbedTLS for the SSL support required in Python. Does MbedTLS have Python bindings? I don't think MbedTLS is a drop-in replacement for OpenSSL (like BoringSSL), is there is a list of equivalent functions somewhere?
python.com -m pip download/install <package-name> requires SSL support: the _ssl and _hashlib modules in stdlib needs to be compiled into the APE. Without SSL support, one needs to download all the necessary wheels locally before installing them with python.com -m pip install pkg.whl -t some_dir.
The _ssl and _hashlib modules are implemented with OpenSSL for both Python 2.7 and Python 3.6.
It is possible to compile OpenSSL 1.1.1k with Cosmopolitan, by providing the right flags and a few minor changes to the source code.
Compiling everything with -Os and using the MODE=tiny cosmopolitan.a, we get:
| component | size |
|---|---|
| APE + most C stdlib extensions | 2.6 MB |
unicodedata + CJK/multibytecodecs |
1.6 MB |
python stdlib as .pyc files1 |
2.6 MB |
pip + setuptools as .pyc files |
2.0 MB |
| total without SSL | 8.8 MB |
_ssl + _hashlib via OpenSSL |
2.2 MB |
| total with OpenSSL | 11 MB |
- OpenSSL increases the size of the APE almost as much as the zipped stdlib.
-
python.com -m pip download <package>works if the APE is linked with OpenSSL.python.com -m pip install <package-name>works if a target directory is given, but can't install packages directly into the APE's zip store (ETXTBSY, APE is not self-modifiable #166). - OpenSSL when using Cosmopolitan does not pass all the tests in its test suite (haven't examined why).
- PEP 543 aims for a unified Python TLS API that is not so dependent on OpenSSL, but it is for Python 3.7+, which will not compile with Cosmopolitan.
- Cosmopolitan supports MbedTLS (#179) and it passes all the tests included in the repo.
- There is a package called
python-mbedtlswhich supports Python 3.6: it wraps MbedTLS to provide (some?) cryptographic facilities, but the README explicitly says that this is not a drop-in replacement for the python stdlib. Also the package uses Cython.pyxfiles, so I'm not sure how that will compile in the APE.
1 this is by ignoring failing libraries like asyncio, tkinter, turtle.py, .exe files, some platform-specific stuff, etc. I imagine it's possible to reduce further if size is really an issue.
@jart @pkulchenko I would like to know if it possible to use MbedTLS for the SSL support required in Python. Does MbedTLS have Python bindings?
@ahgamut, I did find the same library, as you already referenced, but I can't comment on the rest, as I haven't used it.
@jart here's a quick summary of stuff to be examined further:
- missing implementation of
rewinddir, currently commented out; also affects PHP - missing
forkptyimplementation (declared intermios.h), currently commented out -
SO_REUSEADDRinsysv/consts.shshould be 1 for Windows? (see this comment) - Is SSL necessary? MbedTLS Python bindings vs OpenSSL compiled with Cosmopolitan (see this comment)
-
python27.comorpython36.com? 3.6 has more features and is EOL'd only at the end of this year -
python.comdoes not pass all provided benchmark tests yet (partial on Linux, untested on Windows) - does
python.comassumptions hold on Windows (os/pathlib/socketmodules)? - simple C extensions can be compiled with
python.comlike stdlib modules (see here). Is there a less involved way for this? Might be better to focus on #81 instead. - some form of tree-shaking for the Python stdlib so that APEs can be smaller
I'm still excited about porting Python and now have time available to help.
python27.com or python36.com? 3.6 has more features and is EOL'd only at the end of this year
Python3000 can no longer be safely ignored. I'd recommend we just do that unless there's big blockers. Or both. It'll make people unhappy if we publish only Python2. Speaking of which, I've decided that I do want to start distributing "Actually Portable Python". I've mentioned before, language authors should ideally incorporate Cosmo into their official release processes. Until that happens, we can demonstrate the demand exists by distributing ourselves.
MbedTLS
If you got OpenSSL to build then I'd say stick with that. I chose MbedTLS for redbean because I wanted something tinier and I wouldn't agree to the OpenSSL license. However Python is already huge and it appears OpenSSL finally fixed its license. So it looks good to me. I'd even support checking-in both OpenSSL and Python3 to third party.
FYI: Anything less than Python 3.5 are EOL now: https://devguide.python.org/#status-of-python-branches
I suggest targeting Python 3.6+, as you probably won't able to get help or any support for Python core developers for older versions.
@kissgyorgy
https://github.com/ahgamut/cpython/tree/cosmo_py27 -- Python 2.7.18 is EOL of course https://github.com/ahgamut/cpython/tree/cosmo_py36 -- Python 3.6.14 is supported till the end of this calendar year
Python 3.7 and above cannot be built without threads.
Both the above repos are roughly equal in terms of modifications/functionality. Python2.7 is easier to debug because Python3 is unicode by default, which raises complaints with locale/encoding stuff. I'm running the tests for Python 3.6 right now.
@jart I thought it would be better to add Python to the repo after passing as many tests as possible, like it was done in #61.
The Python 3.6.14 source contains a couple of its external dependencies: mpdecimal and libffi (for _ctypes). Split them into separate packages (third_party/libmpdecimal) or keep them within third_party/python36?
Okay the latest commit to cosmo_py36 adds a shell script to run a selected list of tests from the Python 3.6.14 test suite.
- ~~192~~ ~~220~~ 256 tests pass without complaint
- ~~108~~ ~~75~~ 36 tests fail. Common failure reasons include multithreading, ctypes, and locale/unicode encodings. I'll go through the verbose output to see if some testcases can be skipped.
- ~~104~~ ~~109~~ 112 tests are not considered because:
- they depend on multithreading
- they depend on external packages like libbz2, OpenSSL
- they error out
- they stall (possibly because I've changed the
threadingmodule to usedummy_thread, and the modules/tests that rely on this try to generate large numbers of threads)
- 3 tests do nothing
The test results differ if python.com is used instead of python.com.dbg. Maybe the APE ZIP store imports are interfering with the testing?
Raised the number of passing tests by commenting out tests related to handling various unicode encodings (the Cosmoplitan build currently supports only UTF-8).
75% of applicable tests passing sounds pretty good considering the maturity of the language. I'm not too concerned about Python 3.6 vs. 3.9.
some form of tree-shaking for the Python stdlib so that APEs can be smaller
That'd be nice considering modules like unicodedata alone is 17% of the binary. We can always create a really good build config as a stopgap. If we can make the assumption that modules have a single root and don't do sneak imports then it should be relatively straightforward to modify tool/build/zip.c to encode Python module deps into the ELF linkage and then maybe add a few STATIC_IMPORT statements here and there.
I am not sure how many testcases are failing because of an indirect dependency on threads. If only those particular tests are skipped I think the APE would have a higher pass rate on the test suite.
Also, using the test module for Python 3.6 APE does different things that just running the tests outright.
python.com -m test -W test_string_literals fails with the latin9 encoding, but python.com Lib/test/test_string_literals.py passes.
@jart here's the current performance of python.com on the Python test suite.
Use: the cosmo_py36 repo for building python.com. (The APE built in the cosmopolitan repo has some runtime issues)
Run at root of directory: ./python.com -E -Wd -m test -x test_json test_subprocess (two tests excluded because crashing aborts).
Including the two above, there are 407 tests. Current pass rate: 290/345 = 84%.
- 264 tests pass without complaint. Most of the non-OS parts of Python are fine in the Cosmopolitan build.
- 3 tests do nothing.
test_dtrace test_future4 test_largefileI don't think any of these are important - 62 tests are skipped, because some part of the stdlib (notably
_thread) is missing. Maybe 5-6 of these are important.
test_asyncgen test_asynchat test_asyncio test_bz2
test_concurrent_futures test_crypt test_ctypes test_curses
test_dbm_gnu test_dbm_ndbm test_devpoll test_docxmlrpc test_fork1
test_ftplib test_httpservers test_idle test_imaplib test_ioctl
test_kqueue test_lzma test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_ossaudiodev test_poplib
test_pty test_queue test_readline test_smtpnet test_socketserver
test_spwd test_sqlite test_ssl test_startfile test_tcl
test_telnetlib test_thread test_threaded_import
test_threadedtempfile test_threading test_threading_local
test_threadsignals test_timeout test_tix test_tk test_ttk_guionly
test_ttk_textonly test_turtle test_urllib2_localnet
test_urllib2net test_urllibnet test_wait3 test_wait4
test_winconsoleio test_winreg test_winsound test_wsgiref
test_xmlrpc_net test_zipfile64
- 76 (+ 2) tests fail.
test_aifc test_audioop test_cmath test_cmd_line
test_cmd_line_script test_code test_codeccallbacks
test_codecencodings_cn test_codecencodings_hk
test_codecencodings_iso2022 test_codecencodings_jp
test_codecencodings_kr test_codecencodings_tw test_codecs
test_compileall test_datetime test_descr test_distutils test_email
test_epoll test_fileinput test_gdb test_getpass test_httplib
test_imp test_import test_importlib test_inspect test_io
test_lib2to3 test_locale test_logging test_mailbox test_math
test_mmap test_multibytecodec test_nntplib test_openpty test_os
test_pathlib test_pdb test_pickle test_pickletools test_plistlib
test_posix test_pyclbr test_pydoc test_regrtest test_repl
test_reprlib test_runpy test_sax test_selectors test_signal
test_smtpd test_smtplib test_sndhdr test_socket
test_source_encoding test_string_literals test_strptime test_sunau
test_support test_sys test_sys_settrace test_time test_trace
test_ucn test_unicode test_urllib test_urllib2 test_venv test_wave
test_xml_etree test_xml_etree_c test_zipfile test_json test_subprocess
Of these 78 failures:
- 26 tests actually pass if
python.com -m testis run from a different location. This is because thetestpackage changes thesys.pathimports for the test env, and it affects how the APE loads files. Eg.test_string_literalspasses when called from theLibdirectory. - some failures are due to missing imports and should be skipped. Eg.
test_aifc,test_wave, andtest_audiooprequire theaudiooppackage which is not built. (threads issue here as well) - some failures may be important to fix. Eg
test_cmath/test_mathfail because of floating-point error in computingacos.
I'll post the exact error logs if necessary.
(The APE built in the cosmopolitan repo has some runtime issues).
By this I mean that running python.com -m test from inside the cosmopolitan repo fails because os.WNOHANG is missing: this is likely because some headers are not included (libc/sysv/consts/w.h). This may be occurring in other files apart from Modules/posixmodule.c, I'll submit a PR with a list of changes.
When I run python.com on Windows, it breaks my ability to review already typed commands using the up and down arrows (like #199 but only while python.com is running. I think the _ssl builtin module is missing. I guess Redbean's SSL support could be used for this?
@Keithcat1 I think the "up-arrow-to-view-previous-line" requires readline+(terminfo/curses) to be compiled along with python.com, otherwise you get stuff like [[A when you press Up in the REPL.
I've tried adding readline+curses to Python2.7 (see this commit), but curses fails at the linker stage for some reason. readline+terminfo gets compiled but the required functionality isn't there yet.
@Keithcat1 The Python shell should now work perfectly on Windows. See 5029e20befb339507bf1d2e57c2efe5d5175c57f
Contributions welcome on getting Python SSL to work. In the meantime you might consider using redbean as your SSL frontend. You can use the Fetch() Lua API to reverse proxy requests to your APE Python binary.
@jart as mentioned in #235, using .pyc files in the ZIP store halves the APE startup time.
When running python.com -Svc "2+2", most of the time is in loading the Python modules (.py/.pyc). Importing all the compiled C extension modules only takes around 10% percent of the overall time.
There has to be a way to load the Python modules faster, hopefully by avoiding some of the file-searching/C-Python-indirection involved. I don't think it's possible to do the entire importing within C, due to how _frozen_importlib works.
I'm trying to see what can be done with sys.path_hooks and/or sys.meta_path.
Yes using those .pyc files is going to have a huge impact, since it takes Python about a millisecond to load each .py. I created a fastpython branch in 214e3a68a9358b2ee7dc0cb27a63c06256979317 and run make -j8 o//tool/build/deltaify.com o//third_party/python/python.com && o//third_party/python/python.com -sSBc 'print(2+2)' |& o//tool/build/deltaify.com here's what shows up:
1
7 __zipos_opendir("zip!.python/")
1 __zipos_stat(".python/encodings/__init__.cpython36m-x86_64-cosmo.so") → enoent
0 __zipos_stat(".python/encodings/__init__.abi3.so") → enoent
0 __zipos_stat(".python/encodings/__init__.so") → enoent
0 __zipos_open(".python/encodings/__pycache__/__init__.cpython-36.pyc") enoent
0 __zipos_open(".python/encodings/__init__.py")
0 __zipos_fstat(".python/encodings/__init__.py")
0 __zipos_lseek(".python/encodings/__init__.py", 0)
1 __zipos_fstat(".python/encodings/__init__.py")
0 __zipos_read(".python/encodings/__init__.py", cap=5643, off=0) → got=5642
8 __zipos_read(".python/encodings/__init__.py", cap=1, off=5642) → got=0
1821 __zipos_close(".python/encodings/__init__.py")
77 __zipos_open(".python/__pycache__/codecs.cpython-36.pyc") enoent
2 __zipos_open(".python/codecs.py")
0 __zipos_fstat(".python/codecs.py")
0 __zipos_lseek(".python/codecs.py", 0)
0 __zipos_fstat(".python/codecs.py")
0 __zipos_read(".python/codecs.py", cap=36277, off=0) → got=36276
0 __zipos_read(".python/codecs.py", cap=1, off=36276) → got=0
2615 __zipos_close(".python/codecs.py")
126 __zipos_opendir("zip!.python/encodings")
3 __zipos_open(".python/encodings/__pycache__/aliases.cpython-36.pyc") enoent
1 __zipos_open(".python/encodings/aliases.py")
3 __zipos_fstat(".python/encodings/aliases.py")
0 __zipos_lseek(".python/encodings/aliases.py", 0)
0 __zipos_fstat(".python/encodings/aliases.py")
0 __zipos_read(".python/encodings/aliases.py", cap=15578, off=0) → got=15577
8 __zipos_read(".python/encodings/aliases.py", cap=1, off=15577) → got=0
905 __zipos_close(".python/encodings/aliases.py")
2 __zipos_open(".python/encodings/__pycache__/utf_8.cpython-36.pyc") enoent
0 __zipos_open(".python/encodings/utf_8.py")
0 __zipos_fstat(".python/encodings/utf_8.py")
0 __zipos_lseek(".python/encodings/utf_8.py", 0)
0 __zipos_fstat(".python/encodings/utf_8.py")
0 __zipos_read(".python/encodings/utf_8.py", cap=1006, off=0) → got=1005
0 __zipos_read(".python/encodings/utf_8.py", cap=1, off=1005) → got=0
357 __zipos_close(".python/encodings/utf_8.py")
2 __zipos_open(".python/encodings/__pycache__/latin_1.cpython-36.pyc") enoent
0 __zipos_open(".python/encodings/latin_1.py")
0 __zipos_fstat(".python/encodings/latin_1.py")
0 __zipos_lseek(".python/encodings/latin_1.py", 0)
0 __zipos_fstat(".python/encodings/latin_1.py")
0 __zipos_read(".python/encodings/latin_1.py", cap=1265, off=0) → got=1264
0 __zipos_read(".python/encodings/latin_1.py", cap=1, off=1264) → got=0
328 __zipos_close(".python/encodings/latin_1.py")
2 __zipos_open(".python/__pycache__/io.cpython-36.pyc") enoent
0 __zipos_open(".python/io.py")
0 __zipos_fstat(".python/io.py")
0 __zipos_lseek(".python/io.py", 0)
0 __zipos_fstat(".python/io.py")
0 __zipos_read(".python/io.py", cap=3518, off=0) → got=3517
1 __zipos_read(".python/io.py", cap=1, off=3517) → got=0
288 __zipos_close(".python/io.py")
2 __zipos_open(".python/__pycache__/abc.cpython-36.pyc") enoent
1 __zipos_open(".python/abc.py")
0 __zipos_fstat(".python/abc.py")
0 __zipos_lseek(".python/abc.py", 0)
0 __zipos_fstat(".python/abc.py")
0 __zipos_read(".python/abc.py", cap=8728, off=0) → got=8727
0 __zipos_read(".python/abc.py", cap=1, off=8727) → got=0
635 __zipos_close(".python/abc.py")
2 __zipos_open(".python/__pycache__/_weakrefset.cpython-36.pyc") enoent
0 __zipos_open(".python/_weakrefset.py")
0 __zipos_fstat(".python/_weakrefset.py")
0 __zipos_lseek(".python/_weakrefset.py", 0)
0 __zipos_fstat(".python/_weakrefset.py")
0 __zipos_read(".python/_weakrefset.py", cap=5706, off=0) → got=5705
0 __zipos_read(".python/_weakrefset.py", cap=1, off=5705) → got=0
1127 __zipos_close(".python/_weakrefset.py")
2 __zipos_open(".python/__pycache__/_bootlocale.cpython-36.pyc") enoent
0 __zipos_open(".python/_bootlocale.py")
0 __zipos_fstat(".python/_bootlocale.py")
0 __zipos_lseek(".python/_bootlocale.py", 0)
0 __zipos_fstat(".python/_bootlocale.py")
0 __zipos_read(".python/_bootlocale.py", cap=1313, off=0) → got=1312
0 __zipos_read(".python/_bootlocale.py", cap=1, off=1312) → got=0
241 __zipos_close(".python/_bootlocale.py")
119 __zipos_open(".python/__pycache__/locale.cpython-36.pyc") enoent
2 __zipos_open(".python/locale.py")
0 __zipos_fstat(".python/locale.py")
0 __zipos_lseek(".python/locale.py", 0)
0 __zipos_fstat(".python/locale.py")
0 __zipos_read(".python/locale.py", cap=77301, off=0) → got=77300
0 __zipos_read(".python/locale.py", cap=1, off=77300) → got=0
4265 __zipos_close(".python/locale.py")
53 __zipos_open(".python/__pycache__/re.cpython-36.pyc") enoent
2 __zipos_open(".python/re.py")
0 __zipos_fstat(".python/re.py")
0 __zipos_lseek(".python/re.py", 0)
0 __zipos_fstat(".python/re.py")
0 __zipos_read(".python/re.py", cap=15553, off=0) → got=15552
0 __zipos_read(".python/re.py", cap=1, off=15552) → got=0
920 __zipos_close(".python/re.py")
70 __zipos_open(".python/__pycache__/enum.cpython-36.pyc") enoent
2 __zipos_open(".python/enum.py")
0 __zipos_fstat(".python/enum.py")
0 __zipos_lseek(".python/enum.py", 0)
0 __zipos_fstat(".python/enum.py")
0 __zipos_read(".python/enum.py", cap=33607, off=0) → got=33606
0 __zipos_read(".python/enum.py", cap=1, off=33606) → got=0
2856 __zipos_close(".python/enum.py")
5 __zipos_open(".python/__pycache__/types.cpython-36.pyc") enoent
1 __zipos_open(".python/types.py")
0 __zipos_fstat(".python/types.py")
0 __zipos_lseek(".python/types.py", 0)
0 __zipos_fstat(".python/types.py")
0 __zipos_read(".python/types.py", cap=8871, off=0) → got=8870
8 __zipos_read(".python/types.py", cap=1, off=8870) → got=0
975 __zipos_close(".python/types.py")
124 __zipos_open(".python/__pycache__/functools.cpython-36.pyc") enoent
2 __zipos_open(".python/functools.py")
0 __zipos_fstat(".python/functools.py")
0 __zipos_lseek(".python/functools.py", 0)
0 __zipos_fstat(".python/functools.py")
0 __zipos_read(".python/functools.py", cap=31347, off=0) → got=31346
0 __zipos_read(".python/functools.py", cap=1, off=31346) → got=0
2936 __zipos_close(".python/functools.py")
2 __zipos_stat(".python/collections/__init__.cpython36m-x86_64-cosmo.so") → enoent
0 __zipos_stat(".python/collections/__init__.abi3.so") → enoent
60 __zipos_stat(".python/collections/__init__.so") → enoent
77 __zipos_open(".python/collections/__pycache__/__init__.cpython-36.pyc") enoent
1 __zipos_open(".python/collections/__init__.py")
0 __zipos_fstat(".python/collections/__init__.py")
2 __zipos_lseek(".python/collections/__init__.py", 0)
0 __zipos_fstat(".python/collections/__init__.py")
0 __zipos_read(".python/collections/__init__.py", cap=45813, off=0) → got=45812
0 __zipos_read(".python/collections/__init__.py", cap=1, off=45812) → got=0
4174 __zipos_close(".python/collections/__init__.py")
50 __zipos_open(".python/__pycache__/_collections_abc.cpython-36.pyc") enoent
2 __zipos_open(".python/_collections_abc.py")
0 __zipos_fstat(".python/_collections_abc.py")
0 __zipos_lseek(".python/_collections_abc.py", 0)
0 __zipos_fstat(".python/_collections_abc.py")
0 __zipos_read(".python/_collections_abc.py", cap=26393, off=0) → got=26392
0 __zipos_read(".python/_collections_abc.py", cap=1, off=26392) → got=0
3573 __zipos_close(".python/_collections_abc.py")
7 __zipos_open(".python/__pycache__/operator.cpython-36.pyc") enoent
0 __zipos_open(".python/operator.py")
0 __zipos_fstat(".python/operator.py")
0 __zipos_lseek(".python/operator.py", 0)
0 __zipos_fstat(".python/operator.py")
0 __zipos_read(".python/operator.py", cap=10864, off=0) → got=10863
8 __zipos_read(".python/operator.py", cap=1, off=10863) → got=0
1511 __zipos_close(".python/operator.py")
2 __zipos_open(".python/__pycache__/keyword.cpython-36.pyc") enoent
0 __zipos_open(".python/keyword.py")
0 __zipos_fstat(".python/keyword.py")
0 __zipos_lseek(".python/keyword.py", 0)
0 __zipos_fstat(".python/keyword.py")
0 __zipos_read(".python/keyword.py", cap=2212, off=0) → got=2211
0 __zipos_read(".python/keyword.py", cap=1, off=2211) → got=0
340 __zipos_close(".python/keyword.py")
58 __zipos_open(".python/__pycache__/heapq.cpython-36.pyc") enoent
2 __zipos_open(".python/heapq.py")
0 __zipos_fstat(".python/heapq.py")
0 __zipos_lseek(".python/heapq.py", 0)
0 __zipos_fstat(".python/heapq.py")
0 __zipos_read(".python/heapq.py", cap=22930, off=0) → got=22929
0 __zipos_read(".python/heapq.py", cap=1, off=22929) → got=0
1499 __zipos_close(".python/heapq.py")
2 __zipos_open(".python/__pycache__/reprlib.cpython-36.pyc") enoent
0 __zipos_open(".python/reprlib.py")
0 __zipos_fstat(".python/reprlib.py")
0 __zipos_lseek(".python/reprlib.py", 0)
1 __zipos_fstat(".python/reprlib.py")
0 __zipos_read(".python/reprlib.py", cap=5337, off=0) → got=5336
0 __zipos_read(".python/reprlib.py", cap=1, off=5336) → got=0
836 __zipos_close(".python/reprlib.py")
2 __zipos_open(".python/__pycache__/_dummy_thread.cpython-36.pyc") enoent
0 __zipos_open(".python/_dummy_thread.py")
0 __zipos_fstat(".python/_dummy_thread.py")
0 __zipos_lseek(".python/_dummy_thread.py", 0)
0 __zipos_fstat(".python/_dummy_thread.py")
0 __zipos_read(".python/_dummy_thread.py", cap=5168, off=0) → got=5167
0 __zipos_read(".python/_dummy_thread.py", cap=1, off=5167) → got=0
899 __zipos_close(".python/_dummy_thread.py")
48 __zipos_open(".python/__pycache__/weakref.cpython-36.pyc") enoent
2 __zipos_open(".python/weakref.py")
0 __zipos_fstat(".python/weakref.py")
0 __zipos_lseek(".python/weakref.py", 0)
0 __zipos_fstat(".python/weakref.py")
0 __zipos_read(".python/weakref.py", cap=20467, off=0) → got=20466
0 __zipos_read(".python/weakref.py", cap=1, off=20466) → got=0
2505 __zipos_close(".python/weakref.py")
64 __zipos_opendir("zip!.python/collections")
2 __zipos_open(".python/collections/__pycache__/abc.cpython-36.pyc") enoent
0 __zipos_open(".python/collections/abc.py")
0 __zipos_fstat(".python/collections/abc.py")
0 __zipos_lseek(".python/collections/abc.py", 0)
0 __zipos_fstat(".python/collections/abc.py")
0 __zipos_read(".python/collections/abc.py", cap=69, off=0) → got=68
0 __zipos_read(".python/collections/abc.py", cap=1, off=68) → got=0
526 __zipos_close(".python/collections/abc.py")
50 __zipos_open(".python/__pycache__/sre_compile.cpython-36.pyc") enoent
2 __zipos_open(".python/sre_compile.py")
0 __zipos_fstat(".python/sre_compile.py")
0 __zipos_lseek(".python/sre_compile.py", 0)
0 __zipos_fstat(".python/sre_compile.py")
0 __zipos_read(".python/sre_compile.py", cap=19339, off=0) → got=19338
0 __zipos_read(".python/sre_compile.py", cap=1, off=19338) → got=0
2168 __zipos_close(".python/sre_compile.py")
68 __zipos_open(".python/__pycache__/sre_parse.cpython-36.pyc") enoent
2 __zipos_open(".python/sre_parse.py")
0 __zipos_fstat(".python/sre_parse.py")
0 __zipos_lseek(".python/sre_parse.py", 0)
0 __zipos_fstat(".python/sre_parse.py")
0 __zipos_read(".python/sre_parse.py", cap=36537, off=0) → got=36536
0 __zipos_read(".python/sre_parse.py", cap=1, off=36536) → got=0
3980 __zipos_close(".python/sre_parse.py")
3 __zipos_open(".python/__pycache__/sre_constants.cpython-36.pyc") enoent
0 __zipos_open(".python/sre_constants.py")
0 __zipos_fstat(".python/sre_constants.py")
0 __zipos_lseek(".python/sre_constants.py", 0)
0 __zipos_fstat(".python/sre_constants.py")
0 __zipos_read(".python/sre_constants.py", cap=6822, off=0) → got=6821
0 __zipos_read(".python/sre_constants.py", cap=1, off=6821) → got=0
1151 __zipos_close(".python/sre_constants.py")
2 __zipos_open(".python/__pycache__/copyreg.cpython-36.pyc") enoent
0 __zipos_open(".python/copyreg.py")
1 __zipos_fstat(".python/copyreg.py")
0 __zipos_lseek(".python/copyreg.py", 0)
0 __zipos_fstat(".python/copyreg.py")
0 __zipos_read(".python/copyreg.py", cap=7008, off=0) → got=7007
0 __zipos_read(".python/copyreg.py", cap=1, off=7007) → got=0
984 __zipos_close(".python/copyreg.py")
87 __zipos_open(".python/__pycache__/os.cpython-36.pyc") enoent
2 __zipos_open(".python/os.py")
0 __zipos_fstat(".python/os.py")
0 __zipos_lseek(".python/os.py", 0)
0 __zipos_fstat(".python/os.py")
0 __zipos_read(".python/os.py", cap=37527, off=0) → got=37526
0 __zipos_read(".python/os.py", cap=1, off=37526) → got=0
2931 __zipos_close(".python/os.py")
2 __zipos_open(".python/__pycache__/stat.cpython-36.pyc") enoent
0 __zipos_open(".python/stat.py")
0 __zipos_fstat(".python/stat.py")
1 __zipos_lseek(".python/stat.py", 0)
0 __zipos_fstat(".python/stat.py")
0 __zipos_read(".python/stat.py", cap=5039, off=0) → got=5038
0 __zipos_read(".python/stat.py", cap=1, off=5038) → got=0
623 __zipos_close(".python/stat.py")
49 __zipos_open(".python/__pycache__/posixpath.cpython-36.pyc") enoent
2 __zipos_open(".python/posixpath.py")
0 __zipos_fstat(".python/posixpath.py")
0 __zipos_lseek(".python/posixpath.py", 0)
0 __zipos_fstat(".python/posixpath.py")
2 __zipos_read(".python/posixpath.py", cap=15773, off=0) → got=15772
0 __zipos_read(".python/posixpath.py", cap=1, off=15772) → got=0
1561 __zipos_close(".python/posixpath.py")
2 __zipos_open(".python/__pycache__/genericpath.cpython-36.pyc") enoent
0 __zipos_open(".python/genericpath.py")
0 __zipos_fstat(".python/genericpath.py")
0 __zipos_lseek(".python/genericpath.py", 0)
0 __zipos_fstat(".python/genericpath.py")
0 __zipos_read(".python/genericpath.py", cap=4757, off=0) → got=4756
1 __zipos_read(".python/genericpath.py", cap=1, off=4756) → got=0
716 __zipos_close(".python/genericpath.py")
1625 4
If you do it with the verbose flag you mentioned:
1
8 import _frozen_importlib # frozen
1 import _imp # builtin
0 import sys # builtin
0 import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
263 import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
207 import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
3 import '_io' # <class '_frozen_importlib.BuiltinImporter'>
153 import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
3 import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
0 import _weakref # previously loaded ('_weakref')
1995 import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
2609 # code object from zip!.python/encodings/__init__.py
57 # code object from zip!.python/codecs.py
127 import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
903 import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x100080a5d208>
3 # code object from zip!.python/encodings/aliases.py
16 import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x100080a5def0>
327 import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x1000800e4518>
63 # code object from zip!.python/encodings/utf_8.py
75 import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b83ac8>
221 import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
119 # code object from zip!.python/encodings/latin_1.py
306 import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b5c780>
637 # code object from zip!.python/io.py
807 # code object from zip!.python/abc.py
50 # code object from zip!.python/_weakrefset.py
73 import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b62208>
158 import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b5d5c0>
203 import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b5cc50>
49 # code object from zip!.python/_bootlocale.py
1 import '_locale' # <class '_frozen_importlib.BuiltinImporter'>
4446 import '_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b4cc50>
979 # code object from zip!.python/locale.py
2937 # code object from zip!.python/re.py
953 # code object from zip!.python/enum.py
3080 # code object from zip!.python/types.py
50 # code object from zip!.python/functools.py
4251 import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
2787 # code object from zip!.python/collections/__init__.py
818 # code object from zip!.python/_collections_abc.py
1482 import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b717b8>
102 # code object from zip!.python/operator.py
3 import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
357 import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x100080af4b70>
3 # code object from zip!.python/keyword.py
1413 import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c54048>
85 # code object from zip!.python/heapq.py
3 import '_heapq' # <class '_frozen_importlib.BuiltinImporter'>
102 import 'heapq' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c54128>
808 import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
448 # code object from zip!.python/reprlib.py
3 # code object from zip!.python/_dummy_thread.py
58 import '_dummy_thread' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c685f8>
57 import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x100080ac6be0>
376 import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
2091 import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c445c0>
153 # code object from zip!.python/weakref.py
385 import 'weakref' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b71b38>
117 import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x100080d8d320>
3 # code object from zip!.python/collections/abc.py
52 import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c441d0>
379 import 'types' # <_frozen_importlib_external.SourceFileLoader object at 0x100080d7f0b8>
2176 import 'enum' # <_frozen_importlib_external.SourceFileLoader object at 0x100080d655c0>
62 # code object from zip!.python/sre_compile.py
3974 import '_sre' # <class '_frozen_importlib.BuiltinImporter'>
620 # code object from zip!.python/sre_parse.py
106 # code object from zip!.python/sre_constants.py
84 import 'sre_constants' # <_frozen_importlib_external.SourceFileLoader object at 0x100080e2aac8>
52 import 'sre_parse' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c81518>
868 import 'sre_compile' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c442b0>
3 # code object from zip!.python/copyreg.py
9 import 'copyreg' # <_frozen_importlib_external.SourceFileLoader object at 0x100080affcf8>
342 import 're' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b4d278>
2838 import 'locale' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b4cd30>
61 # code object from zip!.python/os.py
589 import 'errno' # <class '_frozen_importlib.BuiltinImporter'>
51 # code object from zip!.python/stat.py
3 import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
1644 import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x100080c5c9b0>
493 # code object from zip!.python/posixpath.py
3 # code object from zip!.python/genericpath.py
1 import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b17a58>
208 import 'posixpath' # <_frozen_importlib_external.SourceFileLoader object at 0x100080b176d8>
86 import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x100080d65160>
3 Python 3.6.14+ (cosmopolitan)
0 [GCC 9.2.0] on cosmo
1317 4
3 # clear builtins._
0 # clear sys.path
0 # clear sys.argv
0 # clear sys.ps1
0 # clear sys.ps2
0 # clear sys.last_type
0 # clear sys.last_value
0 # clear sys.last_traceback
0 # clear sys.path_hooks
0 # clear sys.path_importer_cache
0 # clear sys.meta_path
0 # clear sys.__interactivehook__
0 # clear sys.flags
0 # clear sys.float_info
0 # restore sys.stdin
0 # restore sys.stdout
0 # restore sys.stderr
0 # cleanup[2] removing builtins
0 # cleanup[2] removing sys
0 # cleanup[2] removing _frozen_importlib
0 # cleanup[2] removing _imp
0 # cleanup[2] removing _warnings
0 # cleanup[2] removing _weakref
0 # cleanup[2] removing _frozen_importlib_external
0 # cleanup[2] removing _io
0 # cleanup[2] removing marshal
0 # cleanup[2] removing posix
2 # cleanup[2] removing encodings
0 # cleanup[2] removing codecs
0 # cleanup[2] removing _codecs
0 # cleanup[2] removing encodings.aliases
0 # cleanup[2] removing encodings.utf_8
0 # cleanup[2] removing _signal
0 # cleanup[2] removing __main__
0 # destroy __main__
0 # cleanup[2] removing encodings.latin_1
0 # cleanup[2] removing io
0 # destroy io
0 # cleanup[2] removing abc
0 # cleanup[2] removing _weakrefset
0 # destroy _weakrefset
0 # cleanup[2] removing _bootlocale
0 # destroy _bootlocale
0 # cleanup[2] removing _locale
0 # cleanup[2] removing locale
0 # destroy locale
0 # cleanup[2] removing re
0 # cleanup[2] removing enum
0 # cleanup[2] removing types
0 # destroy types
0 # cleanup[2] removing functools
0 # cleanup[2] removing _functools
0 # cleanup[2] removing collections
0 # cleanup[2] removing _collections_abc
0 # cleanup[2] removing operator
0 # destroy operator
0 # cleanup[2] removing _operator
0 # cleanup[2] removing keyword
0 # destroy keyword
0 # cleanup[2] removing heapq
0 # cleanup[2] removing _heapq
0 # cleanup[2] removing itertools
0 # cleanup[2] removing reprlib
0 # destroy reprlib
0 # cleanup[2] removing _dummy_thread
0 # destroy _dummy_thread
0 # cleanup[2] removing _collections
0 # cleanup[2] removing weakref
0 # destroy weakref
0 # cleanup[2] removing collections.abc
0 # cleanup[2] removing sre_compile
0 # cleanup[2] removing _sre
0 # cleanup[2] removing sre_parse
0 # cleanup[2] removing sre_constants
0 # destroy sre_constants
0 # cleanup[2] removing copyreg
0 # cleanup[2] removing os
0 # cleanup[2] removing errno
0 # cleanup[2] removing stat
0 # cleanup[2] removing _stat
0 # cleanup[2] removing posixpath
0 # cleanup[2] removing genericpath
0 # cleanup[2] removing os.path
0 # destroy _signal
0 # destroy encodings
0 # destroy re
0 # destroy enum
0 # destroy sre_compile
0 # destroy _locale
0 # destroy copyreg
0 # destroy functools
0 # destroy _stat
0 # destroy genericpath
0 # destroy os
0 # destroy _functools
0 # destroy _collections_abc
0 # destroy heapq
0 # destroy collections.abc
0 # destroy _operator
0 # destroy _heapq
0 # destroy _collections
0 # destroy collections
0 # destroy itertools
0 # destroy sre_parse
0 # destroy _sre
0 # destroy abc
0 # destroy errno
0 # destroy stat
0 # destroy posixpath
0 # cleanup[3] wiping _frozen_importlib
0 # destroy _frozen_importlib_external
0 # cleanup[3] wiping _imp
0 # cleanup[3] wiping _warnings
0 # cleanup[3] wiping _weakref
0 # cleanup[3] wiping _io
0 # cleanup[3] wiping marshal
0 # cleanup[3] wiping posix
0 # cleanup[3] wiping codecs
0 # cleanup[3] wiping _codecs
0 # cleanup[3] wiping encodings.aliases
0 # cleanup[3] wiping encodings.utf_8
0 # cleanup[3] wiping encodings.latin_1
1 # cleanup[3] wiping sys
374 # cleanup[3] wiping builtins
due to how _frozen_importlib works
Could you help me understand what that is? I've noticed a few C sources that seem to have Python binary byte code embedded in them. Do you know the commands for regenerating that?
So I got the freeze program working in d522a88d. I wish I had a dollar for every time I've seen Python crash with "Unable to get the locale encoding". I built what I hope is a Python compiler in 0c6581f9. It seemed to work fine for loading modules until I used it for early stage module loading, which seems to follow different rules. It looks like the source code for the encodings module needs to be embedded uncompiled?
Could you help me understand what that is? I've noticed a few C sources that seem to have Python binary byte code embedded in them. Do you know the commands for regenerating that?
Python handles the loading of modules using the code in Lib/importlib/_bootstrap.py and Lib/importlib/_bootstrap_external.py. But there's a circular problem here because these files which handle the imports cannot be imported the way they import other modules.
The solution is via _frozen_importlib: the above two .py files are compiled into bytecode and "frozen" into raw bytes in importlib.inc. So _frozen_importlib is the first module that the interpreter loads, and it is used for loading along other modules during startup.
The Python Makefile contains a target called regen-importlib (see here), wherein if you change the _bootstrap.py files, they will be converted into the raw bytes seen in importlib.inc.
@jart Regarding the verbose logs seen above with deltaify.com, some of the time measurements for imports are misleading, because the delay is actually caused before the import happens. Example below.
As a C extension, the _signal module takes very little time to load, as expected
221 import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
But the _functools modules, which is also a C extension takes a lot more time:
50 # code object from zip!.python/functools.py
4251 import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
The time is not spent in the actual import of _functools; the time is spent in loading, parsing, and executing the code in functools.py which then calls import _functools. If _functools is imported by itself before functools.py is executed, the import takes much less time.
You can confirm this by loading all the C extensions early, before any Python modules are even considered:
https://github.com/jart/cosmopolitan/blob/3d0347e26efc05ad98ba70e9ce3963077458e4c6/third_party/python/Python/pylifecycle.c#L304-L310
If I call PyImport_ImportModule("_functools") (or any other C extension like _sre or _signal) right after _PyImport_Zip_Init() completes, for me the logs show that _functools by itself takes minimal time.
The slowness is only when it comes handling the Python side of things (so much indirection!).
During startup, the interpreter loads extension modules in the following manner (can view by setting gdb breakpoints in debug build and examining frames):
- important internals like
_frozen_importlib,_imp,sysetc. are loaded first - some code in startup calls
PyImport_ImportModule - after some indirection this resolves to calling the
__import__in_frozen_importlib._bootstrap.pywith the necessary parameters -
_frozen_importlib._bootstrap.pyfirst checks if the module is already loaded or cached -
_frozen_importlib._bootstrap.pythen looks throughsys.meta_pathandsys.path_hooksto find a loader for importing the module - currently, the available loaders are :
- Python's own
zipimporterfor loading from zip archives (not sure if this can be faster than if done at the Cosmo level) -
BuiltinImporterfor built-in C extension modules, and -
SourceFileLoaderfor python packages and files
- Python's own
-
BuiltinImporterworks only for C extensions, and is very fast. -
SourceFileLoaderis slow because it does some filesystem calls, parses.pyfiles, tries to write.pycfiles, executes the loaded files
The options for speedup are:
- re-arrange imports of C extensions (like with
_functoolsabove) so that Python part of importing is boxed in to one spot - find a way to import a module without relying on Python
machinery(if I justfopena file in the ZIP store, is it possible to somehow convert that into loading the module?) - have
zipimportread the ZIP store of the APE when it is initialized and mark import locations viaread_directory - insert an entry at the start of
sys.meta_paththat allows us to shortcut viacosmo_importerC extension - insert an entry at the start of
sys.path_hookssimilar to above - larges-scale changes to
importlib/_frozen_importlibwhere CPython stuff is changed to raw C as much as possible (not sure if this is worth the effort)
So I got the freeze program working in d522a88. I wish I had a dollar for every time I've seen Python crash with "Unable to get the locale encoding".
I would also have gotten a decent amount of money :laughing: Alas, Python2.7 did not suffer from this "locale encoding" affliction, it just used ASCII everywhere and only relied on importing .py files at the end of the startup process.
I built what I hope is a Python compiler in 0c6581f.
By compiler do you mean "compiles to .pyc" or "compiles to raw bytes and use in C like frozen_importlib"?
A Python compiler in C sounds awesome, let me try it out.
Is pycomp.com faster than running python.com -m compileall? I thought we could just have a minimal python.com with compileall in tool/build/ along with zipobj.com, as updating that would be easier.
It seemed to work fine for loading modules until I used it for early stage module loading, which seems to follow different rules. It looks like the source code for the encodings module needs to be embedded uncompiled?
I've explained what I've gleaned about the import process above. I'm pretty sure you can import encodings from a encodings/__init__.pyc file without issue; in the cosmo_py36 fork I used a filter to ensure only .pyc were added to python.com, and it had no issue finding encodings.pyc.
See cabb0a7e. It's now officially a working 1.1mb pycomp.com binary which turns .py files into .pyc files. I found an objdump tool for pyc files that let me confirm that it 100% works. What I want is to compile the pyc files using the makefile, have the makefile then put the pyc files inside the zip, and then instruct python to load those instead.
The problem is that Python loads the .pyc. Then it silently ignores it and reads the .py afterwards anyway. Could you help me find out why?
I feel really uncomfortable with the way Python crisscrosses C and Python in the bootstrap process. The only module loading process I understand so far is the static linking one it uses for C. Look at how elegant the compiler is:
https://github.com/jart/cosmopolitan/blob/cabb0a7ede6b0855b0ccbd59545b7b255c6cea8c/third_party/python/pycomp.c#L43-L189
@jart with #248 the APE startup time has been reduced to 0.044s.
I feel really uncomfortable with the way Python crisscrosses C and Python in the bootstrap process.
A simple function call goes through 5 different files! All that indirection makes it unnaturally difficult to follow program flow. That's the price to pay for modifying Python I guess.
We can reduce APE size a bit further by excluding docstrings: just have to put a #ifdef around WITH_DOC_STRINGS in pyconfig.h, and have pycomp use optimize=2.
With MODE=tiny, only .pyc files in the ZIP store, and removing all docstrings python.com is at 6MB.
With MODE=tiny, only .pyc files in the ZIP store, but keeping all docstrings python.com is at 6.8MB.
The only module loading process I understand so far is the static linking one it uses for C.
@jart I got an idea of how modules in general are loaded, and this is one way to avoid criss-crossing the C-Python border when importing Python files. The basic idea is:
find a way to import a module without relying on Python machinery (if I just fopen a file in the ZIP store, is it possible to somehow convert that into loading the module?)
Consider a file hello.py (or it's equivalent hello.pyc).
Importing hello.py is slower than a C extension as it involves criss-crossing between C and Python + filesystem calls.
Importing hello.pyc would be slightly faster, but there is still a lot of criss-crossing to slow things down.
import hello first checks for a key "hello" in sys.modules to see if the module already exists. So, if we can somehow add hello as a module to sys.modules before the import statement is executed, it would save time.
Example in Python, but I'm pretty sure it can be written in C using the CPython API (and avoid indirection when possible for eg. with fopen), because it uses only the most basic modules.
import sys
import marshal
import types
def preload_module_from_file(name, file):
mod = types.ModuleType(name) # Can use PyModule_CreateObject in the C API
mod.__dict__["__name__"] = name
mod.__dict__["__file__"] = file
if name in sys.modules.keys():
return
sys.modules[name] = mod
if file.endswith(".pyc"):
with open(file, "rb") as f:
raw = f.read()
# can validate magic number+timestamp+size here
# or just skip the first 12 bytes
code = marshal.loads(raw[12:])
exec(code, mod.__dict__)
else:
with open(file, "r") as f:
exec(f.read(), mod.__dict__)
preload_module_from_file("hello", "hello.pyc") # faster when done in C w/ APE ZIP store
import hello # faster because preloaded
Similarly, if we call preload_module_from_file("encodings", "zip!./python/Lib/encodings/__init__.pyc") in C, before the actual import happens, it should be faster than just calling the import outright. (note that all the dependencies of encodings need to already be preloaded, otherwise there is no speedup).
If the C version is fast enough, module startup does not needs to involve crossing to the Python side as much, and there will be a net speed gain. This also opens the possibility of a macro PRELOAD_PYIMPORT(modname, filename) to be used in the C code.
@jart I came across Stackless Python 3.6.13, might be useful to port the stackless module to Cosmopolitan (the microthreads/coroutines don't seem to rely on pthreads).
After the last update, I'm having problems building Python. Here is what may or may not be a relevant block of text (there's a lot of it). ... build/bootstrap/ar.com rcsD o//third_party/python/python-stdlib-dirs.a o//third_party/python/Lib/.zip.o o//third_party/python/Lib/asyncio/.zip.o o//third_party/python/Lib/collections/.zip.o o//third_party/python/Lib/dbm/.zip.o o//third_party/python/Lib/distutils/.zip.o o//third_party/python/Lib/distutils/command/.zip.o o//third_party/python/Lib/distutils/tests/.zip.o o//third_party/python/Lib/email/.zip.o o//third_party/python/Lib/email/mime/.zip.o o//third_party/python/Lib/encodings/.zip.o o//third_party/python/Lib/ensurepip/.zip.o o//third_party/python/Lib/ensurepip/bundled/.zip.o o//third_party/python/Lib/html/.zip.o o//third_party/python/Lib/http/.zip.o o//third_party/python/Lib/importlib/.zip.o o//third_party/python/Lib/json/.zip.o o//third_party/python/Lib/logging/.zip.o o//third_party/python/Lib/msilib/.zip.o o//third_party/python/Lib/multiprocessing/.zip.o o//third_party/python/Lib/multiprocessing/dummy/.zip.o o//third_party/python/Lib/sqlite3/.zip.o o//third_party/python/Lib/unittest/.zip.o o//third_party/python/Lib/urllib/.zip.o o//third_party/python/Lib/venv/.zip.o o//third_party/python/Lib/venv/scripts/common/.zip.o o//third_party/python/Lib/venv/scripts/nt/.zip.o o//third_party/python/Lib/venv/scripts/posix/.zip.o o//third_party/python/Lib/wsgiref/.zip.o o//third_party/python/Lib/xml/.zip.o o//third_party/python/Lib/xml/dom/.zip.o o//third_party/python/Lib/xml/etree/.zip.o o//third_party/python/Lib/xml/parsers/.zip.o o//third_party/python/Lib/xml/sax/.zip.o o//third_party/python/Lib/xmlrpc/.zip.o o//third_party/python/Lib/test/.zip.o o//third_party/python/Lib/test/xmltestdata/.zip.o o//third_party/python/Lib/test/test_email/.zip.o o//third_party/python/Lib/test/test_email/data/.zip.o o//third_party/python/Lib/test/sndhdrdata/.zip.o o//third_party/python/Lib/test/test_asyncio/.zip.o o//third_party/python/Lib/test/pycache/.zip.o o//third_party/python/Lib/test/audiodata/.zip.o o//third_party/python/Lib/test/imghdrdata/.zip.o o//third_party/python/Lib/test/decimaltestdata/.zip.o o//third_party/python/Lib/test/test_import/.zip.o o//third_party/python/Lib/test/test_import/data/.zip.o o//third_party/python/Lib/test/test_import/data/package/.zip.o o//third_party/python/Lib/test/test_import/data/package2/.zip.o o//third_party/python/Lib/test/test_import/data/circular_imports/.zip.o o//third_party/python/Lib/test/test_import/data/circular_imports/subpkg/.zip.o o//third_party/python/Lib/test/libregrtest/.zip.o o//third_party/python/Lib/test/libregrtest/pycache/.zip.o o//third_party/python/Lib/test/leakers/.zip.o o//third_party/python/Lib/test/test_json/.zip.o o//third_party/python/Lib/test/eintrdata/.zip.o o//third_party/python/Lib/test/support/.zip.o o//third_party/python/Lib/test/support/pycache/.zip.o o//third_party/python/Lib/test/test_importlib/.zip.o o//third_party/python/Lib/test/test_importlib/extension/.zip.o o//third_party/python/Lib/test/test_importlib/frozen/.zip.o o//third_party/python/Lib/test/test_importlib/import/.zip.o o//third_party/python/Lib/test/test_importlib/builtin/.zip.o o//third_party/python/Lib/test/test_importlib/source/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project2/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project2/parent/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project2/parent/child/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/portion2/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/portion2/foo/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project3/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project3/parent/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project3/parent/child/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/portion1/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/portion1/foo/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/both_portions/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/both_portions/foo/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project1/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project1/parent/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/project1/parent/child/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/.zip.o o//third_party/python/Lib/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test/.zip.o o//third_party/python/Lib/test/test_warnings/.zip.o o//third_party/python/Lib/test/test_warnings/data/.zip.o o//third_party/python/Lib/test/capath/.zip.o o//third_party/python/Lib/test/dtracedata/.zip.o o//third_party/python/Lib/test/subprocessdata/.zip.o o//third_party/python/Lib/test/crashers/.zip.o o//third_party/python/Lib/test/cjkencodings/.zip.o o//third_party/python/Lib/test/test_tools/.zip.o o//third_party/python/Lib/test/tracedmodules/.zip.o
[J[30;101merror[94;49m:tool/build/ar.c:173:ar.com.tmp.10960[0m: check failed on keith-pc pid 10961
CHECK_NE(-1, (fd = open(args.p[i], O_RDONLY)));
→ 0xffffffffffffffff (-1)
!= 0xffffffffffffffff ((fd = open(args.p[i], O_RDONLY)))
o//third_party/python/Lib/test/pycache/.zip.o
ENOENT[2]
[35mo/build/bootstrap/ar.com.tmp.10960 \[0m
rcsD
o//third_party/python/python-stdlib-dirs.a
...
I'm also having problems with make. Quite often, the make process it just randomly stops instead of completing the build, even though there doesn't seem to be an error. Could we move to CMake and Ninja? One advantage is that building targets is easy, it would probably look like this:
cmake -G Ninja -DCMAKE_BUILD_TYPE=release .. ninja redbean python examples But oh, wouldn't this break the vendored GCC? It might also make compiling on Windows easier / harder?
CHECK_NE(-1, (fd = open(args.p[i], O_RDONLY))); → 0xffffffffffffffff (-1) != 0xffffffffffffffff ((fd = open(args.p[i], O_RDONLY))) o//third_party/python/Lib/test/pycache/.zip.o ENOENT[2] �[35mo/build/bootstrap/ar.com.tmp.10960 \�[0m rcsD o//third_party/python/python-stdlib-dirs.a
@Keithcat1 this error is due to a couple of unnecessary lines in python.mk. It goes away once #254 is merged.
I'm also having problems with make. Quite often, the make process it just randomly stops instead of completing the build, even though there doesn't seem to be an error.
@jart this happens when I make -j4 as well. The error is usually with the zoneinfo objects. My guess is it started with e963d9c8e3cb3d29e924c344f41975b63a62fef5, but I could be wrong.
@Keithcat1 The issues you encountered should be addressed by recent commits. I've made numerous other fixes and improvements too! For example, we now have really excellent completion. Here's a screencast demo.
@ahgamut I want to credit you somehow when we move forward with publicizing this contribution. One suggestion I have is the following language. Does it meet your approval?
Python 3.6.14+ (Actually Portable Python)
[GCC 9.2.0] on cosmo
Type "help", "copyright", "credits" or "license" for more information.
>>> credits()
Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
for supporting Python development. See www.python.org for more information.
Thanks go to github.com/ahgamut for porting Python to Cosmopolitan Libc.
Python 3.6.14+ (Actually Portable Python) [GCC 9.2.0] on cosmo Type "help", "copyright", "credits" or "license" for more information. >>> credits() Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands for supporting Python development. See www.python.org for more information. Thanks go to github.com/ahgamut for porting Python to Cosmopolitan Libc.
This is wonderful @jart. Thank you!
It should be possible to change pip so that it downloads packages and then stores them inside the python.com zip file, right. I know that extension modules aren't supported, but pure Python packages should be. By the way why are we using Python 3.6 and not Python 3.9? I'm guessing the attempt to port Python to Cosmopolitan was started a long time ago and just now finished. I'm a little confused about stackless threads. Do they let you utilize your CPU more fully than if you were just running one normal Python thread?
Yes pure Python packages will be supported, and they can be inside or outside the zip. Eventually native ones like Numpy will be supported too. @ahgamut did we remove that and need to add it back? To clarify, I want zip loading to be the first choice since it's the fast easy trademark nifty path. But if it's not in the zip, have it fall back to things like PYTHONPATH and current directory.
@Keithcat1
It should be possible to change pip so that it downloads packages
Using pip requires SSL, and Python relies on OpenSSL by default. OpenSSL has not yet been added to the Cosmopolitan repo.
OpenSSL bloats the APE considerably. It would be wonderful if we could wrap MBedTLS and use it for Python's _ssl module`.
The current option you have is to download the packages through some other method. Check the README on my cosmo_py36 fork to see what you can for pip right now.
An additional wrinkle is some Python packages rely on _thread and some glibc-specific stuff, so even if the installation works, the packages themselves raise complaints later.
and then stores them inside the python.com zip file, right.
I don't think the Python APE can self-modify just yet; I got an ETXTBSY when I tried to pip install directly into the APE ZIP store. So the modifications to pip might take a while. (@jart pls confirm)
There are two options:
- keep the downloaded packages in
~/.local/lib/python3.6/site-packagesor in the same directory as Python. - Python wheels are just zip files, so you can unzip them and add the necessary folders into the APE (possibly add as
.pycfiles).
I know that extension modules aren't supported
If you add the source and the right recipe to python.mk, simple C extensions like greenlet or markupsafe ought to work. I know because they are part of my cosmo_py36 fork of APE Python and I used them to run Flask.
Stuff like numpy is much harder, but likely still possible. There is scope for some interesting tricks with libpython.a.
By the way why are we using Python 3.6 and not Python 3.9? I'm guessing the attempt to port Python to Cosmopolitan was started a long time ago and just now finished.
Python 3.7 and above require pthread by default, which Cosmopolitan does not currently support. So I picked Python3.6. I would like to use the newest version also, but I do not know of a way to port them yet.
I'm a little confused about stackless threads. Do they let you utilize your CPU more fully than if you were just running one normal Python thread?
The stackless module has not been added to the Cosmopolitan repo yet. I only came across it a few days back.
I'm also not sure about how the stackless module works. I think they are similar to coroutines in Go? I tried out this old benchmark that compares Go's microthreads to Stackless with Stackless Python 3.6.13, and Go 1.15.6, and Stackless Python is faster within a single process (GOMAXPROCS=1). (additional detail about the benchmark here)
Stackless Python is apparently used in the servers for a MMORPG, which is why I thought it might be interesting speed-wise to add to the Python APE. Has anyone written a web framework with the stackless module?
did we remove that and need to add it back? To clarify, I want zip loading to be the first choice since it's the fast easy trademark nifty path. But if it's not in the zip, have it fall back to things like PYTHONPATH and current directory.
@jart Here are the entries in sys.path for the APE:
- is the current directory
''. This is added after startup, is default behavior for Python, and is for importing files in the current dir (like for simple scripts: if I hadfoo.pyandbar.pyin the current dir, andbar.pyhadimport foo, Python would use this entry and selectfoo.pyfrom nearby). You can put this after the ZIP store (by changing something insite.py) if you want. ~~but it would likely interfere with how people expect Python to behave.~~ - is the APE zip store. I think this is right is where it should be, it doesn't interfere with
'', and it is the fast path import for any script that isn't in the current dir of the interpreter. - is the user's directory for Python packages. This is added by
site.pyif the directory exists, and can be disabled viapython.com -s. Packages can be installed here bypip install --user <my_package>. I have used this to run some simple packages like Flask.
I don't think we need to add any more directories.
If a particular location is really necessary, we can add it to limited_search_path in Modules/getpath.c, or allow the use of PYTHONPATH/PYTHONHOME. I prefer to avoid PYTHONPATH/PYTHONHOME because it confuses the APE sometimes when I have multiple Python versions available, especially in Windows (for eg. if I am in a conda Python 3.7 venv, the APE would try and import a 3.7 lib).
(BTW the autocomplete in the APE is beautiful, I got a happy surprise when the text appeared in gray for completion)
@jart some decisions regarding python.com:
SSL support would allow the usage of pip and thereby install custom packages to the user's local directories.
There are three options:
- skip SSL for now, because
ensurepip,pip, and quite a few packages rely onthreading. Would be annoying if we allowedpipinstalls and then a bunch of packages had complaints about how the APE works (for e.g. Dash requires the glibc version during installation, why?!?!) - use OpenSSL: would need a PR for
third_party/opensslwith OpenSSL1.1.1k (it worked with the amalgamation earlier). I would prefer to avoid this because OpenSSL increases the APE size considerably, and I have not gotten any of the OpenSSL tests to pass (pip installworked on Linux though). - use MbedTLS: would need to write a version of
Modules/_ssl.candModules/_hashlib.cthat use MbedTLS bindings instead of OpenSSL. If this is possible, I think it is the best option, because APE size would not increase as much, and we can avoid adding OpenSSL to the repo. I tried to do this a while back, but I was unsuccessful.
Stackless Python adds a custom stackless module to the Python stdlib to provide lightweight coroutines.
- The coroutines seem pretty fast (see my comment here, for a simple benchmark
python.comwithstacklessbeats Go within a single process on my system). The syntax looks weird though. Stackless Python claims to be the backbone of a MMORPG, so maybe there is something to the speed claims. - I was able to compile
stacklesswithout any issue: had to resolve a few merge conflicts between the stackless fork of Python 3.6 and mycosmo_py36fork. You can see it here: https://github.com/ahgamut/cpython/tree/stackless-cosmo
Do you think the stackless module should be added to Cosmopolitan repo? We can try to make an existing web framework use stackless instead of threads to see speedup, or a possible bigger goal can be a new, stackless-based, Python web framework unique to Cosmopolitan.
I'm glad you enjoyed the readline interface. Definite no on stackless. We have better ways of simulating threads. Can't Flask do its thing using fork multiprocessing, just like gunicorn used to do? You know, if the goal is performance...
What I'd be interested in learning is whatever trick Pyston used to get the 30% performance boost. We obviously wouldn't want to incorporate their JIT work due to what happened with Unladen Swallow, but I seem to recall them writing a blog post where they said Python had some esoteric thing for debugging that no one really uses but made code really slow, so they removed it with great success. I'd like to know what that is.
How many kilobytes did OpenSSL add? It looks like it's 3mb on Alpine. What could it be doing? Our MbedTLS impl is more pareto optimized but you're still realistically looking at 200-500kb. Rewriting _ssl.c to integrate it with Python would be painful but it could be done.
It's also worth mentioning that incorporating chibicc or gcc+blinkenlights isn't out of the question, since it'd be nice to let people on any platform build their own native extensions without needing to install a compiler.
How many kilobytes did OpenSSL add?
OpenSSL adds around 2.2mb (from my comment here) to python.com.
Rewriting _ssl.c to integrate it with Python would be painful but it could be done.
Decoupling Python from OpenSSL is a known problem: there was an attempt with PEP 543 but it got withdrawn.
I'm not sure about adding OpenSSL to the repo. OpenSSL also has a 3.0.0 beta out now, so might be worth it to wait and see how the new version improves things.
Can't Flask do its thing using fork multiprocessing, just like gunicorn used to do? You know, if the goal is performance...
Flask relies on the threading module in imports, and also depends on greenlet (via Werkzeug IIRC). I haven't looked through the web framework options on Python to see if there's any library doesn't rely on threads. When I made this example GIF, I just did the minimum possible to get it to work. Notably, I did not try it in a production environment.
(python.com has been upgraded since I tried the Flask example, maybe I should try it again and see if there are any changes.)
The debugging changes made by Pyston are outlined here: https://github.com/pyston/pyston/wiki/Semantic-changes
Pyston is on Python 3.8 though, so I'm not sure if all the changes can be followed.
There's nothing stopping us from checking-in the code for commonly used native extensions, such as greenlet, and shipping it as part of python.com.
Well, commonly used depends on what workflows/packages are the (first) target with python.com.
I only looked up greenlet as a specific package when the Flask installation failed. If I had tried a different framework, perhaps I wouldn't consider greenlet that important.
numpy-dependent stuff is common for me, but Python is "general purpose": web frameworks, computational stuff, midsize scripts that are unwieldy in bash etc.
What would you like to target first? I'm not sure targeting a Python webserver+framework is worthwhile especially since redbean exists (unless redbean+python, but that would definitely reduce speed).
The ideal would be to have 95%+ of the stdlib in python.com, add a few "highly dependent" third_party extensions (need some way to figure these out) and allow pip + compilation from source (via chibicc?!) for any additional packages.
By the way, one question I forgot to answer earlier is that, it is possible to modify the zip store while the APE binary is running, but you need to call OpenExecutable which is a cool hack that's currently only being used by redbean. It currently only works on a couple operating systems and we're working to polyfill it across platforms with a slow copy fallback path. But generally speaking the zip executable, best expectation is to think of it as fast read only access for now.
What would you like to target first?
I want to get as many tests passing as possible first and incorporate them into the build. The Python unit tests are laying bare all the subtle shortcomings in the libc implementation which is great. Then I want to create an HTML page with green / yellow / red boxes in a table that gives people a birds eye overview of how complete Actually Portable Python has progressed. Once we have that we can launch, since it means happy users with realistic expectations and a call to action for anyone interested in helping.
@ahgamut When you say that extension modules are supported, do you mean that they have to be statically linked or that they can be loaded from disk? @jart The time.sleep function doesn't work, it raises an exception.
The tempfile module is broken, at least on Windows.
I wasn't able to build Python.com (make mode=dbg) with asan. The binary wouldn't run when started, just immediately exiting. I did get this error once though. error: could not map asan shadow memory
Can you fix the thing where make will sometimes stop?
I tried building Python.com with Clang and flto to make it faster / smaller: export CFLAGS=-flto export CXXFLAGS=-flto make -j8 MODE=llvm -O o/llvm/third_party/python But the build either kept stopping at some point fo r no reason or giving an error stating that one of the files in the build got too big, I don't have the exact error message.
Would it be possible to use PyPy instead of regular Cpython? I'm not really expecting you to do this but am curious if it would work. Does Cosmopolitan even support JIT code?
Thanks for all your hard work on this project, it's very exciting and I look forward to new stuff!
@Keithcat1
Extension modules are possible, but require a lot of effort right now. Extensions have to be linked statically, and you would need to make some changes in the source code of the extension in order to compile + link it along with libpython.a.
If you would like to experiment with how extensions are added to python.com, I would suggest using the my cosmo_py36 fork of CPython which contains all the necessary changes to add extensions statically via python's regular build process.
The repo is at: https://github.com/ahgamut/cpython/tree/cosmo_py36
The outline of the process to add extensions is at: https://github.com/ahgamut/cpython/tree/cosmo_py36#what-about-c-extensions
An example with the greenlet module and its _greenlet extension is in this commit: https://github.com/ahgamut/cpython/commit/0274a49934d4f8aa1fc98c10ebf15d694608e4c2
Extension modules are possible, but require a lot of effort right now.
I wish you wouldn't think of it that way. The people who maintain each platform put in a lot of work to create a prescripted path that's specific to their platform for installing python modules. For example, on Debian if you want Numpy you just apt install python-numpy. The tradeoff is that if you take the easy path like that, your program will only run on Debian. We've created a meta-platform of our own, that lets people build Python apps that not only run on every Linux distro, but all the other operating systems too. But since it's new, someone needs to do all that trailblazing which will make things easy for people in the future. We're the ones doing that.
@jart you are right, perhaps calling it "lot of effort" makes it seem unnecessarily daunting. If someone wanted Actually Portable CPython extensions (for e.g. _sqlite, greenlet, or markupsafe), for a few lines' worth of code, Cosmopolitan makes that possible today, which is amazing.
I think that customizing setuptools (via chibicc, dlopen, or converting setup.py to a Makefile recipe) can unlock a more hands-off approach, wherein we would not need to edit the source code at all. Combine that with a polyfilled OpenExecutable, and many more exciting things will happen.
@jart Would this help for adding shared object support? https://github.com/fancycode/MemoryModule I'm guessing loading shared objects or DLLs using the OS default functions isn't the hard part. Py2EXE actually had functionality that would bundle everything up in one zip file and load extension modules directly using the above C library.
@Keithcat1, I hope so. I already brought MemoryModule up in related discussions in https://github.com/jart/cosmopolitan/issues/97#issuecomment-874415031 and #137. I think the challenge is to not use the OS default functions if we want to be able to load APE-based extension modules.
The challenge isn't implementing the code loading. The challenge is what gets bundled, and what is loadable. cosmopolitan.a is 32mb with debug symbols stripped. It defines 4,000 functions.
So here's what we'd have to do first in order to move forward with a project like dynamic linking. We need an ABI requirements doc. We can study the most popular native extensions. Like numpy. Run readelf or nm to get the undef symbols they need, and make a list.
OK so we now have a pretty nice static analyzer for Python sources, which turns them into ELF objects. https://github.com/jart/cosmopolitan/blob/master/third_party/python/pyobj.c Stuff like this:
Symbol table '.symtab' contains 26 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 SECTION LOCAL DEFAULT 1
2: 0000000000000000 0 SECTION LOCAL DEFAULT 2
3: 0000000000000000 0 SECTION LOCAL DEFAULT 3
4: 0000000000000000 0 SECTION LOCAL DEFAULT 4
5: 0000000000000000 0 SECTION LOCAL DEFAULT 5
6: 0000000000000000 0 SECTION LOCAL DEFAULT 6
7: 0000000000000000 0 SECTION LOCAL DEFAULT 7
8: 0000000000000000 0 SECTION LOCAL DEFAULT 8
9: 0000000000000000 0 SECTION LOCAL DEFAULT 9
10: 0000000000000000 0 SECTION LOCAL DEFAULT 10
11: 0000000000000000 0 SECTION LOCAL DEFAULT 11
12: 0000000000000000 52 OBJECT LOCAL DEFAULT 4 zip+lfile:.python/antigravity.py
13: 0000000000000000 294 OBJECT LOCAL DEFAULT 5 zip+cdir:.python/antigravity.py
14: 0000000000000000 53 OBJECT LOCAL DEFAULT 7 zip+lfile:.python/antigravity.pyc
15: 0000000000000000 294 OBJECT LOCAL DEFAULT 8 zip+cdir:.python/antigravity.pyc
16: 0000000000000034 306 OBJECT GLOBAL DEFAULT 4 py:antigravity
17: 0000000000000035 547 OBJECT GLOBAL DEFAULT 7 pyc:antigravity
18: 0000000000000000 0 OBJECT GLOBAL HIDDEN UND pyc:webbrowser
19: 0000000000000000 0 OBJECT GLOBAL HIDDEN UND pyc:hashlib
20: 0000000000000000 0 OBJECT GLOBAL DEFAULT 10 pyc:antigravity.__file__
21: 0000000000000000 0 OBJECT GLOBAL DEFAULT 10 pyc:antigravity.webbrowser
22: 0000000000000000 0 OBJECT GLOBAL DEFAULT 10 pyc:antigravity.hashlib
23: 0000000000000000 0 OBJECT GLOBAL DEFAULT 10 pyc:antigravity.geohash
24: 0000000000000000 0 OBJECT GLOBAL HIDDEN UND .python/
25: 0000000000000000 0 OBJECT GLOBAL HIDDEN UND __zip_start
It works well enough that I'm learning some depressing things about the Python standard library. For example, the help() function depends on http.server. Right now the way the static analyzer is set up is we can make weird dependencies like that "weak" in elf terms by putting a try statement around them. That way we can make reasonably small binaries.
It seems that connecting to for example localhost:6060 using urllib.requests.urlretrieve doesn't work. urllib.error.URLError: <urlopen error [Errno 10051] ENETUNREACH[10051]>
There appears to be a regression the re module too. These issues will all be resolved as we get the unit tests integrated in the build.
You seem to have broken Python on Windows but not on WSL: Fatal Python error: Py_Initialize: can't initialize sys standard streams
Current thread 0x0000000000000000 (most recent call first): 7770000ffde0 00000069abda __die+0x3e 7770000ffdf0 000000683413 Py_FatalError+0x66 7770000ffe10 000000684088 _Py_InitializeEx_Private+0x414 7770000ffe50 0000006840f7 Py_InitializeEx+0x13 7770000ffe60 00000068410c Py_Initialize+0x13 7770000ffe70 0000004170b7 Py_Main+0x667 7770000fff80 00000040bd79 main+0x1bb 7770000fffe0 000000401890 cosmo+0x40 7770000ffff0 0000006b1f0a _jmpstack+0x17
Also, how does the size-reducing native-module-linkage thing you did work? I know that extension modules are compicated, but is it possible to get CTYPES working relatively easily?
Fatal Python error: Py_Initialize: can't initialize sys standard streams Current thread 0x0000000000000000 (most recent call first): 7770000ffde0 00000069abda __die+0x3e 7770000ffdf0 000000683413 Py_FatalError+0x66 7770000ffe10 000000684088 _Py_InitializeEx_Private+0x414 7770000ffe50 0000006840f7 Py_InitializeEx+0x13 7770000ffe60 00000068410c Py_Initialize+0x13 7770000ffe70 0000004170b7 Py_Main+0x667 7770000fff80 00000040bd79 main+0x1bb 7770000fffe0 000000401890 cosmo+0x40 7770000ffff0 0000006b1f0a _jmpstack+0x17
IIRC, this error is a delayed form of the classic Unable to get the locale encoding. I've seen this when I was getting _codecs/encodings to load on Windows in the first place.
The error at that point was the necessary codec module was not getting imported correctly in get_locale_encoding during startup, but the interpreter figures out that something is awry and crashes only later, when setting up sys.stdin/sys.stdout. Let me see if I can get Windows running and test if this is similar.
Thanks for the reports! That helps narrow things down. I should have a fix pushed shortly.
You asked earlier about how to use the new Python compiler. Right now it's able to be used via the makefile config. Earlier today I added a tutorial to the examples folder showing how an independent package can be configured to build a Python file into a 1.9mb static executable. Please take a look? https://github.com/jart/cosmopolitan/tree/master/examples/pyapp
https://github.com/jart/cosmopolitan/blob/51904e2687c04d7ae20410cd94c2148972d6bae6/examples/pyapp/pyapp.mk#L1-L116
As for ctypes, it'd be nice if Cosmopolitan supported dynamic linking, because it's frequently requested. However it's something I personally do not want for myself, so it's unlikely that I'll be implementing it myself. The focus has always been doing static linking better than anyone else. Cosmopolitan is sort of like FreeBSD/NetBSD/OpenBSD-style multitenant monorepo.
Any dependencies you need, we can add them to the third_party folder. It might even be possible using GitHub's .gitowners feature for me to simply authorize people to have their own folders within this repo that they can independently maintain. There's also the possibility of setting up a web GUI which could be the easier solution for folks who don't need the full power the make build provides.
OK I think I've finally figured out the optimal strategy for informing the PYOBJ.COM static analyzer about implicit dependencies. Basically it looks like this:
if __name__ == 'PYOBJ.COM':
AF_APPLETALK = 0
AF_ASH = 0
AF_ATMPVC = 0
AF_ATMSVC = 0
AF_AX25 = 0
AF_BRIDGE = 0
AF_CAN = 0
# ....
That way if a module like socket is doing a weird import star, we can still have it declare its dependencies, while making minimal changes to the upstream sources.
OK I think I've finally figured out the optimal strategy for informing the PYOBJ.COM static analyzer about implicit dependencies. Basically it looks like this:
@jart the above strategy is for statements like from _module import *. Import star statements use the __all__ attribute of the module to decide which objects to show, or fall back to (?) showing everything.
I am unclear on this: given a from _module import * statement, does PYOBJ.COM declare a symbol for
- just
_module(and then its components recursively), or - each element in the
*(likeAF_BRIDGE,AF_CANabove, but no recursion)?
I will read the source to find out, but an external explanation would also be useful.
Additional note: If we have to declare dependences with an if __name__ == 'PYOBJ'.com for import star statements, then we can also do the same for the importlib.import_module("module_str") or __import__("module_str") hacks which are seen in the Lib/test/test_something.py files.
Any dependencies you need, we can add them to the third_party folder. It might even be possible using GitHub's .gitowners feature for me to simply authorize people to have their own folders within this repo that they can independently maintain. There's also the possibility of setting up a web GUI which could be the easier solution for folks who don't need the full power the make build provides.
Adding a (pure-python) third_party package is as easy as adding the right folder to /zip/.python.
For example, I downloaded the plotext package separately, and manually added it to the APE ZIP store.
This is pure-python package with zero dependencies external dependencies, and it works via the APE on Linux:
No changes to the plotext source were required! This is a good sign, it means @jart's efforts to ensure compatible-with-CPython-yet-better behavior are working. (Windows is still an issue, APE erroring with os.py expecting posix.F_LOCK)
I will try to find more such simple packages to test APE compatibility behavior.
I think there are a few more places where Cosmopolitan would need to show compatibility with glibc/CPython (for example Lib/platform.py:_sys_version() needs a conditional for Cosmopolitan to avoid potential complaints.)
That's amazing @ahgamut. Also good news @Keithcat1! All the issues you brought to our attention should now be resolved by b5f743cdc384044299957bd94bb836eed2de26d4. I've confirmed that the HTTP client works on every single one of the operating systems we support. We now have 148 of Python's unit test programs incorporated into our build too, in order to ensure things stay fixed! That puts us at about 50% of the way there towards full automation.The tests run pretty fast too since they're fully parallelized:
Out of curiosity how long do the Python unit tests take when you run them the normal way? I seem to recall it taking a long time. Nice thing about small static binaries is that it makes the build / test latency 10x faster. It also means that all we need to do in order to test each program is write a script that does:
set -ex
for os in freebsd openbsd netbsd rhel7 rhel5 xnu win7 win10; do
scp test_stat.com $os: &&
ssh $os ./test_stat.com || exit
done
I like to think of it as the maximum consciousness approach to development since it makes life as a developer so much more pleasant for everyone involved over the long term, so we can have more impact focusing on the complexity that matters. So I'm sure Actually Portable Python will be a nice change of pace for anyone who feels like they're trapped in a tooling stack that feels like a messy bachelor's pad.
Anyway I think what we have is novel and definitely good enough for an alpha release. Please feel free to share any ideas on things like a logo, web design, or promotional strategies, since I can imagine a whole lot of people benefiting from this work.
I like to think of it as the maximum consciousness approach to development since it makes life as a developer so much more pleasant for everyone involved over the long term, so we can have more impact focusing on the complexity that matters. So I'm sure Actually Portable Python will be a nice change of pace for anyone who feels like they're trapped in a tooling stack that feels like a messy bachelor's pad.
Indeed, testing/debugging/updating APE Python from the cosmo build system is a breeze compared to what I was doing earlier with the regular Python repo.
Anyway I think what we have is novel and definitely good enough for an alpha release. Please feel free to share any ideas on things like a logo, web design, or promotional strategies, since I can imagine a whole lot of people benefiting from this work.
These are some points I noted down as "possible benefits of porting Python to Cosmopolitan Libc". Perhaps it can help you decide on strategies.
- APE Python is actually portable across operating systems.
- Python is slow -- APE Python is surely faster. Should we try benchmarking something?
- Py2 vs Py3 vs new features -- APE Python is 3.6 (not yet EOL), and right before too many new features
- Python is bloated, not for quick bash-like stuff -- APE Python is fast, statically-linked + PYOBJ.COM tree-shaking means we can have small binaries. I thought a
plotextanimation of observing some system tasks (ala Task Manager/htop/blinkenlights) or a nice TUI across systems might be nice examples. -
pipand installing arbitrary pure-Python third-party packages into the APE -- I tried this yesterday, but I haven't understoodOpenExecutableclearly. - Many third-party packages depend on Python's
threadingmodule. Options are: do nothing, spoofthreadingwithdummy_threading(ouch), Stackless Python (rejected earlier), spoof viagreenlet(have not tested fully), or any alternate method. - actually portable CPython extensions -- we know it can be done, possibly via multiple methods (full-fledged dynamic linking,
MemoryModuleas mentioned by @Keithcat1 and @pkulchenko, adding the extension source alagreenletto the Python stdlib in the repo). Would porting (a subset of) Numpy to APE Python be worth it? I found a minimalpiconumpy, but it is too small and really easy to port. Let me see.
The reason I wanted CTYPES is because it would allow me to use closed-source libraries like http://www.un4seen.com/bass.html which is good for playing audio but can't be statically linked. It would also make it easier to deal with libraries like SDL2 that might not be compilable with Cosmopolitan for a while. Lastly, it would make it possible for APE Python to just download prebuilt wheels for Windows, Mac and Linux and then load the right DLLs depending on which platform the APE binary is running on.
@jart libc/complex.h declares functions for use with complex float/complex double, but I get a linker error when trying to use csqrt.
Are these functions implemented? If not, I'd like to try implementing some. Is there a reference I can use?
@jart bottle.py is a minimal web framework that may be useful if added to the repo. The package is a single file, and I confirmed that the initial example works if I comment out disable import _thread in the source file.
https://bottlepy.org/docs/0.12/
What I'd be interested in learning is whatever trick Pyston used to get the 30% performance boost. We obviously wouldn't want to incorporate their JIT work due to what happened with Unladen Swallow, but I seem to recall them writing a blog post where they said Python had some esoteric thing for debugging that no one really uses but made code really slow, so they removed it with great success. I'd like to know what that is.
@jart with #264 and #281 we've disabled all the debugging elements mentioned in the Pyston wiki:
- [x]
_Py_CheckFunctionResult - [x]
PyErr_BadInternalCall - [x] tracemalloc
- [x] allocator hooks -- Py36 doesn't have that much allocator-related code compared to Py37+
- [x] recursion checking -- Pyston added back recursion checking, but using the C stack pointer. This change was made after I added #264 (didn't update my local version of Pyston!). How does Cosmopolitan handle stack overflows?
The other areas when Pyston has speedups are either JIT or implementation-defined. Let me see if any of those have equivalents in Cosmopolitan.
Trying some benchmarks might be fun if we have something better than dummy_threading.
Stack overflow checking is relatively straightforward with Cosmopolitan.
forceinline int Py_EnterRecursiveCall(const char *where) {
const char *rsp, *bot;
extern char ape_stack_vaddr[] __attribute__((__weak__));
if (!IsTiny()) {
rsp = __builtin_frame_address(0);
asm(".weak\tape_stack_vaddr\n\t"
"movabs\t$ape_stack_vaddr+12288,%0" : "=r"(bot));
if (UNLIKELY(rsp < bot)) {
PyErr_Format(PyExc_MemoryError, "Stack overflow%s", where);
return -1;
}
}
return 0;
}
I've added this in 7521bf9e73d340eb36d76f3938eb29e07278ee69.
Are there any simple JIT implementation defined things it's doing that yield big benefits?
Complex math functions like csqrt() need to be ported from Musl. Contributions welcome. Bottle is welcome too.
The reason I wanted CTYPES is because it would allow me to use closed-source libraries like http://www.un4seen.com/bass.html which is good for playing audio but can't be statically linked. It would also make it easier to deal with libraries like SDL2 that might not be compilable with Cosmopolitan for a while. Lastly, it would make it possible for APE Python to just download prebuilt wheels for Windows, Mac and Linux and then load the right DLLs depending on which platform the APE binary is running on.
ctypes isn't able to function as a stopgap since it only works if you're integrating with the platform-specific userspace tooling. Native libraries need to be checked-in to the cosmo third_party directory and linked statically. Otherwise you can interop with external native code via subprocesses, which should serve as a reasonable stopgap.
What we should do is provide an alternative to the platform specific tooling, by integrating chibicc. That way, rather than dropping .so files into your python modules dir you would instead drop .c files, and the Python runtime will compile them and generate bindings automatically, and it would do so in a manner that behaves precisely the same across platforms.
Are there any simple JIT implementation defined things it's doing that yield big benefits?
They've mentioned something like "reducing comparisons for attribute lookups" but I haven't looked at those parts of the source just yet.
Complex math functions like csqrt() need to be ported from Musl. Contributions welcome. Bottle is welcome too.
Bottle is a single file. complex math from musl let me check.
Complex functions are nice to have; the reason I was asking about them is because I've gotten quite close in my attempts to get Numpy into APE Python. Everything compiles, but linker is missing symbols -- probably I've mixed up porting some recipes or I am adding unnecessary files into the Makefile.
- numpy requires Cython to convert some files into C for the build. It might be useful to add Cython in the monorepo for optimization (
pyobj.com+ Cython outputs a C file to be compiled) or potential Cython+chibicc magic for extensions - some parts of numpy have complaints if built with
-DPy_BUILD_CORE - numpy requires some kind of BLAS to build -- they provide a "lite" version along with the source, but they warn that it's pretty slow. There is some conditional compilation involved with
setup.pythat I haven't understood fully (something like if external BLAS, don't compile these files). - speaking of BLAS, it is possible to build the
libblas.aandlibcblas.afrom LAPACK via the Cosmopolitan build system. LAPACK requireslibgfortranthough.
How much slower could its BLAS subroutines be? LAPACK is a good library, but I'll need to check-in a FORTRAN compiler. We used to have a FORTRAN standard library checked-in to third party. LAPACK is underwhelming compared to MKL which sadly we can't use since it doesn't respect the freedom to hack. From what I hear BLIS is the closest alternative that meets our licensing requirements. https://github.com/flame/blis/blob/master/docs/graphs/large/l3_perf_skx_nt1.pdf So we might want to cherry-pick some of its subroutines to accellerate things like matrix multiplication.
If Cython works by generating a .c file and handing it over to the gcc command then we should totally include it. I can update it to integrate with an embedded chibicc runtime once it's ready.
-DPy_BUILD_CORE
I don't like that define because it makes commands like make o//third_party/python/Python/errors.ss break. I wish it wasn't needed and I've been considering having it be a define at the top of each .c file that needs it, similar to PY_SSIZE_T_CLEAN.
LAPACK is a good library, but I'll need to check-in a FORTRAN compiler.
The vendored gcc executable in the repo has gfortran built-in? I just used the OBJECTIFY.f rules in build/rules.mk and it worked.
We used to have a FORTRAN standard library checked-in to third party.
I don't know what functions libgfortran provides, yesterday the LAPACK build needed string concatenation (there is a // b string concatenation operationin FORTRAN.
If Cython works by generating a .c file and handing it over to the gcc command then we should totally include it.
Cython just handles the Python -> C conversion (+ it has some additional syntax that help in optimizing said conversion). The generated C file can be used for anything.
@jart the documentation for BLIS says it requires pthreads, even with mulltithreading disabled. pthreads is currently not implemented in Cosmopolitan, so I'll look into this at a later time.
Reading this maybe we should use OpenBLAS with DYNAMIC_ARCH=1 with the exception of MODE=opt since that uses -march=native. https://news.ycombinator.com/item?id=28736136
That discussion is a little handwavy. People who use Linux as a desktop will never see eye-to-eye with people who simply want sweet sweet unadulterated performance that Microsoft and Apple would never in a million years offer, since it makes GUIs slow. Anyway our luck is about to change. See https://github.com/jart/cosmopolitan/pull/282
Cython is a tool that allows creating Python extension modules by converting Pythonic code into a C file which can call between C and Python. Might it be worth allowing to import .pyx (Cython files) which internally converts them to C files and then compiles them? Since it would already be an extension module, no bindings need to be generated since Cython does that already. Also when you say using external subprocesses, do you mean compiling one executable file for each platform that my Cosmo app will run on and then send messages to and from it? It seems clunky, but thinking about it it could be tough to for example get integer sizes right on all platforms.
No only the compiler would run as a subprocess. (At least until we can retool chibicc to not rely on _exit() to free() its memory). It would use the same ABI for all platforms. One compilation. Otherwise it's not build once run anywhere.
@jart one possible way to add many small speedups is to use METH_FASTCALL instead of METH_VARARGS when specifying the arguments for a CPython function (a Python function written in C). METH_FASTCALL prevents creation of an unnecessary Python tuple of args, and instead just uses an array of PyObject *. The tradeoff is that there is some boilerplate to write to use METH_FASTCALL properly.
METH_FASTCALL went through a lot of changes in Python 3.7 (git log -i --grep="FASTCALL" between Python v3.7.12 and v3.6.15), and is much faster + nicer to use.
METH_FASTCALL is still considered an internal detail of Python 3.7 -- I hope this means it can be added to APE Python without any major compatibility issues.
Example of manually adding METH_FASTCALL (both the below commits cannot be ported to the monorepo ATM, because Py_Arg_UnpackStack and a bunch of other internals need to be moved first):
- python/cpython@84b388bb809
- python/cpython@fda6d0acf0
An automated way of adding METH_FASTCALL in Python is to use the Argument Clinic generator and generate the necessary clinic/*.inc files.
I tried using Argument Clinic + METH_FASTCALL in 3.6 in https://github.com/ahgamut/cpython/commit/2b417a690735b8f004257eddc526ece66dd135b4 for a few methods. The related tests pass, but no noticeable speedup.