Update python libraries and tests from CPython 3.11

Open youknowone opened this issue 1 year ago • 7 comments

Though it will be sometimes blocked by rust side issues, don't worry! Please just create PR about the broken test and and create issues related to the new bugs.

If you are new to updating cpython libraries, this guide will be useful: https://github.com/RustPython/RustPython/wiki/How-to-update-test-files

When updating module Lib/X, please don't forget to also update Lib/test/test_X

A few libraries doesn't need to be updated because CPython also kept it same as previous version. In that case, please leave a comment to let us mark it as done. Thank you!

Tips

To get specific version of CPython source code, try git checkout v3.11.2 from CPython repository.
Please check #4064 for example. Not the entire changes but the each commits. One of the middle size example is https://github.com/RustPython/RustPython/commit/17e12dea1ec11b871a9e5d3a4a26e2f9ad83fde2 . Sometimes it can be really simple like https://github.com/RustPython/RustPython/pull/4064/commits/9571a68c01284191d4b18229694c303dad8992ff
Please create one PR for one library+test pair whenever possible. Once you stuck by dependencies, go to one of the dependency first and finish it before restarting the stuck one unless they become simpler to be updated at the same time. Normally adding more libraries make passing tests (a lot) harder.
To run tests, try cargo run --features ssl -- -m test -v <test_name>. e.g. If you changed test_unicode.py, cargo run --features ssl -- -m test -v test_unicode
- If it runs too slow, try to add --release like cargo run --release --features ssl -- -m test -v test_unicode

List of libraries

[ ] __future__.py
- [ ] maybe related to #4060?
[x] __hello__.py #4570
[x] __phello__ #4570
[ ] _collections_abc.py
[ ] _compat_pickle.py
[ ] _compression.py #4688
[ ] _markupbase.py
[x] ~~_osx_support.py~~ unchanged
- [x] #4670
[ ] _py_abc.py
[ ] _pycodecs.py
[ ] _pydecimal.py
[ ] _pyio.py
[ ] _sitebuiltins.py
[ ] _threading_local.py
[ ] _weakrefset.py
[ ] abc.py
[ ] aifc.py
[x] antigravity.py — #4591
[x] argparse.py — #4592
[ ] ast.py
[ ] asynchat.py
[ ] asyncio
- [ ] #3118
- [ ] #3859
[ ] asyncore.py
[ ] base64.py
[ ] bdb.py
[ ] binhex.py
[ ] bisect.py
[ ] bz2.py
[x] calendar.py #4669
[ ] cgi.py
[ ] cgitb.py
[ ] chunk.py
[ ] cmd.py
[ ] code.py
[ ] codecs.py
- See also #3861
[ ] codeop.py
[ ] collections
- See also #3418
[ ] colorsys.py
[ ] compileall.py
[ ] concurrent
[x] configparser.py #4595
[x] contextlib.py #4649 #4659
[ ] contextvars.py
- [ ] #3188
[x] copy.py https://github.com/RustPython/RustPython/pull/4674
[ ] copyreg.py
[ ] csv.py
[ ] ctypes
- [ ] #1217
- See also #2364
[ ] dataclasses.py
[ ] datetime.py
[ ] dbm
[ ] decimal.py
[ ] difflib.py
[ ] dis.py
- see also #3846
[ ] distutils
[ ] doctest.py
[ ] email
[ ] encodings
[x] ensurepip #4590
[ ] enum.py
[ ] filecmp.py
[ ] fileinput.py
[ ] fnmatch.py
[ ] formatter.py
[ ] fractions.py
[ ] ftplib.py
[ ] functools.py
[ ] gc.py
- #4158
[ ] genericpath.py
[ ] getopt.py
[ ] getpass.py
[ ] gettext.py
[ ] glob.py
[ ] graphlib.py
[ ] gzip.py #4688
[ ] hashlib.py
[ ] heapq.py
[ ] hmac.py
[ ] html
[ ] http
[ ] imghdr.py
[ ] imp.py
[x] importlib #4561
- [x] #4559
- #4565
[ ] inspect.py
[ ] io.py
- [ ] #3960
- [ ] #4662
- See also #4072
[ ] ipaddress.py
[ ] json
[ ] keyword.py
[ ] linecache.py
[x] locale.py #4537
- See also #3850
[ ] logging
[ ] mailbox.py
- [ ] #4072
[ ] mimetypes.py
[ ] multiprocessing
- See also #3965
[ ] netrc.py
[ ] nntplib.py
[ ] ntpath.py
[ ] nturl2path.py
[ ] numbers.py
[ ] opcode.py
[ ] operator.py
[ ] optparse.py
[ ] os.py
- [ ] #3960
- See also #4053
[ ] pathlib.py
- [ ] #3960
[ ] pdb.py
[ ] pickle.py
- See also #3876
[ ] pickletools.py
[ ] pkgutil.py
[ ] platform.py
[ ] plistlib.py
[ ] posixpath.py
[ ] pprint.py
[ ] pty.py
[ ] py_compile.py
[ ] pydoc.py
- [ ] pydoc_data
[ ] queue.py
- See also #3608
[ ] quopri.py
[ ] random.py
[ ] re.py
[ ] reprlib.py
[ ] rlcompleter.py
[ ] runpy.py
[ ] sched.py
[ ] secrets.py
[ ] selectors.py
[ ] shelve.py
[ ] shlex.py
[ ] shutil.py
- [ ] #3960
[ ] signal.py
[ ] site.py
- [ ] #4662
- #4550
[ ] smtpd.py
[ ] smtplib.py
[ ] sndhdr.py
[ ] socket.py
[ ] socketserver.py
[ ] sqlite3
[ ] sre_compile.py
[ ] sre_constants.py
[ ] sre_parse.py
[ ] ssl.py
- See also #3927
[ ] stat.py
[ ] statistics.py
[ ] string.py
[ ] stringprep.py
[ ] struct.py
[ ] subprocess.py
[ ] sunau.py
[x] sysconfig.py #4569
[ ] tabnanny.py
[ ] tarfile.py
[ ] telnetlib.py
[ ] tempfile.py
[ ] textwrap.py
[ ] this.py
[ ] threading.py
[ ] timeit.py
[ ] token.py
[ ] tokenize.py
[ ] tomllib
[x] trace.py #4586
[ ] traceback.py
[ ] tty.py
[ ] types.py
[ ] typing.py
[x] unittest #4560
[ ] urllib
[ ] uu.py
[ ] uuid.py
[ ] venv
- [ ] #3960
[ ] warnings.py
- [ ] #4013
[ ] weakref.py
[ ] webbrowser.py
[ ] wsgiref
[ ] xdrlib.py
[ ] xml
[ ] xmlrpc
[ ] zipapp.py
[ ] zipfile.py
[ ] zipimport.py
[x] test.support #4537
- [x] #4540
- [ ] #4538
- [x] #4539
- [ ] #4541

List of tests without python libraries

[ ] test/test___all__.py
[ ] test/test__locale.py
[ ] test/test__opcode.py
[ ] test/test_abstract_numbers.py
[ ] test/test_array.py
- See also #3876
[ ] test/test_asdl_parser.py
[ ] test/test_asyncgen.py
- [ ] #3859
[x] test/test_atexit.py #4621
[ ] test/test_audioop.py
[ ] test/test_audit.py
[x] ~~test/test_augassign.py~~ unchanged
[x] test/test_baseexception.py — #4624
[ ] test/test_bigaddrspace.py
[x] test/test_bigmem.py #4625
[ ] test/test_binascii.py
- See also #4374
[x] ~~test/test_binop.py~~ unchanged
[x] test/test_bool.py #4631
[x] test/test_buffer.py #4632
[x] ~~test/test_bufio.py~~ unchanged
[ ] test/test_builtin.py
[ ] test/test_bytes.py
[ ] test/test_bz2.py
[ ] test/test_c_locale_coercion.py
[ ] test/test_call.py
[ ] test/test_charmapcodec.py
[ ] test/test_check_c_globals.py
[ ] test/test_class.py
[ ] test/test_clinic.py
[ ] test/test_cmath.py
[ ] test/test_cmd_line.py
[ ] test/test_cmd_line_script.py
[ ] test/test_code_module.py
[ ] test/test_codeccallbacks.py
[ ] test/test_compare.py
[ ] test/test_compile.py
[ ] test/test_complex.py
[ ] test/test_contains.py
[ ] test/test_context.py
[ ] test/test_contextlib_async.py
[ ] test/test_coroutines.py
[ ] test/test_crashers.py
[ ] test/test_csv.py
[ ] test/test_decorators.py
[ ] test/test_defaultdict.py
[ ] test/test_deque.py
[ ] test/test_descr.py
[ ] test/test_descrtut.py
[ ] test/test_devpoll.py
[ ] test/test_dict.py
[ ] test/test_dict_version.py
[ ] test/test_dictcomps.py
[ ] test/test_dictviews.py
[ ] test/test_docxmlrpc.py
[ ] test/test_dtrace.py
[ ] test/test_dynamic.py
[ ] test/test_dynamicclassattribute.py
[ ] test/test_eintr.py
[ ] test/test_embed.py
[ ] test/test_enumerate.py
[ ] test/test_eof.py
[ ] test/test_epoll.py
[ ] test/test_errno.py
[ ] test/test_except_star.py
[ ] test/test_exception_group.py
[ ] test/test_exception_hierarchy.py
[ ] test/test_exception_variations.py
[ ] test/test_exceptions.py
[ ] test/test_extcall.py
[ ] test/test_faulthandler.py
[ ] test/test_fcntl.py
[ ] test/test_file.py
[ ] test/test_file_eintr.py
[ ] test/test_fileio.py
[ ] test/test_fileutils.py
[ ] test/test_finalization.py
[ ] test/test_float.py
[ ] test/test_flufl.py
[ ] test/test_fork1.py
[ ] test/test_format.py
[ ] test/test_frame.py
[ ] test/test_frozen.py
[ ] test/test_fstring.py
[ ] test/test_funcattrs.py
[ ] test/test_future.py
[ ] test/test_future3.py
[ ] test/test_future4.py
[ ] test/test_future5.py
[ ] test/test_gdb.py
[ ] test/test_generator_stop.py
[ ] test/test_generators.py
[ ] test/test_genericalias.py
[ ] test/test_genericclass.py
[ ] test/test_genexps.py
[ ] test/test_getpath.py
[ ] test/test_global.py
[ ] test/test_grammar.py
[ ] test/test_grp.py
[ ] test/test_hash.py
[ ] test/test_hashlib.py
[ ] test/test_index.py
[ ] test/test_int.py
[ ] test/test_int_literal.py
[ ] test/test_interpreters.py
[ ] test/test_ioctl.py
[ ] test/test_isinstance.py
[ ] test/test_iter.py
- [ ] #3960
[ ] test/test_iterlen.py
[ ] test/test_itertools.py
[ ] test/test_keywordonlyarg.py
[ ] test/test_kqueue.py
[ ] test/test_largefile.py
[ ] test/test_launcher.py
[ ] test/test_list.py
- See also #1750
[ ] test/test_listcomps.py
[ ] test/test_lltrace.py
[ ] test/test_long.py
[ ] test/test_longexp.py
[ ] test/test_marshal.py
- See also #3458
[ ] test/test_math.py
[ ] test/test_memoryio.py
[ ] test/test_memoryview.py
[ ] test/test_metaclass.py
[ ] test/test_minidom.py
[ ] test/test_mmap.py
- See also #3847
[ ] test/test_module.py
[ ] test/test_multibytecodec.py
[ ] test/test_named_expressions.py
[ ] test/test_nis.py
[ ] test/test_numeric_tower.py
[ ] test/test_opcache.py
[ ] test/test_openpty.py
[ ] test/test_ordered_dict.py
[ ] test/test_ossaudiodev.py
[ ] test/test_osx_env.py
[ ] test/test_patma.py
[ ] test/test_peepholer.py
[ ] test/test_pep646_syntax.py
[ ] test/test_picklebuffer.py
[ ] test/test_pkg.py
[ ] test/test_poll.py
[ ] test/test_popen.py
[ ] test/test_positional_only_arg.py
[ ] test/test_posix.py
- See also #4494
[ ] test/test_pow.py
[ ] test/test_print.py
[ ] test/test_property.py
- See also #4067
[ ] test/test_pulldom.py
[ ] test/test_pwd.py
[ ] test/test_pyexpat.py
- See also #3430
[ ] test/test_raise.py
[ ] test/test_range.py
[ ] test/test_readline.py
[ ] test/test_regrtest.py
[ ] test/test_repl.py
[ ] test/test_resource.py
[ ] test/test_richcmp.py
[ ] test/test_robotparser.py
[ ] test/test_sax.py
[ ] test/test_scope.py
[ ] test/test_script_helper.py
[ ] test/test_select.py
[ ] test/test_set.py
- See also #3992
[ ] test/test_setcomps.py
[ ] test/test_slice.py
[ ] test/test_smtpnet.py
[ ] test/test_sort.py
[ ] test/test_source_encoding.py
[ ] test/test_spwd.py
[ ] test/test_stable_abi_ctypes.py
[ ] test/test_startfile.py
[ ] test/test_strftime.py
[ ] test/test_string.py
[ ] test/test_string_literals.py
[ ] test/test_strptime.py
[ ] test/test_strtod.py
[ ] test/test_structseq.py
See also #4063
[ ] test/test_subclassinit.py
[ ] test/test_sundry.py
[ ] test/test_super.py
- See also #3865
[x] test/test_support.py
- [ ] #4538
- [x] #4539
[x] test/test_syntax.py #4602
- [ ] #4479
[x] test.test_sys #4569
- [x] #4123
- See also #3859
- See also #4541
[ ] test/test_sys_setprofile.py
[ ] test/test_sys_settrace.py
[x] test/test_syslog.py #4569
[ ] test/test_tcl.py
[ ] test/test_thread.py
[ ] test/test_threadedtempfile.py
[ ] test/test_threadsignals.py
[ ] test/test_time.py
- See also #3850
- See also #4157
[ ] test/test_timeout.py
[ ] test/test_tix.py
[ ] test/test_tuple.py
[ ] test/test_type_annotations.py
[ ] test/test_type_cache.py
[ ] test/test_type_comments.py
[ ] test/test_typechecks.py
[x] test/test_types.py #4683
[ ] test/test_ucn.py
[ ] test/test_unary.py
[ ] test/test_unicode.py
[ ] test/test_unicode_file.py
[ ] test/test_unicode_file_functions.py
[ ] test/test_unicode_identifiers.py
[x] test/test_unicodedata.py #4678
[x] ~~test/test_univnewlines.py~~ unchanged
[x] test/test_unpack.py #4676
[ ] test/test_unpack_ex.py
[ ] test/test_unparse.py
[x] test/test_userdict.py #4666
[x] test/test_userlist.py #4666
[x] test/test_userstring.py #4658
[x] ~~test/test_utf8_mode.py~~ unchanged
[x] ~~test/test_utf8source.py~~ unchanged
[ ] test/test_wait3.py
[ ] test/test_wait4.py
[ ] test/test_weakset.py #4657
[ ] test/test_winconsoleio.py
[ ] test/test_winreg.py
[ ] test/test_winsound.py
[x] test/test_with.py #4648
[x] ~~test/test_yield_from.py~~ unchanged
[x] test/test_zlib.py #4647
[ ] test/test_asyncio
[ ] test/test_capi
[x] test/test_import #4599
[ ] test/test_tools

Not yet added libraries

These libraris are not added yet. Pure python one will be possible while others are not.

[ ] _bootsubprocess.py
[ ] _pycodecs.py
[ ] cProfile.py
[ ] crypt.py
[ ] curses
[ ] idlelib
[ ] imaplib.py
[ ] imp.py
- [ ] See also #4075
[ ] lzma.py
[ ] mailcap.py
[ ] modulefinder.py
- [ ] #3846
[ ] msilib
[ ] pipes.py
[ ] poplib.py
[ ] pstats.py
[ ] pyclbr.py
[ ] symtable.py
[ ] tkinter
[ ] tracemalloc.py
[ ] turtle.py
[ ] wave.py
[ ] zoneinfo

Feb 24 '23 12:02 youknowone

Updating asyncio is going to require fleshing out contextvars. (See #3188.)

Feb 25 '23 16:02 fanninpm

Pasting the comment (https://github.com/RustPython/RustPython/pull/4592#pullrequestreview-1318321535) here too for anyone else who might need it:

You can use the following command to test your changes locally (in a manner similar to RustPython's CI):

# this runs all of the tests (not necessary if you know which test is affected)
cargo run --release --features ssl,jit -- -m test -j 1 -u all --slowest --fail-env-changed -v
# this runs only the test suite named "test_whatever" (usually located at `Lib/test/test_whatever.py`)
cargo run --release --features ssl,jit -- -m test -j 1 -u all --slowest --fail-env-changed -v test_whatever

For a list of all resources and more command-line options, you can execute cargo run --release --features ssl,jit -- -m test -h

Feb 28 '23 22:02 DimitrisJim

The test/test_augassign.py file is unchanged.

Mar 04 '23 16:03 dalinaum

test/test_binop.py is unchanged. test/test_bufio.py is unchanged.

Mar 04 '23 18:03 dalinaum

test_yield_from.py is unchanged.

Mar 07 '23 14:03 dalinaum

test/test_utf8source.py is unchanged. test/test_utf8_mode.py is unchanged.

Mar 07 '23 15:03 dalinaum

test_univnewlines.py remains unchanged.

Mar 09 '23 18:03 dalinaum

test_html.py and test_htmlparser.py remain unchanged.

Mar 17 '23 05:03 tdub0

test_unicode_identifiers.py remains unchanged.

Mar 23 '23 17:03 dalinaum

test_contains remain unchanged.

Mar 25 '23 07:03 JaeyoungAhn

test_int_literal.py remain unchanged

Mar 25 '23 08:03 Snowapril

test\test_operators.py remain unchanged. operators.py remain unchanged.

Mar 25 '23 08:03 jyj0816

test_file.py test_charmapcodec.py test_cmath.py test_code_module.py test_codeccallbacks.py test_defaultdict.py test_descr.py test_dictcomps.py test_dynamicclassattribute.py test_errno.py test_fcntl.py test_file.py - unchanged

Apr 02 '23 15:04 Masorubka1

test_future3.py test_future4.py test_future5.py test_index.py test_ioctl.py test_isinstance.py test_iterlen.py test_keywordonlyarg.py test_longexp.py test_memoryio.py test_memoryview.py test_named_expressions.py test_openpty.py - unchanged

test_launcher.py - Cpython specific

Apr 04 '23 10:04 Masorubka1

bisect.py was updated in https://github.com/RustPython/RustPython/pull/4774 , but its corresponding check box is unchecked

Apr 08 '23 11:04 MegasKomnenos

Thanks to everyone who participated in this series of patches! I am going to close this issue in favor of #5104

Oct 23 '23 03:10 youknowone

RustPython RustPython copied to clipboard

Update python libraries and tests from CPython 3.11

List of libraries

List of tests without python libraries

Not yet added libraries

RustPython
RustPython copied to clipboard