truffleruby
truffleruby copied to clipboard
Ruby 3.2 support
We are working on Ruby 3.2 support for the next release (23.1).
Any help is appreciated whether it is:
- implementing one of the Ruby 3.2 changes in TruffleRuby, and ensuring it is tested in specs or MRI tests and otherwise add specs. If adding specs please leave a comment here to mention you are working on adding these specs to avoid conflicts & duplicated work.
- adding specs for Ruby 3.2 changes, which is best done in ruby/spec directly, see https://github.com/ruby/spec/issues/1016
To make it easier to find good getting-started issues, I annotated items with easy
, medium
and hard
.
There is also pure ruby
for changes which should only affect Ruby code and not require any change to Java sources or any Java knowledge.
If there is no pure ruby
then some Java code is involved, but it should still be fairly straightforward if easy
or medium
.
hard
typically requires more knowledge about TruffleRuby implementation details.
[import]
means this needs the import of Ruby 3.2 sources to happen first.
If you'd like to start on some task please mention it here or edit the description directly for committers (to avoid duplicated work).
Currently RUBY_VERSION is still 3.1.x. So to work on 3.2 changes, follow this documentation.
Full list of changes for Ruby 3.2.0
NOTE: https://rubyreferences.github.io/rubychanges/3.2.html gives more details for many features and changes.
From https://github.com/ruby/ruby/blob/master/doc/NEWS/NEWS-3.2.0.md:
This document is a list of user-visible feature changes since the 3.1.0 release, except for bug fixes.
Note that each entry is kept to a minimum, see links for details.
Language changes
-
[x] [parser, YARP] Anonymous rest and keyword rest arguments can now be passed as arguments, instead of just used in method parameters. [Feature #18351]
def foo(*) bar(*) end def baz(**) quux(**) end
-
[x] [medium, java] A proc that accepts a single positional argument and keywords will no longer autosplat. [Bug #18633]
proc{|a, **k| a}.call([1, 2]) # Ruby 3.1 and before # => 1 # Ruby 3.2 and after # => [1, 2]
-
[ ] [medium, java] (@andrykonchin) Constant assignment evaluation order for constants set on explicit objects has been made consistent with single attribute assignment evaluation order. With this code:
foo::BAR = baz
foo
is now called beforebaz
. Similarly, for multiple assignments to constants, left-to-right evaluation order is used. With this code:foo1::BAR1, foo2::BAR2 = baz1, baz2
The following evaluation order is now used:
-
foo1
-
foo2
-
baz1
-
baz2
-
-
[x] [parser, YARP, need to implement Find pattern in translator] "Find pattern" is no longer experimental. [Feature #18585]
-
[x] @eregon Methods taking a rest parameter (like
*args
) and wishing to delegate keyword arguments throughfoo(*args)
must now be marked withruby2_keywords
(if not already the case). In other words, all methods wishing to delegate keyword arguments through*args
must now be marked withruby2_keywords
, with no exception. This will make it easier to transition to other ways of delegation once a library can require Ruby 3+. Previously, theruby2_keywords
flag was kept if the receiving method took*args
, but this was a bug and an inconsistency. A good technique to find the potentially-missingruby2_keywords
is to run the test suite, for where it fails find the last method which must receive keyword arguments, useputs nil, caller, nil
there, and check each method/block on the call chain which must delegate keywords is correctly marked asruby2_keywords
. [Bug #18625] [Bug #16466]def target(**kw) end # Accidentally worked without ruby2_keywords in Ruby 2.7-3.1, ruby2_keywords # needed in 3.2+. Just like (*args, **kwargs) or (...) would be needed on # both #foo and #bar when migrating away from ruby2_keywords. ruby2_keywords def bar(*args) target(*args) end ruby2_keywords def foo(*args) bar(*args) end foo(k: 1)
Core classes updates
Note: We're only listing outstanding class updates.
Fiber
-
[ ] [medium, java] Introduce Fiber.[] and Fiber.[]= for inheritable fiber storage. Introduce Fiber#storage and Fiber#storage= (experimental) for getting and resetting the current storage. Introduce
Fiber.new(storage:)
for setting the storage when creating a fiber. [Feature #19078]Existing Thread and Fiber local variables can be tricky to use. Thread-local variables are shared between all fibers, making it hard to isolate, while Fiber-local variables can be hard to share. It is often desirable to define unit of execution ("execution context") such that some state is shared between all fibers and threads created in that context. This is what Fiber storage provides.
def log(message) puts "#{Fiber[:request_id]}: #{message}" end def handle_requests while request = read_request Fiber.schedule do Fiber[:request_id] = SecureRandom.uuid request.messages.each do |message| Fiber.schedule do log("Handling #{message}") # Log includes inherited request_id. end end end end end
You should generally consider Fiber storage for any state which you want to be shared implicitly between all fibers and threads created in a given context, e.g. a connection pool, a request id, a logger level, environment variables, configuration, etc.
Fiber::Scheduler
- [ ] [needs other fiber scheduler parts first] Introduce
Fiber::Scheduler#io_select
for non-blocking IO.select. [Feature #19060]
IO
-
[x] [hard, java] Introduce IO#timeout= and IO#timeout which can cause IO::TimeoutError to be raised if a blocking operation exceeds the specified timeout. [Feature #18630]
STDIN.timeout = 1 STDIN.read # => Blocking operation timed out! (IO::TimeoutError)
-
[x] @moste00 [easy, pure ruby] Introduce
IO.new(..., path:)
and promoteFile#path
toIO#path
. [Feature #19036]
Class
-
[x] [easy, java] Class#attached_object, which returns the object for which the receiver is the singleton class. Raises TypeError if the receiver is not a singleton class. [Feature #12084]
class Foo; end Foo.singleton_class.attached_object #=> Foo Foo.new.singleton_class.attached_object #=> #<Foo:0x000000010491a370> Foo.attached_object #=> TypeError: `Foo' is not a singleton class nil.singleton_class.attached_object #=> TypeError: `NilClass' is not a singleton class
Data
-
[x] @moste00 [medium, pure ruby] New core class to represent simple immutable value object. The class is similar to Struct and partially shares an implementation, but has more lean and strict API. [Feature #16122]
Measure = Data.define(:amount, :unit) distance = Measure.new(100, 'km') #=> #<data Measure amount=100, unit="km"> weight = Measure.new(amount: 50, unit: 'kg') #=> #<data Measure amount=50, unit="kg"> weight.with(amount: 40) #=> #<data Measure amount=40, unit="kg"> weight.amount #=> 50 weight.amount = 40 #=> NoMethodError: undefined method `amount='
Encoding
- [x] [easy, pure ruby] Encoding#replicate has been deprecated and will be removed in 3.3. [Feature #18949]
- [ ] [easy, java, might already be the case] The dummy
Encoding::UTF_16
andEncoding::UTF_32
encodings no longer try to dynamically guess the endian based on a byte order mark. UseEncoding::UTF_16BE
/UTF_16LE
andEncoding::UTF_32BE
/UTF_32LE
instead. This change speeds up getting the encoding of a String. [Feature #18949] - [x] [medium, java] Limit maximum encoding set size by 256.
If exceeding maximum size,
EncodingError
will be raised. [Feature #18949]
Enumerator
- [x] [easy, pure ruby] Enumerator.product has been added. Enumerator::Product is the implementation. [Feature #18685]
Exception
- [x] [easy, some ruby, some java] Exception#detailed_message has been added. The default error printer calls this method on the Exception object instead of #message. [Feature #18564]
Hash
- [x] Hash#shift now always returns nil if the hash is empty, instead of returning the default value or calling the default proc. [Bug #16908]
Integer
- [x] @nirvdrum [easy, java] Integer#ceildiv has been added. [Feature #18809]
Kernel
- [ ] [easy, java] Kernel#binding raises RuntimeError if called from a non-Ruby frame (such as a method defined in C). [Bug #18487]
MatchData
- [x] @rwstauner [easy, java] MatchData#byteoffset has been added. [Feature #13110]
- [x] @rwstauner [easy, pure ruby] MatchData#deconstruct has been added. [Feature #18821]
- [x] @rwstauner [easy, pure ruby] MatchData#deconstruct_keys has been added. [Feature #18821]
Module
- [x] [already done, just need to untag specs/tests] Module.used_refinements has been added. [Feature #14332]
- [x] [easy, java] Module#refinements has been added. [Feature #12737]
- [x] [medium, java] Module#const_added has been added. [Feature #17881]
- [x] [easy, java] Module#undefined_instance_methods has been added. [Feature #12655]
Proc
- [x] [easy, seems already be the case, need to untag specs] Proc#dup returns an instance of subclass. [Bug #17545]
- [x] [easy, java] Proc#parameters now accepts lambda keyword. [Feature #15357]
Process
- [x] Added
RLIMIT_NPTS
constant to FreeBSD platform
Regexp
-
[x] The cache-based optimization is introduced. Many (but not all) Regexp matching is now in linear time, which will prevent regular expression denial of service (ReDoS) vulnerability. [Feature #19104]
-
[ ] [medium, java] Regexp.linear_time? is introduced. [Feature #19194]
-
[x] @rwstauner [easy, pure ruby] Regexp.new now supports passing the regexp flags not only as an Integer, but also as a String. Unknown flags raise ArgumentError. Otherwise, anything other than
true
,false
,nil
or Integer will be warned. [Feature #18788] -
[ ] [hard, java, would need changes in tregex backtracker] Regexp.timeout= has been added. Also, Regexp.new new supports timeout keyword. See [Feature #17837]
Refinement
- [x] [easy, java] Refinement#refined_class has been added. [Feature #12737]
Set
- [x] [easy, pure Ruby] Set is now available as a built-in class without the need for
require "set"
. [Feature #16989] It is currently autoloaded via the Set constant or a call to Enumerable#to_set.
String
- [x] [easy, java] String#byteindex and String#byterindex have been added. [Feature #13110]
- [ ] [medium, depends on jcodings] Update Unicode to Version 15.0.0 and Emoji Version 15.0. [Feature #18639] (also applies to Regexp)
- [x] [medium, java] String#bytesplice has been added. [Feature #18598]
- [x] [easy, pure ruby] String#dedup has been added as an alias to String#-@. [Feature #18595]
Struct
-
[x] @rwstauner [easy, pure ruby] A Struct class can also be initialized with keyword arguments without
keyword_init: true
on Struct.new [Feature #16806]Post = Struct.new(:id, :name) Post.new(1, "hello") #=> #<struct Post id=1, name="hello"> # From Ruby 3.2, the following code also works without keyword_init: true. Post.new(id: 1, name: "hello") #=> #<struct Post id=1, name="hello">
Thread
- [x] [medium, java] Thread.each_caller_location is added. [Feature #16663]
Thread::Queue
- [x] [medium, java] Thread::Queue#pop(timeout: sec) is added. [Feature #18774]
Thread::SizedQueue
- [x] [medium, java] Thread::SizedQueue#pop(timeout: sec) is added. [Feature #18774]
- [x] [medium, java] Thread::SizedQueue#push(timeout: sec) is added. [Feature #18944]
Time
-
[x] @rwstauner [easy, pure ruby] Time#deconstruct_keys is added, allowing to use Time instances in pattern-matching expressions [Feature #19071]
-
[ ] [medium] Time.new now can parse a string like generated by Time#inspect and return a Time instance based on the given argument. [Feature #18033]
SyntaxError
- [ ] [easy, java] SyntaxError#path has been added. [Feature #19138]
TracePoint
- [x] TracePoint#binding now returns
nil
forc_call
/c_return
TracePoints. [Bug #18487] - [ ] [medium, java, target_thread not yet implemented] TracePoint#enable
target_thread
keyword argument now defaults to the current thread if a block is given andtarget
andtarget_line
keyword arguments are not passed. [Bug #16889]
UnboundMethod
-
[x] @rwstauner [easy, java]
UnboundMethod#==
returnstrue
if the actual method is same. For example,String.instance_method(:object_id) == Array.instance_method(:object_id)
returnstrue
. [Feature #18798] -
[x] @rwstauner [easy, ruby+java]
UnboundMethod#inspect
does not show the receiver ofinstance_method
. For exampleString.instance_method(:object_id).inspect
returns"#<UnboundMethod: Kernel#object_id()>"
(was"#<UnboundMethod: String(Kernel)#object_id()>"
).
Stdlib updates
CGI
- [x] [import]
CGI.escapeURIComponent
andCGI.unescapeURIComponent
are added. [Feature #18822]
Coverage
-
[ ] [hard, java]
Coverage.setup
now acceptseval: true
. By this,eval
and related methods are able to generate code coverage. [Feature #19008] -
[x] [easy, pure ruby]
Coverage.supported?(mode)
enables detection of what coverage modes are supported. [Feature #19026]
Date
- [x] [easy, pure ruby] Added
Date#deconstruct_keys
andDateTime#deconstruct_keys
same as [Feature #19071]
FileUtils
- [x] [import] Add FileUtils.ln_sr method and
relative:
option to FileUtils.ln_s. [Feature #18925]
IRB
- [x] [import] debug.gem integration commands have been added:
debug
,break
,catch
,next
,delete
,step
,continue
,finish
,backtrace
,info
* They work even if you don't havegem "debug"
in your Gemfile. * See also: What's new in Ruby 3.2's IRB? - [x] [import] More Pry-like commands and features have been added.
*
edit
andshow_cmds
(like Pry'shelp
) are added. *ls
takes-g
or-G
option to filter out outputs. *show_source
is aliased from$
and accepts unquoted inputs. *whereami
is aliased from@
.
Net::Protocol
- [x] [import] Improve
Net::BufferedIO
performance. [GH-net-protocol-14]
Pathname
- [x] Added
Pathname#lutime
. [GH-pathname-20]
Socket
- [ ] Added the following constants for supported platforms.
*
SO_INCOMING_CPU
*SO_INCOMING_NAPI_ID
*SO_RTABLE
*SO_SETFIB
*SO_USER_COOKIE
*TCP_KEEPALIVE
*TCP_CONNECTION_INFO
SyntaxSuggest
- [ ] [import] The feature of
syntax_suggest
formerlydead_end
is integrated in Ruby. [Feature #18159]
UNIXSocket
-
[x] Add support for UNIXSocket on Windows. Emulate anonymous sockets. Add support for File.socket? and File::Stat#socket? where possible. [Feature #19135]
-
[x] [import] The following default gems are updated.
- RubyGems 3.4.1
- abbrev 0.1.1
- benchmark 0.2.1
- bigdecimal 3.1.3
- bundler 2.4.1
- cgi 0.3.6
- csv 3.2.6
- date 3.3.3
- delegate 0.3.0
- did_you_mean 1.6.3
- digest 3.1.1
- drb 2.1.1
- english 0.7.2
- erb 4.0.2
- error_highlight 0.5.1
- etc 1.4.2
- fcntl 1.0.2
- fiddle 1.1.1
- fileutils 1.7.0
- forwardable 1.3.3
- getoptlong 0.2.0
- io-console 0.6.0
- io-nonblock 0.2.0
- io-wait 0.3.0
- ipaddr 1.2.5
- irb 1.6.2
- json 2.6.3
- logger 1.5.3
- mutex_m 0.1.2
- net-http 0.3.2
- net-protocol 0.2.1
- nkf 0.1.2
- open-uri 0.3.0
- open3 0.1.2
- openssl 3.1.0
- optparse 0.3.1
- ostruct 0.5.5
- pathname 0.2.1
- pp 0.4.0
- pstore 0.1.2
- psych 5.0.1
- racc 1.6.2
- rdoc 6.5.0
- readline-ext 0.1.5
- reline 0.3.2
- resolv 0.2.2
- resolv-replace 0.1.1
- securerandom 0.2.2
- set 1.0.3
- stringio 3.0.4
- strscan 3.0.5
- syntax_suggest 1.0.2
- syslog 0.1.1
- tempfile 0.1.3
- time 0.2.1
- timeout 0.3.1
- tmpdir 0.1.3
- tsort 0.1.1
- un 0.2.1
- uri 0.12.0
- weakref 0.1.2
- win32ole 1.8.9
- yaml 0.2.1
- zlib 3.0.0
-
[x] [import] The following bundled gems are updated.
- minitest 5.16.3
- power_assert 2.0.3
- test-unit 3.5.7
- net-ftp 0.2.0
- net-imap 0.3.4
- net-pop 0.1.2
- net-smtp 0.3.3
- rbs 2.8.2
- typeprof 0.21.3
- debug 1.7.1
See GitHub releases like GitHub Releases of Logger or changelog for details of the default gems or bundled gems.
Compatibility issues
-
[x] [easy, pure ruby]
String#to_c
currently treat a sequence of underscores as an end of Complex string. [Bug #19087] -
[x] [easy, pure ruby] Now
ENV.clone
raisesTypeError
as well asENV.dup
[Bug #17767]
Removed constants
The following deprecated constants are removed.
- [x] [easy]
Fixnum
andBignum
[Feature #12005] - [x] [easy, pure ruby]
Random::DEFAULT
[Feature #17351] - [x] [easy, pure ruby]
Struct::Group
- [x] [easy, pure ruby]
Struct::Passwd
Removed methods
The following deprecated methods are removed.
- [x] [easy, pure ruby]
Dir.exists?
[Feature #17391] - [x] [easy, pure ruby]
File.exists?
[Feature #17391] - [x] [easy]
Kernel#=~
[Feature #15231] - [x] [easy]
Kernel#taint
,Kernel#untaint
,Kernel#tainted?
[Feature #16131] - [x] [easy]
Kernel#trust
,Kernel#untrust
,Kernel#untrusted?
[Feature #16131] - [ ] [easy, unclear if we should remove them, I think not]
Method#public?
,Method#private?
,Method#protected?
,UnboundMethod#public?
,UnboundMethod#private?
,UnboundMethod#protected?
[Bug #18729] [Bug #18751] [Bug #18435]
Error printer
- [x] Ruby no longer escapes control characters and backslashes in an error message. [Feature #18367]
Constant lookup when defining a class/module
- [ ] [medium, java] When defining a class/module directly under the Object class by class/module
statement, if there is already a class/module defined by
Module#include
with the same name, the statement was handled as "open class" in Ruby 3.1 or before. Since Ruby 3.2, a new class is defined instead. [Feature #18832]
Stdlib compatibility issues
-
[x] Psych no longer bundles libyaml sources. And also Fiddle no longer bundles libffi sources. Users need to install the libyaml/libffi library themselves via the package manager like apt, yum, brew, etc.
Psych and fiddle supported the static build with specific version of libyaml and libffi sources. You can build psych with libyaml-0.2.5 like this.
$ ./configure --with-libyaml-source-dir=/path/to/libyaml-0.2.5
And you can build fiddle with libffi-3.4.4 like this.
$ ./configure --with-libffi-source-dir=/path/to/libffi-3.4.4
-
[x] [import] Check cookie name/path/domain characters in
CGI::Cookie
. [CVE-2021-33621] -
[x] [import]
URI.parse
return empty string in host instead of nil. [sec-156615]
C API updates
Added C APIs
- [x] [easy, C, can just ignore the argument]
VALUE rb_hash_new_capa(long capa)
was added to created hashes with the desired capacity. - [x] [won't do, CRuby/GIL specific]
rb_internal_thread_add_event_hook
andrb_internal_thread_add_event_hook
were added to instrument threads scheduling. The following events are available:-
RUBY_INTERNAL_THREAD_EVENT_STARTED
-
RUBY_INTERNAL_THREAD_EVENT_READY
-
RUBY_INTERNAL_THREAD_EVENT_RESUMED
-
RUBY_INTERNAL_THREAD_EVENT_SUSPENDED
-
RUBY_INTERNAL_THREAD_EVENT_EXITED
-
- [ ] [medium, C]
rb_debug_inspector_current_depth
andrb_debug_inspector_frame_depth
are added for debuggers.
Removed C APIs
The following deprecated APIs are removed.
- [x] [easy]
rb_cData
variable. - [x] [import] "taintedness" and "trustedness" functions. [Feature #16131]
Implementation improvements
- [ ] [probably already done, need to check specs/tests for it] Fixed several race conditions in Kernel#autoload. [Bug #18782]
The 3.2 import has been merged in https://github.com/oracle/truffleruby/commit/66ca27b87a196707d3822bac2aaca201bb1600d0, so now RUBY_VERSION is "3.2.2" and it's easier implement the rest of 3.2 features. Contributions welcome!
is there anything left to do? I have joined Oracle recently as a fresher, so anything under easy tag would be convenient for me in java or in C. @eregon
Hey, I'm currently working on Feature #19036 !
Just a heads up because I can't edit the issue directly.
@moste00 noted. @AnuravModak I'd say just pick an easy one and mention it here.
So, i am choosing [Feature #12084], is it open for resolving, like i am little confused how to check whether it is opened or not? like i followed the link and gone through the discussion that was not very conclusive Basically, I didn't find any merge or closed tag for this feature in this page so I chose this one, can somebody help here??
I edited the issue description to mark you for that item. Feel free to ask questions on Slack (either public GraalVM Slack or the Oracle one).
I'm taking Feature #16122, the one about the new Data class.
hii , can i work on this Feature Feature #16989
@sarvo123 Sure, feel free to open a PR for it.
I'm taking Feature #16122, the one about the new Data class.
Any luck with this? Seems folks are starting to actually use it in production code.
@sempervictus Hey !
I have some basic skeleton in place, basically you can create classes and instances and ask instances for members but not much else. I got held up the last couple of weeks but I plan to finish it and make a PR in the next 2 weeks, fingers crossed.
Thanks @moste00 - OpenProject decided to go all in on the 3.2 syntax and i'm having to roll back their Data.define
stuff (mildly redundant IMO, but i've been around since green threads were cool and people used continuations, the language has seen worse). If you have a branch somewhere i'm happy to test it - we maintain our own packaging for this stuff on a fork of Arch
There's a core syntax change missing from the description from what i can tell. Seems MRI now allows assignment in the def
line:
def method_name(arg) = call_something_else(arg)
which has to get rewritten to
def method_name(arg)
call_something_else(arg)
end
to prevent crashes on TruffleRuby
Case statements using pattern matches are also not handled correctly by TruffleRuby in 3.2 mode:
case /Storages::(?'provider_name'.*)Storage/.match(provider_type)
in provider_name:
provider_name.downcase
else
raise ArgumentError,
produces syntax error, unexpected keyword_in:
@moste00 Could you open a draft PR with what you have? Data should be pretty easy to add. Most of the effort might be adding specs but there are already some.
@sempervictus
There's a core syntax change missing from the description from what i can tell. Seems MRI now allows assignment in the def line:
That will be fixed by the new parser, you can follow https://github.com/oracle/truffleruby/issues/3117 for that. There is good progress there, it should be done by the next release.
Case statements using pattern matches are also not handled correctly by TruffleRuby in 3.2 mode
TruffleRuby supports the array pattern matching by using the --pattern-matching
flag, but not the hash pattern matching yet. Hence the flag to consider it all not implemented to avoid surprises.
We'll need https://github.com/oracle/truffleruby/issues/3117 and then implementing hash pattern matching to address that.
@eregon @sempervictus Sorry for being late, PR here https://github.com/oracle/truffleruby/pull/3341.
Data
is supported now on truffleruby-head/dev.
I am working on A proc that accepts a single positional argument and keywords will no longer autosplat. [Bug #18633]
I am working on Anonymous rest and keyword rest arguments can now be passed as arguments, instead of just used in method parameters. [Feature #18351]
I am working on Constant assignment evaluation order for constants set on explicit objects has been made consistent with single attribute assignment evaluation order
I am working on Class#attached_object
Working on removing deprecated methods.
Working on removing deprecated constants.
I'll look at 19071, Time#deconstruct_keys
.
It seems like we have {Date,DateTime}#deconstruct_keys
from date_core.c
looking at 18788
Regexp.new now supports passing the regexp flags as a string.
Working on Set is now available as a built-in class without the need for require "set". [Feature #16989]
Working on Struct class can also be initialized with keyword arguments without keyword_init: true on Struct.new [Feature #16806]
We are looking at UnboundMethod#{==,inspect}
as part of HackDays
👋 Manef from Shopify,
Wanted to confirm if the following : [easy, pure ruby] Encoding#replicate has been deprecated and will be removed in 3.3. [[Feature #18949](https://bugs.ruby-lang.org/issues/18949)]
On the CRuby side it seems to be targeted for 3.3 : https://bugs.ruby-lang.org/issues/18949
Is still targeting 3.2 ? Ty !
@manefz It's deprecated in 3.2 and removed in 3.3. So this item is to deprecate it, that is emit the same warning as CRuby 3.2 does when using that method.