Ruby 3.0 support

[eregon]

We plan to work on Ruby 3.0 support for the next release, TruffleRuby/GraalVM 22.0.

Any help is appreciated whether it is:

• implementing one of the Ruby 3 changes in TruffleRuby, and ensuring it is tested in specs or MRI tests and otherwise add specs
• adding specs for Ruby 3 changes, which is best done in ruby/spec directly, see Write specs for new Ruby 3.0 features and changes ruby/spec#823

See this comment: #2453 (comment) for finding good getting-started issues.

See https://github.com/oracle/truffleruby/blob/3e03bed224076c2c301a2490a052568a2d013cfe/doc/contributor/workflow.md#running-specs-for-ruby-30-features for how to run specs for 3.0.

Notes

• Fiber scheduler changes are not implemented because it seems not worth it until Truffle supports VirtualThread on both Native Image and HotSpot.

Full list of changes for Ruby 3.0.0

NOTE: https://rubyreferences.github.io/rubychanges/3.0.html gives more details for many features and changes.
From https://github.com/ruby/ruby/blob/master/doc/NEWS-3.0.0.md and ruby/spec#823:

This document is a list of user visible feature changes
since the 2.7.0 release, except for bug fixes.

Note that each entry is kept to a minimum, see links for details.

• @eregon Initial import of CRuby 3.0 sources

Language changes

See jruby/jruby#6880 regarding parser changes.

• @eregon @chrisseaton @wildmaples Keyword arguments are now separated from positional arguments.
  Code that resulted in deprecation warnings in Ruby 2.7 will now
  result in ArgumentError or different behavior. [Feature #14183]

• @eregon Procs accepting a single rest argument and keywords are no longer
  subject to autosplatting. This now matches the behavior of Procs
  accepting a single rest argument and no keywords.
  [Feature #16166]

  pr = proc{|*a, **kw| [a, kw]}
  
  pr.call([1])
  # 2.7 => [[1], {}]
  # 3.0 => [[[1]], {}]
  
  pr.call([1, {a: 1}])
  # 2.7 => [[1], {:a=>1}] # and deprecation warning
  # 3.0 => [[[1, {:a=>1}]], {}]

• [parser, hard] Arguments forwarding (...) now supports leading arguments.
  [Feature #16378]

  def method_missing(meth, ...)
    send(:"do_#{meth}", ...)
  end

• @razetime [parser, hard] Pattern matching (case/in) is no longer experimental. [Feature #17260]

• @razetime [parser, hard] One-line pattern matching is redesigned. [EXPERIMENTAL]

  • => is added. It can be used like a rightward assignment.
    [Feature #17260]

    0 => a
    p a #=> 0
    
    {b: 0, c: 1} => {b:}
    p b #=> 0

  • in is changed to return true or false. [Feature #17371]

    # version 3.0
    0 in 1 #=> false
    
    # version 2.7
    0 in 1 #=> raise NoMatchingPatternError

• @razetime [parser, hard] Find-pattern is added. [EXPERIMENTAL]
  [Feature #16828]

  case ["a", 1, "b", "c", 2, "d", "e", "f", 3]
  in [*pre, String => x, String => y, *post]
    p pre  #=> ["a", 1]
    p x    #=> "b"
    p y    #=> "c"
    p post #=> [2, "d", "e", "f", 3]
  end

• [parser, hard] Endless method definition is added. [EXPERIMENTAL]
  [Feature #16746]

  def square(x) = x * x

• [easy, undo some of Make interpolated strings frozen #2304] @bjfish Interpolated String literals are no longer frozen when
  # frozen-string-literal: true is used. [Feature #17104]

• [translator, medium] Magic comment shareable_constant_value added to freeze constants.
  See {Magic Comments}[rdoc-ref:doc/syntax/comments.rdoc@Magic+Comments] for more details.
  [Feature #17273]

• [already the case] Deprecation warnings are no longer shown by default (since Ruby 2.7.2).
  Turn them on with -W:deprecated (or with -w to show other warnings too).
  [Feature #16345]

• @bjfish [cleanup, easy] $SAFE and $KCODE are now normal global variables with no special behavior.
  C-API methods related to $SAFE have been removed.
  [Feature #16131] [Feature #17136]

• @Strech [medium] yield in singleton class definitions in methods is now a SyntaxError
  instead of a warning. yield in a class definition outside of a method
  is now a SyntaxError instead of a LocalJumpError. [Feature #15575]

• @Strech [medium] When a class variable is overtaken by the same definition in an
  ancestor class/module, a RuntimeError is now raised (previously,
  it only issued a warning in verbose mode).
  [Bug #14541]

• [medium] @bjfish Accessing a class variable from the toplevel scope is now a RuntimeError.
  [Bug #14541]

• [easy] Assigning to a numbered parameter is now a SyntaxError instead of
  a warning.

Command line options

--help option

• [launcher, medium] @Strech When the environment variable RUBY_PAGER or PAGER is present and has
  a non-empty value, and the standard input and output are tty, the --help
  option shows the help message via the pager designated by the value.
  [Feature #16754]

--backtrace-limit option

• @bjfish [medium] The --backtrace-limit option limits the maximum length of a backtrace.
  [Feature #8661]

Core classes updates

Outstanding ones only.

Array

• [easy, ruby-only] @Strech [Ruby 3.0 support] Add support for Array subclass methods to return Array #2510 The following methods now return Array instances instead of
  subclass instances when called on subclass instances:
  [Bug #6087]

  • Array#drop
  • Array#drop_while
  • Array#flatten
  • Array#slice!
  • Array#slice / Array#[]
  • Array#take
  • Array#take_while
  • Array#uniq
  • Array#*

• [medium, ruby-only with one call from java] @ccocchi [Ruby 3.0 support] Array#slice with arithmetic sequence #2526 Can be sliced with Enumerator::ArithmeticSequence

  dirty_data = ['--', 'data1', '--', 'data2', '--', 'data3']
  dirty_data[(1..).step(2)] # take each second element
  # => ["data1", "data2", "data3"]

Binding

• @aardvark179 [easy, might be already the case] Binding#eval when called with one argument will use "(eval)"
  for __FILE__ and 1 for __LINE__ in the evaluated code.
  [Bug #4352] [Bug #17419]

ConditionVariable

• • @aardvark179 [hard] ConditionVariable#wait may now invoke the block/unblock scheduler
    hooks in a non-blocking context. [Feature #16786]

Dir

• [probably already the case, need to check] @Strech Dir.glob and Dir.[] now sort the results by default, and
  accept the sort: keyword option. [Feature #8709]

ENV

• [easy, pure ruby] ENV.except has been added, which returns a hash excluding the
  given keys and their values. [Feature #15822]

• Windows: Read ENV names and values as UTF-8 encoded Strings
  [Feature #12650]

Encoding

• [hard, jcodings] Added new encoding IBM720. [Feature #16233]

• Changed default for Encoding.default_external to UTF-8 on Windows
  [Feature #16604]

Fiber

• [easy] @aardvark179 Fiber.new(blocking: true/false) allows you to create non-blocking
  execution contexts. [Feature #16786]

• [easy] @aardvark179 Fiber#blocking? tells whether the fiber is non-blocking. [Feature #16786]

• [medium] @eregon Fiber#backtrace and Fiber#backtrace_locations provide per-fiber backtrace.
  [Feature #16815]

• @bjfish The limitation of Fiber#transfer is relaxed. [Bug #17221]

GC

• @bjfish [easy, pure ruby, just keep the state no effect] GC.auto_compact= and GC.auto_compact have been added to control
  when compaction runs. Setting auto_compact= to true will cause
  compaction to occur during major collections. At the moment,
  compaction adds significant overhead to major collections, so please
  test first! [Feature #17176]

Hash

• [easy, pure ruby] @ccocchi Allow Hash#transform_keys to take a hash argument #2464 Hash#transform_keys and Hash#transform_keys! now accept a hash that maps
  keys to new keys. [Feature #16274]

• [easy, pure ruby] @wildmaples Implement Hash#except #2463 Hash#except has been added, which returns a hash excluding the
  given keys and their values. [Feature #15822]

IO

• [hard, need to check side effects] @aardvark179 IO#nonblock? now defaults to true. [Feature #16786]

• • [hard] @aardvark179 IO#wait_readable, IO#wait_writable, IO#read, IO#write and other
    related methods (e.g. IO#puts, IO#gets) may invoke the scheduler hook
    #io_wait(io, events, timeout) in a non-blocking execution context.
    [Feature #16786]

Kernel

• [medium, java] @andrykonchin Kernel#clone when called with the freeze: false keyword will call
  #initialize_clone with the freeze: false keyword.
  [Bug #14266]

• [medium, java] @andrykonchin Kernel#clone when called with the freeze: true keyword will call
  #initialize_clone with the freeze: true keyword, and will
  return a frozen copy even if the receiver is unfrozen.
  [Feature #16175]

• @aardvark179 [easy, might be already the case] Kernel#eval when called with two arguments will use "(eval)"
  for __FILE__ and 1 for __LINE__ in the evaluated code.
  [Bug #4352]

• [easy] Kernel#lambda now warns if called without a literal block.
  [Feature #15973]

• • [hard] @aardvark179 Kernel.sleep invokes the scheduler hook #kernel_sleep(...) in a
    non-blocking execution context. [Feature #16786]

Module

• [hard] @bjfish Module#include and Module#prepend now affect classes and modules
  that have already included or prepended the receiver, mirroring the
  behavior if the arguments were included in the receiver before
  the other modules and classes included or prepended the receiver.
  [Feature #9573]

  class C; end
  module M1; end
  module M2; end
  C.include M1
  M1.include M2
  p C.ancestors #=> [C, M1, M2, Object, Kernel, BasicObject]

• @bjfish [medium, java] Module#public, Module#protected, Module#private, Module#public_class_method,
  Module#private_class_method, toplevel "private" and "public" methods
  now accept single array argument with a list of method names. [Feature #17314]

• [medium, java] @gogainda Module#attr_* methods now return an array of method names #2498 Module#attr_accessor, Module#attr_reader, Module#attr_writer and Module#attr
  methods now return an array of defined method names as symbols.
  [Feature #17314]

• [easy] @gogainda Module#alias_method now returns the defined alias as a symbol.
  [Feature #17314]

Mutex

• [already the case] Mutex is now acquired per-Fiber instead of per-Thread. This change
  should be compatible for essentially all usages and avoids blocking when
  using a scheduler. [Feature #16792]

Proc

• [medium] @bjfish Proc#== and Proc#eql? are now defined and will return true for
  separate Proc instances if the procs were created from the same block.
  [Feature #14267]

Queue / SizedQueue

• • [hard] @aardvark179 Queue#pop, SizedQueue#push and related methods may now invoke the
    block/unblock scheduler hooks in a non-blocking context.
    [Feature #16786]

Ractor

• [will add later since experimental & unstable in CRuby 3.0] New class added to enable parallel execution. See rdoc-ref:ractor.md for
  more details.

Random

• [easy, pure ruby] @eregon Random::DEFAULT now refers to the Random class instead of being a Random instance,
  so it can work with Ractor.
  [Feature #17322]

• [easy, pure ruby] @eregon Random::DEFAULT is deprecated since its value is now confusing and it is no longer global,
  use Kernel.rand/Random.rand directly, or create a Random instance with Random.new instead.
  [Feature #17351]

String

• @bjfish [easy] The following methods now return or yield String instances
  instead of subclass instances when called on subclass instances:
  [Bug #10845]

  • String#*
  • String#capitalize
  • String#center
  • String#chomp
  • String#chop
  • String#delete
  • String#delete_prefix
  • String#delete_suffix
  • String#downcase
  • String#dump
  • String#each_char
  • String#each_line
  • String#gsub
  • String#ljust
  • String#lstrip
  • String#partition
  • String#reverse
  • String#rjust
  • String#rpartition
  • String#rstrip
  • String#scrub
  • String#slice!
  • String#slice / String#[]
  • String#split
  • String#squeeze
  • String#strip
  • String#sub
  • String#succ / String#next
  • String#swapcase
  • String#tr
  • String#tr_s
  • String#upcase

• @bjfish [easy] The following methods now return or yield String instances
  instead of subclass instances when called on subclass instances:
  [Bug #10845]

  • String#each_grapheme_cluster
  • String#scan

Symbol

• [might already be the case, easy] Symbol#to_proc now returns a lambda Proc. [Feature #16260]

• @bjfish [easy, java] Symbol#name has been added, which returns the name of the symbol
  if it is named. The returned string is frozen. [Feature #16150]

Fiber

• • @aardvark179 [hard] Introduce Fiber.set_scheduler for intercepting blocking operations and
    Fiber.scheduler for accessing the current scheduler. See
    rdoc-ref:fiber.md for more details about what operations are supported and
    how to implement the scheduler hooks. [Feature #16786]

• @aardvark179 [hard] Fiber.blocking? tells whether the current execution context is
  blocking. [Feature #16786]

• • @aardvark179 [hard] Thread#join invokes the scheduler hooks block/unblock in a
    non-blocking execution context. [Feature #16786]

Thread

• @bjfish [easy, just noop there is no deadlock detection] Thread.ignore_deadlock accessor has been added for disabling the
  default deadlock detection, allowing the use of signal handlers to
  break deadlock. [Bug #13768]

Warning

• [should already be the case, easy] @Strech [Ruby 3.0 support] Add category kwarg to Kernel.warn and Warning.warn #2533 Warning#warn now supports a category keyword argument.
  [Feature #17122]

Stdlib updates

Set

• SortedSet has been removed for dependency and performance reasons.

• Set#join is added as a shorthand for .to_a.join.

• Set#<=> is added.

Socket

• [hard] Add :connect_timeout to TCPSocket.new [Feature #17187]

Compatibility issues

Excluding feature bug fixes.

• [done for Regexp, todo for Range, medium] @tomstuart @MattAlp Regexp literals and all Range objects are frozen. [Feature #8948] [Feature #16377] [Feature #15504]

  /foo/.frozen? #=> true
  (42...).frozen? # => true

• Embed Range literals in the AST Embed Range literals in the AST #2622

• [medium] @bjfish EXPERIMENTAL: Hash#each consistently yields a 2-element array. [Bug #12706]

  • Now { a: 1 }.each(&->(k, v) { }) raises an ArgumentError
    due to lambda's arity check.

• [medium] When writing to STDOUT redirected to a closed pipe, no broken pipe #2532 When writing to STDOUT redirected to a closed pipe, no broken pipe
  error message will be shown now. [Feature #14413]

• [easy, pure ruby] TRUE/FALSE/NIL constants are no longer defined.

• @bjfish [easy, pure ruby] Integer#zero? overrides Numeric#zero? for optimization. [Misc #16961]

• @bjfish [medium] Enumerable#grep and Enumerable#grep_v when passed a Regexp and no block no longer modify
  Regexp.last_match. [Bug #17030]

• Requiring 'open-uri' no longer redefines Kernel#open.
  Call URI.open directly or use URI#open instead. [Misc #15893]

• SortedSet has been removed for dependency and performance reasons.

Stdlib compatibility issues

• @eregon Default gems

  • The following libraries are promoted to default gems from stdlib.

    • English
    • abbrev
    • base64
    • drb
    • debug
    • erb
    • find
    • net-ftp
    • net-http
    • net-imap
    • net-protocol
    • open-uri
    • optparse
    • pp
    • prettyprint
    • resolv-replace
    • resolv
    • rinda
    • set
    • securerandom
    • shellwords
    • tempfile
    • tmpdir
    • time
    • tsort
    • un
    • weakref

  • The following extensions are promoted to default gems from stdlib.

    • digest
    • io-nonblock
    • io-wait
    • nkf
    • pathname
    • syslog
    • win32ole

• @eregon Bundled gems

  • net-telnet and xmlrpc have been removed from the bundled gems.
    If you are interested in maintaining them, please comment on
    your plan to https://github.com/ruby/xmlrpc
    or https://github.com/ruby/net-telnet.

• @eregon SDBM has been removed from the Ruby standard library. [Bug #8446]

  • The issues of sdbm will be handled at https://github.com/ruby/sdbm

• @eregon WEBrick has been removed from the Ruby standard library. [Feature #17303]

  • The issues of WEBrick will be handled at https://github.com/ruby/webrick

C API updates

• @eregon C API functions related to $SAFE have been removed.
  [Feature #16131]

• @eregon C API header file ruby/ruby.h was split. [GH-2991]

  This should have no impact on extension libraries,
  but users might experience slow compilations.

• @eregon Memory view interface [EXPERIMENTAL]

  • The memory view interface is a C-API set to exchange a raw memory area,
    such as a numeric array or a bitmap image, between extension libraries.
    The extension libraries can share also the metadata of the memory area
    that consists of the shape, the element format, and so on.
    Using these kinds of metadata, the extension libraries can share even
    a multidimensional array appropriately.
    This feature is designed by referring to Python's buffer protocol.
    [Feature #13767] [Feature #14722]

• [will add later since experimental & unstable in CRuby 3.0] Ractor related C APIs are introduced (experimental) in "include/ruby/ractor.h".

• @eregon rb_keyword_given_p() was added to find if kwargs were passed from Ruby to C.

Implementation improvements

• already the case New method cache mechanism for Ractor. [Feature #16614]

  • Inline method caches pointed from ISeq can be accessed by multiple Ractors
    in parallel and synchronization is needed even for method caches. However,
    such synchronization can be overhead so introducing new inline method cache
    mechanisms, (1) Disposable inline method cache (2) per-Class method cache
    and (3) new invalidation mechanism. (1) can avoid per-method call
    synchronization because it only uses atomic operations.
    See the ticket for more details.

• @eregon The number of hashes allocated when using a keyword splat in
  a method call has been reduced to a maximum of 1, and passing
  a keyword splat to a method that accepts specific keywords
  does not allocate a hash.

• [already the case, all super are optimized] super is optimized when the same type of method is called in the previous call
  if it's not refinements or an attr reader or writer.

Miscellaneous changes

• @eregon Methods using ruby2_keywords will no longer keep empty keyword
  splats, those are now removed just as they are for methods not
  using ruby2_keywords.

• [already the case] When an exception is caught in the default handler, the error
  message and backtrace are printed in order from the innermost.
  [Feature #8661]

• [easy, cleanup] Accessing an uninitialized instance variable no longer emits a
  warning in verbose mode. [Feature #17055]

[eregon]

I'll start by importing the CRuby 3.0.2 files and report here when done.

[chrisseaton]

@wildmaples and I can handle keyword argument as part of our current work. Do you want to note that down to avoid conflict?

[eregon]

OK, I added you in the description for keyword arguments-related changes.
That's probably the easiest way to keep track, but we could also use the wiki or a spreadsheet if that's more convenient.

[rubyFeedback]

headius also had that for jruby recently for ruby 3.x - are you folks doing a friendly competition race? :)

On a more on-topic note (but applicable to jruby too) - I don't know of a simple way, but like if it would be
possible to evaluate how much time and knowlege is required for different issues then perhaps new folks
could help. E. g. ruby users who know ruby fairly well but aren't the best at java (e. g. at the basic level;
to explain this a bit I wrote a lot of ruby, and I am writing java almost daily now, mostly porting important
ruby classes into java, which works surprisingly well too - but I am still munching through java tutorials so
I don't feel close to being able to contribute. Which is why having like a list of "todo" tasks with different
"difficulty" levels may be helpful. But I understand that doing so manually is too much work for the devs;
would be nice if there is a way to somehow have this as a table with an entry indicating the required
expert level. Perhaps this may also help newcomers.)

[eregon]

@rubyFeedback Some of the points above can be implemented with only Ruby code (e.g., if the mentioned method is already implemented in Ruby or if it's a new method which seems like it can be implemented with Ruby code). Those are generally all relatively easy.
The ones needing changes to Java code are typically more involved, and of course they require knowing some Java.

We could potentially do a list of the points which seem easy contributions.
One way to help here would be to start a table (e.g. google sheets or in a CSV) with one row per checkbox, and a single-line description of the task. Then we can add columns for difficulty, assignee, etc.

[eregon]

I annotated Ractor as [will add later since experimental & unstable in CRuby 3.0].
In CRuby 3.0 Ractor is experimental and has many bugs, so I don't think we need to support right away for general Ruby 3.0 support.
It's also a larger effort, so that will probably be done later on its own.
One possibility is using https://github.com/marcandre/backports/tree/master/lib/backports/ractor which is pure Ruby (but likely not so fast for deep cloning/freezing).

[eregon]

I annotated each change above with easy, medium and hard.
There is also pure ruby for changes which should only affect Ruby code and not require any change to Java sources or any Java knowledge.
If there is no pure ruby then some Java code is involved, but it should still be fairly straightforward if easy or medium.

hard typically requires more knowledge about TruffleRuby implementation details.

If you'd like to start on some task please mention it here or edit the description directly for committers.

[gogainda]

@eregon I would like to try some easy java task

[eregon]

@gogainda Then any task which is easy but not pure ruby should fall into that category.
You can just choose and mention it here.
Here is one I spotted from a quick look, feel free to pick that one or another:

[easy] Module#alias_method now returns the defined alias as a symbol. [Feature #17314]