eval-dev-quality Ruby support

We want to add language support for Ruby.

[x] Install Ruby in Docker containers
[x] Introduce Ruby as a new language
[x] Register the Ruby language #327
[x] Implement the ExecuteTests method in the Ruby language #327
[x] Implement write-tests task
- [x] Create "plain" repository for Ruby
- [x] Translate the light repository using a LLM
- [x] Support Ruby in symflower test
[x] Implement code_repair task
- [x] Translate the mistakes repository using a LLM
- [x] Implement the Mistakes method in the Ruby language
- [x] Add missing tests #327
[x] Implement transpile task
- The transpile repository used examples from the light repository, so no need to translate them again
- [x] Change the way we get the origin/destination languages for transpilation
  - [x] The transpile package names must be e.g. for Go transpile-java, transpile-ruby, instead of just transpile so it is clear which language we are transpiling from
- [x] Add missing tests #327
[x] Documentation
- [x] Update documentation with new Ruby support #327
- [x] New features #327
  - [x] Supported providers
  - [x] New Reward points
[x] Merge fixes #327

Follow-up

[ ] collect all mistakes and remove duplicates: https://github.com/symflower/eval-dev-quality/pull/326#discussion_r1732496767

Jul 31 '24 05:07 ahumenberger

For new transpilation tasks, there needs to be some small changes to how the eval figures out which languages are targeted. Cause right now if we have Go as transpilation source, it automatically assumes Java must be the target cause no other language existed yet.

Aug 01 '24 09:08 bauersimon

Checking for errors on the Ruby `mistakes` repository

For Java and Go we check for compilation errors with:

Go: go build
Java: mvn compile

Since Ruby is an interpreted language, there is no compilation step. I tried the following approaches.

Check Ruby's syntax with ruby -c lib/<ruby-file> (see Command line options)

argumentsMissing: Syntax OK
endKeywordMissing: lib/end_keyword_missing.rb:9: syntax error, unexpected end-of-input, expecting keyword_end
importMissing: SyntaxOK
typeUnknown: SyntaxOK
variableUnknown: SyntaxOK

Conclusion: only only error caught

Running rake test at the root of the repository

argumentsMissing

Output

Run options: --seed 64839

# Running:

EEE

Finished in 0.000889s, 3373.6712 runs/s, 0.0000 assertions/s.

  1) Error:
TestArgumentMissing#test_argument_missing2:
ArgumentError: wrong number of arguments (given 1, expected 0)
    /home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:1:in `argument_missing'
    /home/rui/dev/ruby/playground/mist/argumentMissing/test/argument_missing_test.rb:15:in `test_argument_missing2'

  2) Error:
TestArgumentMissing#test_argument_missing3:
ArgumentError: wrong number of arguments (given 1, expected 0)
    /home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:1:in `argument_missing'
    /home/rui/dev/ruby/playground/mist/argumentMissing/test/argument_missing_test.rb:22:in `test_argument_missing3'

  3) Error:
TestArgumentMissing#test_argument_missing1:
ArgumentError: wrong number of arguments (given 1, expected 0)
    /home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:1:in `argument_missing'
    /home/rui/dev/ruby/playground/mist/argumentMissing/test/argument_missing_test.rb:8:in `test_argument_missing1'

3 runs, 0 assertions, 0 failures, 3 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

endKeywordMissing

Output

/home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:4: warning: mismatched indentations at 'end' with 'if' at 2
/home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:7: warning: mismatched indentations at 'end' with 'if' at 5
Traceback (most recent call last):
        6: from /usr/lib64/ruby/gems/2.5.0/gems/rake-12.3.3/lib/rake/rake_test_loader.rb:5:in `<main>'
        5: from /usr/lib64/ruby/gems/2.5.0/gems/rake-12.3.3/lib/rake/rake_test_loader.rb:5:in `select'
        4: from /usr/lib64/ruby/gems/2.5.0/gems/rake-12.3.3/lib/rake/rake_test_loader.rb:17:in `block in <main>'
        3: from /usr/lib64/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        2: from /usr/lib64/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        1: from /home/rui/dev/ruby/playground/mist/endKeywordMissing/test/end_keyword_missing_test.rb:2:in `<top (required)>'
/home/rui/dev/ruby/playground/mist/endKeywordMissing/test/end_keyword_missing_test.rb:2:in `require_relative': /home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:9: syntax error, unexpected end-of-input, expecting keyword_end (SyntaxError)
        return 0
                ^
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

importMissing

Output

/home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:9: warning: mismatched indentations at 'end' with 'def' at 5
Run options: --seed 60902

# Running:

EEEEEE

Finished in 0.001216s, 4932.5231 runs/s, 0.0000 assertions/s.

  1) Error:
TestImportMissing#test_invalid_json:
NameError: uninitialized constant TestImportMissing::JSON
    /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:31:in `test_invalid_json'

  2) Error:
TestImportMissing#test_valid_json:
NameError: uninitialized constant JSON
    /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json'
    /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:8:in `test_valid_json'

  3) Error:
TestImportMissing#test_json_with_nested_objects:
NameError: uninitialized constant JSON
    /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json'
    /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:26:in `test_json_with_nested_objects'

  4) Error:
TestImportMissing#test_json_with_numbers:
NameError: uninitialized constant JSON
    /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json'
    /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:20:in `test_json_with_numbers'

  5) Error:
TestImportMissing#test_json_with_array:
NameError: uninitialized constant JSON
    /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json'
    /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:37:in `test_json_with_array'

  6) Error:
TestImportMissing#test_empty_json:
NameError: uninitialized constant JSON
    /home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json'
    /home/rui/dev/ruby/playground/mist/importMissing/test/import_missing_test.rb:14:in `test_empty_json'

6 runs, 0 assertions, 0 failures, 6 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

typeUnknown

Output

Run options: --seed 56356

# Running:

EEE

Finished in 0.001033s, 2904.8122 runs/s, 0.0000 assertions/s.

  1) Error:
TestTypeUnknown#test_type_is_wrong1:
NameError: uninitialized constant Intt
    /home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown'
    /home/rui/dev/ruby/playground/mist/typeUnknown/test/type_unknown_test.rb:8:in `test_type_is_wrong1'

  2) Error:
TestTypeUnknown#test_type_is_wrong3:
NameError: uninitialized constant Intt
    /home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown'
    /home/rui/dev/ruby/playground/mist/typeUnknown/test/type_unknown_test.rb:22:in `test_type_is_wrong3'

  3) Error:
TestTypeUnknown#test_type_is_wrong2:
NameError: uninitialized constant Intt
    /home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown'
    /home/rui/dev/ruby/playground/mist/typeUnknown/test/type_unknown_test.rb:15:in `test_type_is_wrong2'

3 runs, 0 assertions, 0 failures, 3 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

variableUnknown

Output

/home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2: warning: possibly useless use of == in void context
Run options: --seed 59116

# Running:

EEE

Finished in 0.000951s, 3153.5992 runs/s, 0.0000 assertions/s.

  1) Error:
TestVariableUnknown#test_variable_unknown2:
NameError: undefined local variable or method `y' for #<TestVariableUnknown:0x000055b56a9accf0>
    /home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown'
    /home/rui/dev/ruby/playground/mist/variableUnknown/test/variable_unknown_test.rb:15:in `test_variable_unknown2'

  2) Error:
TestVariableUnknown#test_variable_unknown1:
NameError: undefined local variable or method `y' for #<TestVariableUnknown:0x000055b56a9ac160>
    /home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown'
    /home/rui/dev/ruby/playground/mist/variableUnknown/test/variable_unknown_test.rb:8:in `test_variable_unknown1'

  3) Error:
TestVariableUnknown#test_variable_unknown3:
NameError: undefined local variable or method `y' for #<TestVariableUnknown:0x000055b56a6eddc0>
    /home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown'
    /home/rui/dev/ruby/playground/mist/variableUnknown/test/variable_unknown_test.rb:22:in `test_variable_unknown3'

3 runs, 0 assertions, 0 failures, 3 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

Conclusion: It catches all errors, but the structure of syntax errors are different from runtime errors.

Adding a call.rb that calls the function defined in lib/<ruby-source-file, then running ruby call.rb and check the output

Example of call.rb

require_relative 'lib/end_keyword_missing'

end_keyword_missing(1)

argumentsMissing

Output

Traceback (most recent call last):
        1: from call.rb:3:in `<main>'
/home/rui/dev/ruby/playground/mist/argumentMissing/lib/argument_missing.rb:2:in `argument_missing': undefined local variable or method `x' for main:Object (NameError)

endKeywordMissing

Output

Traceback (most recent call last):
        1: from call.rb:1:in `<main>'
call.rb:1:in `require_relative': /home/rui/dev/ruby/playground/mist/endKeywordMissing/lib/end_keyword_missing.rb:9: syntax error, unexpected end-of-input, expecting keyword_end (SyntaxError)
        return 0

importMissing

Output

Traceback (most recent call last):
        1: from call.rb:3:in `<main>'
/home/rui/dev/ruby/playground/mist/importMissing/lib/import_missing.rb:2:in `parse_json': uninitialized constant JSON (NameError)

typeUnknown

Output

Traceback (most recent call last):
        1: from call.rb:3:in `<main>'
/home/rui/dev/ruby/playground/mist/typeUnknown/lib/type_unknown.rb:2:in `type_unknown': uninitialized constant Intt (NameError)

variableUnknown

Output

Traceback (most recent call last):
        1: from call.rb:3:in `<main>'
/home/rui/dev/ruby/playground/mist/variableUnknown/lib/variable_unknown.rb:2:in `variable_unknown': undefined local variable or method `y' for main:Object (NameError)

Conclusion: better error messages. All errors have the same structure

Conclusion

Using ruby -c only checks for syntax errors. We can only catch one error within 5 packages
Using rake test
- We get the errors for all packages, but syntax errors have a different error structure
- Implies running all tests
- A little bit harder to parse
Using the call.rb
- All errors have the same format
- Implies adding a file to each package calling the function with the errors
- Easier to parse

Aug 07 '24 15:08 ruiAzevedo19

Claude suggested rubocop, it has JSON formatted output: https://docs.rubocop.org/rubocop/formatters.html#json-formatter

Aug 08 '24 06:08 bauersimon

All PRs are merged so closing this one.

Nov 07 '24 11:11 Munsio

Ruby support

Follow-up

Checking for errors on the Ruby mistakes repository

Conclusion

Checking for errors on the Ruby `mistakes` repository