Normalization adds whitespaces - inconsistently

[sethherr]

trafficstars

Running into an issue with inconsistent normalization on Bike Index.

Normalization is not adding whitespace locally, and then is failing on CircleCI. Which is very frustrating.

I can't demonstrate that normalize is passing locally, but I promise that it is. Both local and CircleCI are running the same version of ruby, same version of gems, etc.

The end result of this is the same as #319 - but since i18n-tasks no longer uses Psych, and instead uses the standard lib yaml parser, I think this belongs in a separate issue.

I'm trying to solve this, hopefully I figure it out shortly and I'll update here when I do - but in case anyone else runs into this problem, I figure documenting it is worthwhile.

[sethherr]

I was not able to figure out a quick way of diagnosing what the difference was between the CircleCI and local parsing, and just manually edited the whitespace back in locally (https://github.com/bikeindex/bike_index/pull/1646)

If there is a way to do additional diagnostics, or any way of getting to a fix on this, I would be very appreciative!

[AlexanderZagaynov]

I also bumped with this issue, here is how it looks on Travis:

[rachid]

Exactly the same issue here!

[goshatch]

Also bumped into this. Using Circle CI and Linux for local dev.

[spyrbri]

The problem lies in the yaml gem which behaves differently on different operating systems.

# ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
# irb
irb(main):001:0> require "yaml"
=> true
irb(main):002:0> tree =  {"en"=>{"empty"=>nil}}.to_yaml
=> "---\nen:\n  empty: \n"

# ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-darwin19]
# irb
irb(main):001:0> require "yaml"
=> true
irb(main):002:0>  tree =  {"en"=>{"empty"=>nil}}.to_yaml
=> "---\nen:\n  empty:\n"

Notice the space after empty: -> "---\nen:\n empty: \n" vs "---\nen:\n empty:\n"

https://github.com/ruby/psych/issues/467 https://github.com/glebm/i18n-tasks/blob/master/lib/i18n/tasks/data/adapter/yaml_adapter.rb#L21

[goshatch]

Just did a quick test on my local machine (Arch Linux) and on the Circle CI VM. Different behaviours despite similar systems:

Local:

$ uname -a
Linux geofront 5.9.1-arch1-1 #1 SMP PREEMPT Sat, 17 Oct 2020 13:30:37 +0000 x86_64 GNU/Linux
$ ruby -v
ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
$ irb
irb(main):001:0> require "yaml"
=> true
irb(main):002:0> tree =  {"en"=>{"empty"=>nil}}.to_yaml
=> "---\nen:\n  empty:\n"

Circle CI:

circleci:~$ uname -a
Linux b2f5d83f5d3f 4.15.0-1077-aws #81-Ubuntu SMP Wed Jun 24 16:48:15 UTC 2020 x86_64 GNU/Linux
circleci:~$ ruby -v
ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
circleci:~$ irb
irb(main):001:0> require "yaml"
=> true
irb(main):002:0> tree =  {"en"=>{"empty"=>nil}}.to_yaml
=> "---\nen:\n  empty: \n"

One difference that I can see is the much older kernel version on the Circle CI VM.

[spyrbri]

The problem is with the libyaml versions. There is a difference on how the versions 0.2.4 and 0.2.5 handle the nil values.

An easy way to check the version of your libyaml version is the following:

# irb
irb(main):001:0> require "yaml"
=> true
irb(main):002:0> Psych.libyaml_version
=> [0, 2, 5]

Thus not sure if something can be done on the gem. The point is to make sure your dev machine and the CI have the same libyaml version.

[mvz]

Getting the correct libyaml version onto the current ubuntu-latest runner on GitHub Actions is quite involved. How about adding a small post-processing step to the normalization process that removes these trailing spaces, so the behavior is consistent no matter what version of libyaml is used?

[dssjoblom]

Having this problem as well