OpenAI: unexpected token at '```json

[yshmarov]

I get an error when trying to translate with OpenAI

Users/yshmarov/.rbenv/versions/3.3.3/lib/ruby/gems/3.3.0/gems/json-2.7.2/lib/json/common.rb:220:in `parse': unexpected token at '```json (JSON::ParserError)
[
  "Wir werden Sie kontaktieren, um Ihnen mitzuteilen, dass Ihre E-Mail-Adresse auf X__0 geändert wurde.",
  "Erwartetes Ergebnis",
  "Eo",
  "ko",
  "Sie können den API-Schlüssel finden",
  "hier",
  "Bezahlt",
  "Haben Sie bereits ein SR-Konto?",
  "SR arbeitet mit Partnern zusammen, um Ihre Bank sicher zu verbinden. Dies sind regulierte Organisationen, die sicherstellen, dass Ihre Daten geschützt sind.\n\nDies entspricht d

OPENAI_MODEL=gpt-4o OPENAI_API_KEY=sk-***foobar i18n-tasks translate-missing --backend=openai

Does anybody else have this issue?

[yshmarov]

the above issue seems to be related to gpt-4o.

However when I run gpt-3.5-turbo, I get another error:

 i18n-tasks translate-missing --backend=openai
warning: parser/current is loading parser/ruby33, which recognizes 3.3.4-compliant syntax, but you are running 3.3.0.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
#StandWithUkraine
/Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:98:in `next': iteration reached an end (StopIteration)
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:98:in `parse_value'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:90:in `block in parse_value'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:90:in `map'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:90:in `parse_value'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:63:in `from_values'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:46:in `fetch_translations'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:35:in `block in translate_pairs'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:34:in `each'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:34:in `map'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:34:in `translate_pairs'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:17:in `block in translate_forest'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/data/tree/nodes.rb:16:in `each'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/data/tree/nodes.rb:16:in `each'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:16:in `inject'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translators/base_translator.rb:16:in `translate_forest'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/translation.rb:21:in `translate_forest'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/command/commands/missing.rb:51:in `translate_missing'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/command/commander.rb:26:in `run'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/cli.rb:51:in `block in run'
	from /Users/yaroslavshmarov/.gem/ruby/3.3.0/gems/i18n-1.14.5/lib/i18n.rb:351:in `with_locale'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/cli.rb:49:in `run'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/cli.rb:18:in `block in start'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/cli.rb:211:in `auto_output_coloring'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/cli.rb:17:in `start'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/lib/i18n/tasks/cli.rb:11:in `start'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/i18n-tasks-1.0.14/bin/i18n-tasks:15:in `<top (required)>'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/bin/i18n-tasks:25:in `load'
	from /Users/yaroslavshmarov/.rbenv/versions/3.3.0/bin/i18n-tasks:25:in `<main>'

[davidwessman]

Could you add e.g. your i18n-tasks.yml and some more info that would help recreate it?

[yshmarov]

Could you add e.g. your i18n-tasks.yml and some more info that would help recreate it?

Hi David! I am using exclusively the defaults in the yml file, only uncommenting open-ai:

base_locale: en
data:
  read:
  write:
  external:
  yaml:
    write:
      line_width: -1
search:
  exclude:
    - app/assets/images
    - app/assets/fonts
    - app/assets/videos
    - app/assets/builds
translation:
  openai_api_key: "sk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  openai_model: "gpt-3.5-turbo"

It used to work before

[yshmarov]

possibly it does not parse my YML file correctly? I have 6000 lines in en.yml....

[davidwessman]

possibly it does not parse my YML file correctly? I have 6000 lines in en.yml....

Yeah, probably some markup that is not working :/

[davidwessman]

@yshmarov Please come back if you still experience this error and have a way to reproduce it 🙂

[dixpac]

It would be nice/helpful to get error where did it fail...When translating huge local currently we only get error similar to this next': iteration reached an end (StopIteration)