Annotations are not added on top of model files when columns'`comment:` contains Japanese characters in migrations

[tonystrawberry]

Description

When I create a database migration file that contains a new column with comment: containing Japanese characters and execute it, the annotations corresponding to that column should be added but are not added.

To Reproduce

I suppose annotaterb is installed and runs after migrations.

1. Create a database migration for adding a new column to a table and include Japanese characters.

# db_teams/migrate/20250304085321_add_is_winner_to_teams.rb
class AddIsWinnerToTeams < ActiveRecord::Migration[7.1]
  def change
    add_column :teams, :is_winner, :boolean, after: :id, comment: "勝者かどうか"
  end
end

2. Execute the migration.

3. Check that no annotation is generated.

Possible solution

It seems the regex COLUMN_PATTERN in lib/annotate_rb/model_annotator/annotation_diff_generator.rb discards the lines that contains Japanese characters. Since we needed a quick solution for our Rails project, we went ahead, forked the repository and applied a quick patch: https://github.com/3dstylee/annotaterb/pull/1/files

The solution was only changing from

COLUMN_PATTERN = /^#[\t ]+[\w*.`\[\]():]+(?:\(.*?\))?[\t ]+.+$/

to

COLUMN_PATTERN = /^#[\t ]+[\p{Word}*.`\[\]():]+(?:\(.*?\))?[\t ]+.+$/

I am not good at regex but it seems \w was only matching alphanumeric characters. Using \p instead allows Japanese (and possibly other alphabet) characters to be matched.

It seems to work after some checks but we haven't run thorough tests yet.

[hatsu38]

I tried this migration and it worked without any problems.

migration file

class AddIsWinnerToTeams < ActiveRecord::Migration[7.1]
  def change
    add_column :teams, :is_winner, :boolean, after: :id, comment: "勝者かどうか"
  end
end

log

== 20250322091932 AddIsWinnerToTeams: reverting ===============================
-- add_column(:teams, :is_winner, :boolean, {comment: "勝者かどうか"})
   -> 0.0000s
-- remove_column(:teams, :is_winner, :boolean, {comment: "勝者かどうか"})
   -> 0.0013s
== 20250322091932 AddIsWinnerToTeams: reverted (0.0053s) ======================

Annotating models
Annotated (3): app/models/team.rb, spec/models/team_spec.rb, spec/factories/teams.rb

result

# == Schema Information
#
# Table name: teams
#
#  id                      :uuid             not null, primary key
#  name                    :string           not null
#  created_at              :datetime         not null
#  updated_at              :datetime         not null
#  is_winner(勝者かどうか) :boolean
#
class Team < ApplicationRecord
end

[drwl]

@tonystrawberry can you let me know if you are still facing this issue? I believe Japanese comments should be supported.

If you would like your patch to be added to the project, feel free to open a PR with some test cases and I can review.

[tonystrawberry]

@drwl Okay! I will check ASAP and open a PR here if needed.

I have added a patch to a fork of this repository and it works fine now. https://github.com/3dstylee/annotaterb/pull/1

[drwl]

Can you submit a PR? If the unit tests pass I'm happy to add the patch if it does not cause any regressions.

[tonystrawberry]

@drwl Opened a PR here: https://github.com/drwl/annotaterb/pull/239 Please check when you have time. Thank you!

[tonystrawberry]

PR merged: https://github.com/drwl/annotaterb/pull/239 I will close this issue!