WIP: test(locale): check for duplicated entries

[xDivisionByZerox]

trafficstars

This PR will introduce a test that checks every LocaleDefinition for duplicated entries.

[xDivisionByZerox]

I expect the test pipeline to fail for demonstration purposes.

@ST-DDT would you have a look at the implementation and the error log in general? I trust in you and your abilities on such things.

[ST-DDT]

The implementation looks good, maybe you could use this expectation:

expect(
  uniqueDuplication,
  `Found duplicate entries: [${uniqueDuplication
    .sort()
    .join(', ')}]`
).toBe([]);
// The sort should probably be outside of the string template

The toBe check focuses more on the content of the array than the length.

What do you think?

(PS: We have to check if there are/should be exceptions to this rule)

[xDivisionByZerox]

So I need to change about 1,4k locale files since they contain duplicates. Is it reasonable to fix all of this in one (THIS) PR? My current workflow is:

1. sort all entries in a locale files
2. commit with message 'chore(locales): *localeName* *fileName* sort alphabetically'
3. remove duplicated entries
4. commit with message 'chore(locales): *localeName* *fileName* remove duplicates'

My question: Should I provide an additional PR where I sort ALL locale files alphabetically first. This could be much clearer for the git history.

I use the Tyriar.sort-lines VS Code extension to sort locale file entries.

[Shinigami92]

@xDivisionByZerox yes, for me that sounds more safe

1. PR for sort
2. PR for dedup

That way we have a better commit history in main branch

[codecov[bot]]

Codecov Report

Merging #1137 (f999ba9) into main (f0b60b9) will decrease coverage by 0.01%. The diff coverage is 100.00%.

:exclamation: Current head f999ba9 differs from pull request most recent head e418402. Consider uploading reports for the commit e418402 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1137      +/-   ##
==========================================
- Coverage   99.63%   99.61%   -0.02%     
==========================================
  Files        2165     2165              
  Lines      241432   237194    -4238     
  Branches     1021     1017       -4     
==========================================
- Hits       240549   236287    -4262     
- Misses        862      886      +24     
  Partials       21       21              

Impacted Files	Coverage Δ
src/locales/af_ZA/name/female_first_name.ts	100.00% <ø> (ø)
src/locales/af_ZA/name/first_name.ts	100.00% <ø> (ø)
src/locales/af_ZA/name/last_name.ts	100.00% <ø> (ø)
src/locales/af_ZA/name/male_first_name.ts	100.00% <ø> (ø)
src/locales/ar/address/city_name.ts	100.00% <ø> (ø)
src/locales/ar/commerce/department.ts	100.00% <ø> (ø)
src/locales/ar/name/female_first_name.ts	100.00% <ø> (ø)
src/locales/ar/name/last_name.ts	100.00% <ø> (ø)
src/locales/ar/phone_number/formats.ts	100.00% <ø> (ø)
src/locales/ar/team/creature.ts	100.00% <ø> (ø)
... and 213 more

[ST-DDT]

It looks like this PR doesn't really reduce the line count: +10,463 -4,859 (not that it is important)

[Shinigami92]

It looks like this PR doesn't really reduce the line count: +10,463 -4,859 (not that it is important)

Yeah, you are right Somehow some files are generated with fully new content 🤔 Example:

[xDivisionByZerox]

Somehow some files are generated with fully new content 🤔

Oh damn. Good catch. I will revert those files. I guess they slipped through as I played with the generate-locales script.

[xDivisionByZerox]

Those "additions" should be reverted now. They happened due to those files being implemented in a dynamic/logical way, not with a static data set.

[ST-DDT]

@xDivisionByZerox I found a way to simplify the test a bit, feel free to merge: diff

[ST-DDT]

@xDivisionByZerox Did you see my comment here: https://github.com/faker-js/faker/pull/1137#issuecomment-1258454562