faker icon indicating copy to clipboard operation
faker copied to clipboard

feat(locale): Add ku locale

Open arentalb opened this issue 8 months ago • 24 comments
trafficstars

This PR adds Kurdish support for the lorem module in Faker.js. This is the first step towards adding full Kurdish locale support, with more modules to come in future updates.

Ran pnpm run preflight with no issues Generated locales with pnpm run generate:locales Formatted code (pnpm run format) and passed linting (pnpm run lint) Verified tests pass

Let me know if any changes are needed!

arentalb avatar Mar 17 '25 20:03 arentalb

Deploy Preview for fakerjs ready!

Built without sensitive environment variables

Name Link
Latest commit 31ea85d2c519708d2bba3dde7f190b0ac62b13f4
Latest deploy log https://app.netlify.com/projects/fakerjs/deploys/68e0b2437eeccc00080a789e
Deploy Preview https://deploy-preview-3441.fakerjs.dev
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify[bot] avatar Mar 17 '25 20:03 netlify[bot]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 99.97%. Comparing base (dce234e) to head (31ea85d). :warning: Report is 1 commits behind head on next.

Additional details and impacted files
@@           Coverage Diff           @@
##             next    #3441   +/-   ##
=======================================
  Coverage   99.97%   99.97%           
=======================================
  Files        2894     2899    +5     
  Lines      222390   222476   +86     
  Branches      932      932           
=======================================
+ Hits       222337   222423   +86     
  Misses         53       53           
Files with missing lines Coverage Δ
src/locale/index.ts 100.00% <100.00%> (ø)
src/locale/ku_ckb.ts 100.00% <100.00%> (ø)
src/locales/index.ts 100.00% <100.00%> (ø)
src/locales/ku_ckb/index.ts 100.00% <100.00%> (ø)
src/locales/ku_ckb/lorem/index.ts 100.00% <100.00%> (ø)
src/locales/ku_ckb/lorem/word.ts 100.00% <100.00%> (ø)
src/locales/ku_ckb/metadata.ts 100.00% <100.00%> (ø)
:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Mar 17 '25 20:03 codecov[bot]

According to Wikipedia https://en.wikipedia.org/wiki/Kurdish_language

The main varieties of Kurdish are Kurmanji, Sorani, and Southern Kurdish (Xwarîn). The majority of the Kurds speak Kurmanji,[15] and most Kurdish texts are written in Kurmanji and Sorani. Kurmanji is written in the Hawar alphabet, a derivation of the Latin script, and Sorani is written in the Sorani alphabet, a derivation of the Arabic script.

So I wonder if we should give this a script suffix similar to we did for Uzbek and Serbian https://fakerjs.dev/guide/localization.html#available-locales to distinguish Kurdish written in Latin characters and Arabic characters?

matthewmayer avatar Mar 18 '25 00:03 matthewmayer

Kurdish has multiple dialects, but the main ones are Kurmanji and Sorani. Kurmanji can be written in both Latin and Arabic scripts. I plan to add these three variations

ku_KMR_latin → Kurmanji (Latin) ku_KMR_arab → Kurmanji (Arabic) ku_CKB_arab → Sorani (Arabic)

Would this be the correct approach for Faker.js? I want to confirm before proceeding. @matthewmayer

arentalb avatar Mar 18 '25 19:03 arentalb

Hmm I don't think we yet have any other locales where it's necessary to disambiguate by ISO 639-3 code for different dialects. I think it might be confusing to put the language code where we would otherwise put the country code so it probably makes sense to put it as part of the variant suffix. Let's also check how other localisation of open source software handles this.

matthewmayer avatar Mar 19 '25 10:03 matthewmayer

I searched for a solution but couldn’t find anything that exactly fits our needs.

However, I believe we can handle this using the following approach:

ku_IQ_Arab: Sorani (Arabic) IQ represents Iraq, where most Sorani speakers live. ku_TR_Latn: Kurmanji (Latin) TR represents Turkey, where most Kurmanji speakers use the Latin script. ku_SY_Arab: Kurmanji (Arabic) SY represents Syria, where many Kurmanji speakers use the Arabic script.

This approach seems like our best fit.

However, I don’t like using IQ, TR, and SY because Kurdish people are divided among four countries, including Iran. I wish I could use KU instead, but unfortunately, it is not a standard code.

arentalb avatar Mar 19 '25 22:03 arentalb

I agree the country codes aren't ideal. It partly depends if you expect to add data for eg phone module which has country codes , location module which has cities etc. if those are all from one country then a country code would be appropriate. If not then we should try to find a generic solution.

matthewmayer avatar Mar 20 '25 01:03 matthewmayer

Wikipedia uses ku for Kurmanji (with a toggle between ku-latn and ku-arab) and ckb for Sorani

matthewmayer avatar Mar 20 '25 03:03 matthewmayer

Mozilla uses https://pontoon.mozilla.org/sdh/ https://pontoon.mozilla.org/kmr/ https://pontoon.mozilla.org/ckb/

matthewmayer avatar Mar 20 '25 03:03 matthewmayer

I think the cleanest would probably be:

  • ku_kmr_latin → Kurdish (Kurmanji, Latin) - variant kmr_latin in metadata, fakerKU_kmr_latin when precompiled
  • ku_kmr_arab → Kurdish (Kurmanji, Arabic) - variant kmr_arab in metadata, fakerKU_kmr_arab when precompiled
  • ku_ckb → Kurdish (Sorani) - variant ckb in metadata, fakerKU_ckb when precompiled

matthewmayer avatar Mar 20 '25 03:03 matthewmayer

i though we should use only contry code like IQ after the ku_ , but if it is fine to use kmr and ckb like you said that would be the best way , thank you , i will start with ku_ckb

arentalb avatar Mar 20 '25 10:03 arentalb

@matthewmayer I’ve updated the changes and pushed them. Do I need to do anything else besides adding more modules?

arentalb avatar Mar 20 '25 11:03 arentalb

For ease for review please don't add any more modules for now. We prefer to get a small PR reviewed first, once it is approved you can follow up with additional PRs for other dialects and modules.

Nothing to do for now, I will let the other maintainers take a look. Please bear with us it can take a few days or weeks to get initial PR approved.

matthewmayer avatar Mar 20 '25 13:03 matthewmayer

i though we should use only contry code like IQ after the ku_ , but if it is fine to use kmr and ckb like you said that would be the best way , thank you , i will start with ku_ckb

Hello 🙂

Thanks to @matthewmayer to drive this PR in review process for us 🚀 I think we (the @faker-js/maintainers) should have a meeting about how we handle this, because at first look my brain immediately told me also that this differs from our pattern we are used to (except of en_AU_ocker and en_BORK, but even there it is /^[a-z]{2}_[A-Z]{2}.*/) 🤔

Otherwise I would give already an approve ✅

Shinigami92 avatar Mar 22 '25 11:03 Shinigami92

i though we should use only contry code like IQ after the ku_ , but if it is fine to use kmr and ckb like you said that would be the best way , thank you , i will start with ku_ckb

Hello 🙂

Thanks to @matthewmayer to drive this PR in review process for us 🚀 I think we (the @faker-js/maintainers) should have a meeting about how we handle this, because at first look my brain immediately told me also that this differs from our pattern we are used to (except of en_AU_ocker and en_BORK, but even there it is /^[a-z]{2}_[A-Z]{2}.*/) 🤔

Otherwise I would give already an approve ✅

i think en_BORK is the only other example of a locale with a language and variant but no country at the moment. (Admittedly a silly example, and "BORK" should probably be lowercase but i dont want to introduce a breaking change for a silly Easter egg locale).

But in this case when people speaking Kurdish are spread over several countries, it seems more appropriate.

matthewmayer avatar Mar 22 '25 11:03 matthewmayer

i think en_BORK is the only other example of a locale with a language and variant but no country at the moment. (Admittedly a silly example, and "BORK" should probably be lowercase but i dont want to introduce a breaking change for a silly Easter egg locale).

But in this case when people speaking Kurdish are spread over several countries, it seems more appropriate.

These are exactly also my background thoughts 👍

Shinigami92 avatar Mar 22 '25 11:03 Shinigami92

I think we don't have a really good definition for what goes in "variant" at the moment, but i think its basically

"the minimum needed to disambiguate versions of a macro-language", which might include a part 3 code from https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes (e.g. ckb) and/or a script (e.g. latin, arab) and/or another descriptor if there's no standard code (en_GB_cockney_rhyming_slang)?

matthewmayer avatar Mar 22 '25 11:03 matthewmayer

@arentalb I'll be honest with you here. We discussed this PR during the last two weekly meetings and still haven't come to a conclusion. The case you present is quite the hard one.

For starters, lets have a look into our current definition for locale names from our website: https://fakerjs.dev/guide/localization.html#locale-codes. We noticed that the second paragraph was not as clearly written as we'd like it to be.

The same language may be spoken in different countries, with different patterns for addresses, phone numbers etc. Optionally a two-letter uppercase country code can be added after an underscore, following the ISO 3166-1 alpha-2 standard, for example en_US represents English (United States) and en_AU represents English (Australia).

One could argue that the "Optionally" indicates that the country code is optionally entirely, while another one could say that it is only optionally when considering the base locale code (from the first paragraph). By making the country code required, we could at least somewhat restrict the maximum amount of possible faker locales. Furthermore, a lot of modules (example internet, location) are only usful if they are paired with a specific country.

Having to maintain an indefinite amount of locales is sadly a fact we need to consider as maintainers. Since we already have quite a lot of locales, making this decision right now, is not as easy as it might seem. Furthermore, we need to consider the implementation for future locales. Making the country code optionally could potentially lead to a drastic increase of "easter egg" locales like en_BORK as they would then be defined a valid locale.

We totally see the reasoning of your disliking regarding the implementation by definition. But by the current definition, this might be the best option. To get this matter resolved more easily, could you imagine to create a generic ku locale that all other ku_* will fall back to? Or is this not possible due to the nature of the language itself being heavily defined into dialects?

xDivisionByZerox avatar Apr 03 '25 16:04 xDivisionByZerox

Thanks for reviewing the PR — much appreciated!

Regarding your suggestion, yes — I can definitely work on a generic ku locale.

I believe the best candidate for a base locale would be Kurdish Sorani (Central Kurdish) in Arabic script. It's my native dialect and also the most widely understood among Kurdish speakers.

do you agree with using Sorani Arabic script for the generic ku, or would you recommend a different approach?

arentalb avatar Apr 03 '25 18:04 arentalb

First of all, thank you for your kind and considered answer. 🙏

According to wikipedia the largest dialect group is Kurmanji (Northern Kurdish) with 15 to 20 million speakers. Sorani (Central Kurdish) has "only" 6 to 7 million speakers. This fact aside, I was more asking if there would be some overlaps between the different Kurdish dialects. Like do they use the same words for the days of the week (just an example!). If this would be the case, these entries could be put in a generic Kurdish locale. Then you could create the different variants that you suggested previously by reusing the data from the generic Kurdish locale plus some extra words/modules (same as you already did in src/locale/ku_ckb.ts but then with one more locale). By having the generic Kurdish locale, we would have less worries about allowing the current implementation without a country code. If this is not possible and a generic Kurdish locale cant exist, we need to discuss the topic again internally to find a final decision on how to handle these kind of cases.

xDivisionByZerox avatar Apr 03 '25 18:04 xDivisionByZerox

I’ll talk to some of my Kurmanji friends and let you know what makes the most sense

arentalb avatar Apr 03 '25 20:04 arentalb

Thank you for your cooperation. I'm sorry that your first contribution is that complicated 😐

xDivisionByZerox avatar Apr 03 '25 21:04 xDivisionByZerox

In theory could a generic Kurdish locale just contain things like phone numbers and domain names which would be the same in both scripts?

matthewmayer avatar Apr 03 '25 23:04 matthewmayer

english-sorani-kurmanji.pdf

Sorry for the late reply

I’ve translated a set of base words to compare Sorani and Kurmanji - as shown in the file, there are noticeable differences between the two dialects.

While a few words are shared, the overlap doesn’t seem strong enough to build a reliable base fallback, in my opinion. So I don’t think a generic Kurdish locale is the right approach.

That leaves us with two possible options:

1- Fallback to one dialect - I’d suggest Sorani. In my experience, Kurmanji speakers usually understand Sorani, but the reverse isn’t true - Sorani speakers typically understand only about 20–30% of Kurmanji.

2- Treat them as separate locales - This might be the better option overall, especially since Kurmanji has two scripts (Latin and Arabic).

arentalb avatar Apr 04 '25 16:04 arentalb

Thank you for the approval and the detailed instructions. I will resolve the conflicts and update the snapshots as requested

arentalb avatar Oct 03 '25 15:10 arentalb

I checked the workflows and noticed that the test failure comes from the snapshot file in the "ko" section. I didn’t modify that part when creating the pull request , it was part of the conflict resolution, and I accepted the existing changes. Now the workflow fails because of that section.

@xDivisionByZerox

arentalb avatar Oct 04 '25 05:10 arentalb

I checked the workflows and noticed that the test failure comes from the snapshot file in the "ko" section. I didn’t modify that part when creating the pull request , it was part of the conflict resolution, and I accepted the existing changes. Now the workflow fails because of that section.

@xDivisionByZerox

it looks like Git accidentally pulled in some changes to ko because its adjacent to ku in the snapshot test. i have reverted that.

matthewmayer avatar Oct 04 '25 05:10 matthewmayer

@matthewmayer thank you , but it still have a failed test

arentalb avatar Oct 04 '25 09:10 arentalb

Yes noted. That is unrelated to your changes.

matthewmayer avatar Oct 04 '25 12:10 matthewmayer

Thank you for your contribution.

As concluded - through out the discussion in this PR - we will accept additional PRs for the locales ku_kmr_latin and ku_kmr_arab, if you are still interested in providing them.

xDivisionByZerox avatar Oct 04 '25 14:10 xDivisionByZerox