xcstrings-tool Support for reading legacy strings files and providing fallback strings

First off, I was about to write a similar plugin before I did a quick search and found yours. Glad I did because this looks great! 👏

I'm looking into replacing the use of SwiftGen in a project I'm working on with this plugin but there are two features I'd need before being able to do so. Before contributing I'd like to gauge interest to see whether you think the features are suitable additions to your plugin. These are:

The ability to parse legacy string files (for those who haven't migrated to string catalogs).
The ability to fall back to a base localization if translation isn't complete. So for example, going with the English string if the German counterpart is missing.

What do you think? To be clear, if these features are something you think would be valuable additions I'm more than happy to contribute.

Oct 05 '23 06:10 simba909

Hi @simba909, thanks for reaching out! To go into details about your two suggestions:

The ability to parse legacy string files (for those who haven't migrated to string catalogs).

I initially started with on Strings Catalog support only for the following reasons:

I needed to keep the scope narrow to actually ship something
I wanted the tool to be a fresh take on Localization code generation following the modern best practices (Strings Catalogs and LocalizedStringResource)
SwiftGen and R.swift seemed to already be handling legacy formats for existing projects pretty well
There are some extra complexities involved with parsing .strings{dict} files, such as having to discover content across .lproj directories (instead of one single input file), parsing comments from .strings and processing the .stringsdict plist structure.
It was a bit too early to see how .strings and .xcstrings coexisting would look in reality for existing projects and my personal use-case for this tool was an iOS 17 only project

So that means that the tool today is pretty geared towards new projects rather than existing ones. With that said though, in my day-job, I have a much larger project using .strings and .stringsdict with an old version of R.swift and I do want to explore how I'd eventually migrate that to Strings Catalogs/XCStrings Tool so it's good to talk about this 🙂 It sounds like you are a bit further ahead of me in the process 👍

I'd like to understand a bit more from you though on why it is that you want to use XCStrings Tool to generate output for legacy .strings files? Is it because you don't want to use two different tools, or is it because you want a generated output that uses LocalizedStringResource like this plugin offers?

The ability to fall back to a base localization if translation isn't complete. So for example, going with the English string if the German counterpart is missing.

This should already happen when using XCStrings Tool I think. In my project, the source/development language is set to English, so if I define settingsTitle in my catalog with an English value of Settings, the tool will generate symbols that look something like this:

/// Title of the Settings screen
public var settingsTitle: LocalizedStringResource {
    LocalizedStringResource(
        "settingsTitle",
        defaultValue: #"Settings"#, // <-- this here is the important piece
        table: "Localizable",
        bundle: .current
    )
}

When Foundation does a lookup for settingsTitle in a different language, if it doesn't exist, then it'll fallback to using defaultValue instead of the key. Is that what you were after?

What isn't supported is if you wanted to be more intelligent with fallbacks between the user locale and the development language. For example, if you app supported es_MX, es-419 (Latin America), es and had a development language of en. If a user with es_MX locale wanted a string that didn't exist in es_MX but did in es-419 or es, Foundation wouldn't care and would only use the defaultValue which would be in English.

But supporting this lies in how Foundation does localized lookups from your LocalizedStringResource rather than anything particular about how we generate the symbols. We have all the information that we need in LocalizedStringResource to write your own custom localization lookup mechanism that supports this instead. It's what I had to do in our project so I can provide more tips separately if you wanted to dig into that more 🙂

Oct 05 '23 10:10 liamnichols

I'd like to understand a bit more from you though on why it is that you want to use XCStrings Tool to generate output for legacy .strings files? Is it because you don't want to use two different tools, or is it because you want a generated output that uses LocalizedStringResource like this plugin offers?

The main reason we were looking at plain .strings parsing is because we haven't yet had the time to migrate to string catalogs but would like to get the LocalizedStringResource generation. We're also depending on a 3rd party to generate localizations that I don't think has support for string catalogs yet (although that's a different matter). Having looked closer at the differences between the formats though I'm not sure that building in support for parsing the legacy formats is a good direction to take for the plugin but maybe I'm overthinking things 🤔

(in the context of fallback strings)

This should already happen when using XCStrings Tool I think. In my project, the source/development language is set to English, so if I define settingsTitle in my catalog with an English value of Settings, the tool will generate symbols that look something like this:
/// Title of the Settings screen
public var settingsTitle: LocalizedStringResource {
    LocalizedStringResource(
        "settingsTitle",
        defaultValue: #"Settings"#, // <-- this here is the important piece
        table: "Localizable",
        bundle: .current
    )
}
When Foundation does a lookup for settingsTitle in a different language, if it doesn't exist, then it'll fallback to using defaultValue instead of the key. Is that what you were after?

Ah, this is excellent! Totally missed this when looking through the plugin initially. Will have to play around with this a bit because I think it covers all our needs! Thank you 👏🏻

Oct 09 '23 08:10 simba909

because we haven't yet had the time to migrate to string catalogs but would like to get the LocalizedStringResource generation.

That's a great point, and something that I kind of overlooked before. This tool generates LocalizedStringResources which are a modern alternative to NSLocalizedString so there is still a benefit over what existing tools over having XCStrings Tool support the format in the interim 👍

Anyway, I was stuck at the airport yesterday with a 7h delay, so I took the opportunity to take a further dive into seeing if this is doable:

https://github.com/liamnichols/xcstrings-tool/pull/20

From the parsing side, it seems relatively straightforward. We can use PropertyListSerialization to read both file formats (at the expense of not being able to access comments, but I think that is fine) and the parsing logic for pulling out substitutions and plural rules from the .xcstrings format can mostly be reused with some light refactoring.

This gets us to a point where you can invoke the underlying CLI with either formats, but there are a couple of bits specific to the plugin integration that I need to think about still:

Merging Localizable.strings and Localizable.stringsdict into one generated output (since these two files would make up the same localization table)
- This means that the CLI needs to accept multiple input files
Discovering the source/development language to know which .lproj directory to extract strings from
- Maybe we could also just throw all discovered .strings/.stringsdict files into the CLI, but i'm not sure if this will work quite right

I think that we can do this in two steps though:

Finish up the basic .stringsdict support in #20
Support the use case for merging .strings and .stringsdict files
Update the plugin to support discovering the .strings{dict} files
- If this turns out to not be easily possible, the first two steps are still useful if the xcstrings-tool cli is used directly, it's just that I initially tried to keep it hidden as an implementation detail but there is no harm exposing it for more advanced use cases like this.

Oct 09 '23 09:10 liamnichols

Apologies for the late response.

Anyway, I was stuck at the airport yesterday with a 7h delay, so I took the opportunity to take a further dive into seeing if this is doable:

Support for legacy .strings and .stringsdict file formats #20

From a quick glance this looks great! Thank you 🙌🏻 I'll take a closer look at your PR and then come back to this thread when I've gathered my thoughts.

Oct 26 '23 08:10 simba909

Curious if this is still something you're looking to get resolved?

May 03 '24 12:05 Brett-Best

I think so!

I become less motivated to work on it once I found a bit of a blocker that I described at the end of my last comment, but I am still occasionally thinking about migration paths for projects that want to use Strings Catalogs/XCStrings Tool as a modern alternative to the strings generation tool that they used before and I think that this support would be good for that.

It's just that I haven't really found the time to focus on it as it hasn't been as much of a priority for me atm.

May 03 '24 14:05 liamnichols

Hey, so I have made some progress on this today and am happy to say that the following will be supported as part of #20:

Generating source code for *.strings files
Generating source code for *.stringsdict files
Merging *.strings and *.stringsdict into a single generated ouput
- i.e Localizable.strings ➕ Localizable.stringsdict ➡️ Localizable.swift

I'm pretty happy with this as it unlocks a nice migration path into XCStrings Tool (and eventually for you to convert your .strings to .xcstrings), just I need some help to test the .stringsdict parsing some more since I haven't really used it much myself.

So far, I have the following fixture: https://github.com/liamnichols/xcstrings-tool/pull/20/files#diff-9b61fb72cfbd128cecc0be5bf6ba07257df78f2980ab774de382b1627c42db70

If you have .stringsdict files that are more complicated than this fixture, i'd love it if you could share any examples so that I am making sure to cover as many common edge cases as possible 🙏

Jun 02 '24 21:06 liamnichols

I'm sorry I haven't been able to contribute to this effort. I am however very excited to see it merged, I will definitely try this out at the nearest opportunity! Thank you 🙏🏻

Jun 09 '24 14:06 simba909

No worries @simba909, it’s released in 0.5.0 so check it out and if there are any issues we can solve them over time!

Jun 10 '24 04:06 liamnichols