Add language selection support

[bgrgicak]

Motivation for the change, related issues

To unlock Playground to non-English speakers we want to make it easier for sites to be loaded in a local language.

Users can easily switch the site language by adding a language=LANGUAGE_CODE query parameter.

Implementation details

This PR adds a language parameter to the Query API which adds a setSiteLanguage blueprint step.

We could have achieved this without a new step, but it would require adding two steps to the blueprint generator (add constant, run PHP).

The step downloads core, plugin, and theme translations from Download.WordPress.org and moves the files into the translation directory.

Testing Instructions (or ideally a Blueprint)

• Checkout this branch
• Open this Playground url
• Confirm that the theme is translated (the menu label should be Menú)
• Check the admin bar and confirm it's translated
• Open Akismet settings (/wp-admin/options-general.php?page=akismet-key-config&view=start) and confirm they are translated

[bgrgicak]

@adamziel do you know why the documentation build is failing? I tried debugging it locally but didn't get far.

[akirk]

I'd like to add my https://github.com/akirk/playground-step-library/blob/main/steps/setLanguage.js implementation to the discussion here. This is quite imperfect as it should use language packs (see https://api.wordpress.org/translations/core/1.0/ and https://api.wordpress.org/translations/plugins/1.0/?slug=friends) so that we get all the JSON files as well.

But note how we also need to take into account plugins and themes that are installed, we need to get their translations, too.

[bgrgicak]

I'd like to add my https://github.com/akirk/playground-step-library/blob/main/steps/setLanguage.js implementation to the discussion here. This is quite imperfect as it should use language packs (see https://api.wordpress.org/translations/core/1.0/ and https://api.wordpress.org/translations/plugins/1.0/?slug=friends) so that we get all the JSON files as well.

Thank you @akirk this is amazing! I will come back to this and update the code, but let me know in case you would like to take over.

[akirk]

I am not sure I should, I don't know the code base well enough and will be off next week and I don't want to stall this.

I just tried to say that this is possible with the step library but that you also need to iterate the plugins to be installed, something that I think you had not addressed yet.

[brandonpayton]

@adamziel, @bgrgicak, and @akirk, I think this is a good example of a place that calls for separate config languages/concepts (DSLs, not human language translations):

1. Platform/Foundation
2. Actions that follow Platform-level "truth" (most of today's Blueprints)

IMO, human language selection is part of platform truth -- "The current language is X" -- as is FS construction of various mount configurations. Things like mounts can have an order of application, but they are more like platform declarations than actions. They are foundational. They are the basis upon which everything else is built.

Actions are things done upon the established foundation or platform at runtime: Install this, write that, login, etc.

There is a bit of gray area in this conceptualization of Platform vs Runtime, because each runtime action builds upon the version of reality established by the previous action. But I think we can also separate the two concepts by asking:

Is this part of setting up WordPress Playground?

Both language and directory mounts are relevant to the Playground boot process. We want to establish the filesystem before booting, and as part of booting Wordpress, we could download the language for WordPress itself.

Then, at runtime, the installPlugin step could recognize the current language and download the right translation as part of its install work.

[brandonpayton]

Instead of saying "separate DSLs", we could instead say things like:

• Language selection belongs outside the "steps" list in the Blueprints schema
• Directory mounts belong outside the "steps" list in the Blueprints schema

I also meant to ask: What do you think?

[brandonpayton]

Related to the platform vs actions idea, one question is: Does the order in which the step is included matter? Does it need to come before or after other steps?

[bgrgicak]

Does the order in which the step is included matter? Does it need to come before or after other steps?

If we want to translate plugin and themes, it should come at the end.

[bgrgicak]

Sorry I'm focused on wrapping up offline support and will get back to this next week.

[bgrgicak]

Language selection belongs outside the "steps" list in the Blueprints schema Directory mounts belong outside the "steps" list in the Blueprints schema

I agree, but steps are the only thing we have today. We discussed in the past that there could be a way to separate boot actions from runtime actions in the blueprint. These look like good examples of it. By separating them, we could also run only the runtime part of the blueprint on load and skip the boot part if Playground was already booted (for example reload when using browser storage).

[bgrgicak]

I moved the code to compile and implemented downloads in JS instead of PHP.

I had to use downloads.wordpress.org because translate.wordpress.org doesn't support CORS when requests come from localhost. My problem with downloads.wordpress.org is that it requires the plugin/theme version number in the URL.

@akirk @pkevan Do you know of any methods that would allow us to download the latest translations of a plugin/theme without knowing the version number? If not we could add support to the downloads.wordpress.org API or allow CORS requests to translate.wordpress.org from localhost.

Theme/plugin translation code diff

diff --git a/packages/playground/blueprints/src/lib/steps/set-site-language.ts b/packages/playground/blueprints/src/lib/steps/set-site-language.ts
index 885c5a3d..f0de0e55 100644
--- a/packages/playground/blueprints/src/lib/steps/set-site-language.ts
+++ b/packages/playground/blueprints/src/lib/steps/set-site-language.ts
@@ -1,6 +1,12 @@
-import { StepDefinition, StepHandler } from '.';
+import {
+	InstallPluginStep,
+	InstallThemeStep,
+	StepDefinition,
+	StepHandler,
+} from '.';
 import { Blueprint } from '../blueprint';
 import { getWordPressVersion } from '@wp-playground/wordpress-builds';
+import { CorePluginReference, CoreThemeReference } from '../resources';
 
 export interface SetSiteLanguageStep {
 	step: 'setSiteLanguage';
@@ -66,10 +72,88 @@ export const compileSetSiteLanguageSteps = (
 		},
 	];
 
+	const plugins: string[] = [];
+	const themes: string[] = [];
+	for (const step of blueprint.steps) {
+		if (!step || typeof step === 'string') {
+			continue;
+		}
+		if (step.step === 'installPlugin') {
+			const pluginStep = step as InstallPluginStep<CorePluginReference>;
+			if (!pluginStep.pluginZipFile.slug) {
+				continue;
+			}
+			plugins.push(pluginStep.pluginZipFile.slug);
+		} else if (step.step === 'installTheme') {
+			const themeStep = step as InstallThemeStep<CoreThemeReference>;
+			if (!themeStep.themeZipFile.slug) {
+				continue;
+			}
+		}
+	}
+	if (plugins.length) {
+		siteLanguageSteps.push({
+			step: 'mkdir',
+			path: '/wordpress/wp-content/languages/plugins',
+		});
+	}
+	if (themes.length) {
+		siteLanguageSteps.push({
+			step: 'mkdir',
+			path: '/wordpress/wp-content/languages/themes',
+		});
+	}
+
+	for (const plugin of plugins) {
+		siteLanguageSteps.push(
+			{
+				step: 'unzip',
+				extractToPath: `/wordpress/wp-content/languages/plugins/${plugin}`,
+				zipFile: {
+					resource: 'url',
+					caption: `Downloading ${plugin}.mo`,
+					url: `https://downloads.wordpress.org/translation/plugins/${plugin}/stable/${language}.zip`, // TODO find working url
+				},
+			},
+			{
+				step: 'mv',
+				fromPath: `/wordpress/wp-content/languages/plugins/${plugin}/${plugin}-${language}.mo`,
+				toPath: `/wordpress/wp-content/languages/plugins/${plugin}-${language}.mo`,
+			},
+			{
+				step: 'rmdir',
+				path: `/wordpress/wp-content/languages/plugins/${plugin}`,
+			}
+		);
+	}
+
+	for (const theme of themes) {
+		siteLanguageSteps.push(
+			{
+				step: 'unzip',
+				extractToPath: `/wordpress/wp-content/languages/themes/${theme}`,
+				zipFile: {
+					resource: 'url',
+					caption: `Downloading ${theme}.mo`,
+					url: `https://downloads.wordpress.org/translation/themes/${theme}/stable/${language}.zip`, // TODO find working url
+				},
+			},
+			{
+				step: 'mv',
+				fromPath: `/wordpress/wp-content/languages/themes/${theme}/${theme}-${language}.mo`,
+				toPath: `/wordpress/wp-content/languages/themes/${theme}-${language}.mo`,
+			},
+			{
+				step: 'rmdir',
+				path: `/wordpress/wp-content/languages/themes/${theme}`,
+			}
+		);
+	}
+
 	blueprint.steps?.splice(setSiteLanguageStepIndex, 0, ...siteLanguageSteps);
 	return blueprint;
 };
 
 export const setSiteLanguage: StepHandler<SetSiteLanguageStep> = async () => {
-	// Implemented in packages/playground/blueprints/src/lib/compile.ts, search for `compileSetSiteLanguageSteps`
+	// Implemented in packages/playground/blueprints/src/lib/compile.ts, search for 'compileSetSiteLanguageSteps'
 };

[pkevan]

I moved the code to compile and implemented downloads in JS instead of PHP.

I had to use downloads.wordpress.org because translate.wordpress.org doesn't support CORS when requests come from localhost. My problem with downloads.wordpress.org is that it requires the plugin/theme version number in the URL.

@akirk @pkevan Do you know of any methods that would allow us to download the latest translations of a plugin/theme without knowing the version number? If not we could add support to the downloads.wordpress.org API or allow CORS requests to translate.wordpress.org from localhost.

Theme/plugin translation code diff

Not exactly sure what you are trying to do here, but you can use: https://api.wordpress.org/plugins/info/1.0/wordpress-beta-tester.json to grab all versions, then pick the appropriate one. Unsure if there are other methods available, or if it's worth building a specific endpoint on api.wordpress.org.

[bgrgicak]

Not exactly sure what you are trying to do here,

We want to add translation support to Playground. When a user specifies a language, we automatically download all site translations.

but you can use: api.wordpress.org/plugins/info/1.0/wordpress-beta-tester.json to grab all versions, then pick the appropriate one. Unsure if there are other methods available, or if it's worth building a specific endpoint on api.wordpress.org.

Thanks! I want to avoid that because it adds one extra request for each plugin and theme we want to install. This is why a endpoint that accepts just the plugin/theme name without a version would work well for us. It exists on https://translate.wordpress.org/projects/wp-plugins/${plugin}/dev/${lang}/default/export-translations?format=mo but we can't work with it because of CORS issues. Would it be possible to allow CORS requests from localhost on translate.wordpress.org?

[pkevan]

Adding CORS support is a request via systems, see: https://make.wordpress.org/systems/?s=cors

[bgrgicak]

I'm moving this back to draft until I can resolve the CORS issues.

[akirk]

The current plugin version here is actually not really helpful but I see @pkevan on the right track: we need an extra API request.

As I mentioned above, the "correct way" to do this is to use language packs. It has the unfortunate effect—which actually reflects standalone WordPress reality—that you only get a translated plugin when it has been translated sufficiently enough so that a language pack has been generated (90% I believe).

On the positive side, compared to the translate.wordpress.org solution, it has support for json translation files.

You can see the available packs at https://translate.wordpress.org/projects/wp-plugins/friends/language-packs/ or via the API at https://api.wordpress.org/translations/plugins/1.0/?slug=friends which result in URLs like https://downloads.wordpress.org/translation/plugin/friends/2.9.3/de_DE.zip

It's not possible to get "just the latest" language pack for a plugin since that could result in a 404 for languages that just don't have a language pack. On translate.wordpress.org every plugin has a translation project so that URL gives you a MO file (even an empty one) for any language.

Thus, I think the extra request to the translations/plugins endpoint is needed and warranted.

[bgrgicak]

It's not possible to get "just the latest" language pack for a plugin

Is there something preventing us from adding support for fetching the latest version translations?

since that could result in a 404 for languages that just don't have a language pack

A 404 would be ok for Playground.

[brandonpayton]

Language selection belongs outside the "steps" list in the Blueprints schema Directory mounts belong outside the "steps" list in the Blueprints schema

I agree, but steps are the only thing we have today. We discussed in the past that there could be a way to separate boot actions from runtime actions in the blueprint. These look like good examples of it.

I understand. ☺️ I just wanted to take the opportunity to bring up the separation again.

By separating them, we could also run only the runtime part of the blueprint on load and skip the boot part if Playground was already booted (for example reload when using browser storage).

Ah, that's an interesting point! The fact that some can be skipped after initial runtime setup is another hint that they are different kinds of things.

[bgrgicak]

I'd like to add my akirk/playground-step-library@main/steps/setLanguage.js implementation to the discussion here. This is quite imperfect as it should use language packs (see api.wordpress.org/translations/core/1.0 and api.wordpress.org/translations/plugins/1.0?slug=friends) so that we get all the JSON files as well.

@akirk I explored this approach, but it complicates things a lot if we use blueprint steps to get translations. If we do it with PHP it's a simple function call for core translations. With JS it's a zip download, extraction, cleanup...

When it comes to plugin and theme translations it's even more complicated because we need plugin version data and this adds extra requests. I'm now trying to find a way to trigger the download of plugin/theme translations, but for some reason can't find how to do it. 😕

[akirk]

There is a function wp_download_language_pack() that should do the trick.

[bgrgicak]

There is a function wp_download_language_pack() that should do the trick.

That's what I'm using, but it looks like it downloads only core translations.

[bgrgicak]

@adamziel @akirk this should now be ready for review, sorry for the delay.

[akirk]

For reference, I cooked up this little piece of PHP code to install a language pack:

function install_language_pack( $type, $slug, $lang ) {
	if ( ! in_array( $type, [ 'plugin', 'theme' ] ) ) {
		return new WP_Error( 'invalid_type', __( 'Invalid type' ) );
	}
	require_once ABSPATH . 'wp-admin/includes/translation-install.php';
	require_once ABSPATH . 'wp-admin/includes/class-wp-upgrader.php';
	$language_updates = translations_api( $type . 's', compact( 'slug' ) );
	if ( is_wp_error( $language_updates ) ) {
		return $language_updates;
	}
	$packages = array_filter( $language_updates['translations'], fn( $pkg ) => $pkg['language'] === $lang );
	if ( empty( $packages ) ) {
		return;
	}
	$upgrader = new Language_Pack_Upgrader( new Automatic_Upgrader_Skin() );
	$update = (object) $packages[0];
	$update->type = $type;
	$update->slug = $slug;
	return $upgrader->bulk_upgrade( array( $update ) );
}

For example:

install_language_pack( 'plugin', 'friends', 'de_DE' );
=> array(1) {
  [0]=>
  array(7) {
    ["source"]=>
    string(70) "/Users/alex/Sites/localhost/wp/wp-content/upgrade/friends-2.9.3-de_de/"
    ["source_files"]=>
    array(7) {
      [0]=>
      string(22) "friends-de_DE.l10n.php"
      [1]=>
      string(51) "friends-de_DE-7ffd8fc6ef503cd5a5c483bb3aca9bc5.json"
      [2]=>
      string(51) "friends-de_DE-93774191f1ed3f0224434004c216505a.json"
      [3]=>
      string(16) "friends-de_DE.mo"
      [4]=>
      string(51) "friends-de_DE-9a6fba47b355b62e8f6712f5c40b6192.json"
      [5]=>
      string(51) "friends-de_DE-85be8386efdca7f7b168c9c67defc614.json"
      [6]=>
      string(16) "friends-de_DE.po"
    }
    ["destination"]=>
    string(59) "/Users/alex/Sites/localhost/wp/wp-content/languages/plugins"
    ["destination_name"]=>
    string(0) ""
    ["local_destination"]=>
    string(59) "/Users/alex/Sites/localhost/wp/wp-content/languages/plugins"
    ["remote_destination"]=>
    string(60) "/Users/alex/Sites/localhost/wp/wp-content/languages/plugins/"
    ["clear_destination"]=>
    bool(true)
  }
}

[bgrgicak]

For reference, I cooked up this little piece of PHP code to install a language pack:

Thanks @akirk! Let's keep the JS implementation for now, but this will be useful when we get to implementing PHP blueprints.

[adamziel]

LGTM, I just have one note. When I use an invalid language code, like pl instead of pl_PL, I expected it to fail. However, instead of failing, it just booted an english site. Let's make sure to fail in these cases:

http://localhost:5400/website-server/?plugin=friends&theme=twentytwentythree&language=pl

[adamziel]

I added a usage example to the PR description and updated this Blueprint to use the new step:

https://github.com/WordPress/blueprints/blob/trunk/blueprints/translations/blueprint.json