Dialect localisation

[Anachrony]

TL;DR

Enhance locale system to support dialects which differ from full languages by having a parent language to fall back on if not overridden.

What ?

Instead of just having an en/ locale directory for English, have that, but also allow subdirectories for dialects of the language, such as American English, British English, Australian English, etc. These dialects will not have entries for most things, because they will match the generic English language. But they will have the option of overriding a few key words or phrases only where the dialects diverge. For languages that may lack full localisations, especially in mods, they can use a similar system to specify a language to fall back on that is likely to be at least somewhat mutually intelligible.

Why ?

Maintaining an entire new locale for every dialect of every supported language is probably untenable, especially for mods which often only support one or a few languages. By allowing a heirarchy of backup localisations, it's unnecessary to maintain that level of support. This may be even more useful for certain other languages like Latin American Spanish vs. European Spanish than for the example of English listed above. For English, the dialects are all mutually comprehensible, but are nevertheless a consistent source of annoyance in mods that make use of words that differ between them (e.g. aluminium/aluminum, color/colour). An example where it might be helpful for an incomplete language to fall back on another might be Norwegian, which the Crowdin page lists as incomplete. A lot of it is either the same as the Swedish localization or is more or less mutually intelligible, so it could fall back on Swedish as a default until complete. And even when complete in the base game it might still be a good option to fall back on for mods that only have one or the other.

[ssilk]

Even knowing how important it is to support dialects I’m not sure if this problem can be solved. Or should be solved. Because the effort is quite high, it will also probably take some significant more CPU-power and the gameplay value is unknown. (I tend to account an relatively low bp-value)

hope this is not too disappointing.

[Anachrony]

How could it possibly take any significant CPU power to process a few extra partially filled in text files for a selected language compared to all the vastly larger prototype files that already have to load? The locale strings would all be resolved once at startup time, not constantly being dynamically checked against multiple dialects at run time.

Implementing and maintaining dialects using the same system as how full languages are done would be a huge data entry effort that would never happen. Implementing them using the system I described would take as much or as little effort as the people on the crowdin site or modders want to put into it, because they would be fully usable even if a particular dialect had only a single page or less of overrides.

[ssilk]

Hm. It’s not exactly this easy.

Language is loaded at startup, yes, but nothing is translated then. Translation happens in real-time, but this is quite fast, it needs to look into a table and pick that string, instead of the original. If the entry doesn’t exist it uses the default.

Now you come along with dialects. Which is - as you describe - nothing else, than a primary (language) and a secondary (dialect) translation table. So the game looks, if the secondary table entry exists (the dialect), if not, it picks up the primary table, if also no entry there it picks the default.

So this possible extra table-lookup is of course far away from a CPU-killer. It would be hard to measure any differences, and then in the range of nanoseconds for thousands of translations. So please forget about this argument, it’s stupid because the secondary lookup can also be prevented by making it optional only, when a dialect is loaded.

[ickputzdirwech]

Interesting idea, never thought about dialects in games before.

The localisation already falls back to English if no string is available. I don’t think there would be any new file structure needed. A setting of some sorts to tell the game to which language it should fall back before it goes to English would be sufficient.

I however I don’t think this would facilitate the translation process that much. I think a better approach would be to just copy the current state of the main language into a new language folder for the dialect and start from there. Maybe someone could create a small script for github or crowdin to make automatic suggestions, whenever the main language localisation changes.

[Anachrony]

I however I don’t think this would facilitate the translation process that much. I think a better approach would be to just copy the current state of the main language into a new language folder for the dialect and start from there. Maybe someone could create a small script for github or crowdin to make automatic suggestions, whenever the main language localisation changes.

That would be a tremendously burdensome maintenance task to have to update multiple dialects and have to keep duplicate strings in sync if they're 90% the same anyway. It's always a good idea to factor away duplication for maintainability. If dialects are mostly the same as the main language, then people shouldn't have to diff through a bunch of copied over boilerplate to find the salient differences.

This may not seem like a big distinction for the main game, since it only has to be done once and it hasn't changed very much in quite some time (though it remains to be seen what kind of content the expansion might introduce). But for mods, having to duplicate and maintain extra dialects as they continue evolving would be unwanted and pointless effort in most cases. In practice, most mods wouldn't maintain all the dialects, so without the behavior of falling back to the appropriate major language then the dialects would be unusable with mods.

[ickputzdirwech]

That would be a tremendously burdensome maintenance task to have to update multiple dialects and have to keep duplicate strings in sync if they're 90% the same anyway. It's always a good idea to factor away duplication for maintainability. If dialects are mostly the same as the main language, then people shouldn't have to diff through a bunch of copied over boilerplate to find the salient differences.

Maybe I didn’t express it comprehensively but that’s pretty much my point. Regardless of how support for dialects is implemented, you have to go thought hundreds (maybe even thousands?) of strings in order to find the 10% or so where the dialect differs from the main language. Since translation for Factorio are done in crowdin this would have to be done in crowdin as well. This is the tedious process you can’t get rid of.

The question really is: how do you facilitate that process? Checking the strings in-game is not very practical. That’s why I suggested to start with a copy of the main language. That way you can go through the strings in crowdin, quickly approving the strings that are no different in dialect and main language, and changing those that are not.

[Anachrony]

But once you've done that work, the end result is an unmaintainable mess. The stripped down version is more maintainable. And you've only addressed the main game translation, and not the larger issue of mod localization. Mods that don't translate dialects would only work if my feature were implemented, otherwise nobody would want to select their dialect when playing modded games due to worse support. Unlike with full language translation, the failure mode is still completely understandable, merely imperfect. If a string is not easily discoverable in-game, that means it's one that isn't as prominent and thus less critical to the experience. If they find the process of going through every string to be the easiest way to go about it, they have the freedom to do that. But if they don't, they'd have the freedom not to translate every string. Having dialects work as I suggested wouldn't take away the option of doing it exhaustively, but it would raise the possibility of doing it opportunistically for strings of higher importance.

[ickputzdirwech]

But once you've done that work, the end result is an unmaintainable mess.

That's were some clever process outside of Factorio is necessary to notify the dialect translator every time the translation of the main language changes (for example through a suggestion or issue on crowdin). (Does anyone know what happens if an English localisation changes? Do the translations get automatically unapproved or something similar?)

You got a point about the mod support however. For them a fall back to the main language would definitely be helpful. So all in all a reasonable suggestion.

[Tertius]

From a programming point of view, adding support for dependent languages would be no additional permanent CPU requirement, if language support is already implemented as static dictionary with keyword -> text representation. Give every language a parent language. The "root" language is english.
At startup, the game determines the requested language. It looks up its parent language and the parent language of the parent language and so on, so you get a list of languages from actual to parent1 to parent2 to [...) to english.
Then load the languages in reverse order to fill the dictionary. Keywords overwrite previous keywords. First, the language dictionary is filled with english. It is overwritten by keywords of the first parent language, then by the next parent language, finally by the actual language. If the actual language is a dialect with only 20 keywords, all other keywords remain from the parent language(s). The dictionary is merged once from the language chain and is available for the whole remaining runtime. I assume it's already working this way, difference is only 2 fixed tiers with english (default) -> local language with nothing in between.