Friday Facts #436 - Lost in Translation

[kfsone]

Oh man, we added bidi text to our engine a couple years ago and it was such a pain, Mac/Win/Lin: VS Code, Notepad++, Textedit, and Sublime would tie themselves in knots when you started trying to form bidi strings, and trying to make mouse selections?

Since we don't have mods, I get the luxury of creating a per-language glyphmap (atlas) during asset generation, and then I lean into Python libraries to shape arabic fonts and fixup bidirectional text embeds:

Code: Select all

# For bidirectional text operations in font conversion.
python-bidi>=0.4.2,==0.4.*
arabic_reshaper>=3.0,==3.0.*

I've been looking at trying to get out of the mess that is Python 3's ecosystem, Zig looked promising but Rust kinda blew me away with how easy it was to roll out a pythonized module (`from myrust import reshape`) or a limited-use embedded python interpreter (see https://github.com/kfsone/rumao3/)

[kfsone]

Oh man, we added bidi text to our engine a couple years ago and it was such a pain, Mac/Win/Lin: VS Code, Notepad++, Textedit, and Sublime would tie themselves in knots when you started trying to form bidi strings, and trying to make mouse selections?

Since we don't have mods, I get the luxury of creating a per-language glyphmap (atlas) during asset generation, and then I lean into Python libraries to shape arabic fonts and fixup bidirectional text embeds:

Code: Select all

# For bidirectional text operations in font conversion.
python-bidi>=0.4.2,==0.4.*
arabic_reshaper>=3.0,==3.0.*

I've been looking at trying to get out of the mess that is Python 3's ecosystem, Zig looked promising but Rust kinda blew me away with how easy it was to roll out a pythonized module (`from myrust import reshape`) or a limited-use embedded python interpreter (see https://github.com/kfsone/rumao3/)

[EustaceCS]

In the Belarusian localisation the letter "ў" is always displayed in the upper case for some reason. It also looks as if it's bolded as you can see on the screenshot

Yes, the relevant bug report is here viewtopic.php?f=7&t=119359 .

[kfsone]

Wouldn't unicode normalization solve this problem (Thinking of NFKD)? This might need a link to libicu, but should solve almost all these problems.

I'm guessing that's what Hrusa referred to here :

Out-of-the-box solutions are usually bloated to accommodate edge cases for many scripts and languages we can't even render in Factorio (Hieroglyphs, Sumerian Cuneiform, Emoji...).

The problem with that is that Factorio users in various locales will need to report and get their problems fixed, which have likely already been fixed over a much longer time period and much larger user base with an off-the-shelf library like ICU. This is all the more valuable for localisation since users have different experiences there, so bugs are only encountered by a fraction of the user base. I'd easily go with some extra bloat for that benefit (and it's not like the game is tiny so it'd grow a lot percentage-wise).

To paraphrase Douglas Adams: Unicode is big. You just won't believe how vastly, hugely, mind-bogglingly big it is.

Loading a Unicode font into memory is an acceptable pain point for a lot of software, but rarely for a game engine. Rendering unicode fonts is fine when you're a text editor, but you can easily drop 120fps to 15fps by having a dozen player names on screen that way.

Typically a game engine wants to turn strings into textures so it can throw them at the gpu and be done with, rather than doing it per frame. Whenever possible, we tend to want to build a sort of atlas of pre-rendered characters (aka a glyphmap) so that rendering those strings isn't also a bottleneck. One of the bugs I fixed while I was on the WoW server team was an issue where the timing of the way a player zoned, usually to an auction house or somewhere equally busy, meant the client and server got out of synch as to what was in the client's glyphcache for names so for a while you'd see empty nameplates over players until they got cycled out of your glyphcache.

As a game dev, unless you're doing something that doesn't have a lot of gpu pressure like age of empires or civ or something, unicode is not your friend from a rendering perspective.

[mlofinti]

As a game dev, unless you're doing something that doesn't have a lot of gpu pressure like age of empires or civ or something, unicode is not your friend from a rendering perspective.

Factorio already uses UTF-8 for strings and their own glyph cache for rendering (at least as of viewtopic.php?f=58&t=105248). Processing strings for searching with a library like libicu wouldn't touch rendering or the gpu at all, the question is only whether it's too heavy on it's own.

[lburton]

Not directly related to translations, but to search: I often find myself searching using colors or other colloquial terms. One example is looking for "red chips" or "red electronic circuits" instead of advanced circuits, or "blue inserter" instead of fast inserter. Thank you for having such a high level of quality for the game!

I agree having various additional aliases for items / recipes like the also mentioned iron stick vs iron rod would be lovely.

Also in the not-related to translations (more feature requests..) land would be the ability to search for requestors of an item in the logistics network.

[Sameish]

You mentioned in the FFF to note down any suggestions for search in general. The biggest thing I'd like to see added to map search is for it to return placed entities in the word, not just the machines building them or the logistics chests storing them.

The main use case I have is:
My power is running a bit low, I look at the power stats and see I have a LOT of radars, and I want to quickly see all the placed radars on the map to see if I have loads close together which aren't needed. I search 'radar' and all that comes up is the assembly machine that is producing them.

[bobucles]

As a game dev, unless you're doing something that doesn't have a lot of gpu pressure like age of empires or civ or something, unicode is not your friend from a rendering perspective.

Factorio already uses UTF-8 for strings and their own glyph cache for rendering (at least as of viewtopic.php?f=58&t=105248). Processing strings for searching with a library like libicu wouldn't touch rendering or the gpu at all, the question is only whether it's too heavy on it's own.

If it's all string comparisons like that, wouldn't it just be a plain old lookup table? Dunno how streamlined it would really have to get, it's not like people are running remote code execution macros through a search bar.

[EustaceCS]

This is all the more valuable for localisation since users have different experiences there, so bugs are only encountered by a fraction of the user base.

Bugs related to specific whims of specific language are encountered by 100% of that specific language pack users by definition.

The problem with that is that Factorio users in various locales will need to report and get their problems fixed, which have likely already been fixed over a much longer time period and much larger user base with an off-the-shelf library like ICU.

It's... much more complicated than that, to say the very least.
Decent starter on Unicode problem is here: https://tonsky.me/blog/unicode/
Libraries prepared by someone else are nice - but these libraries will contain already-made choices for some very important strategic decisions.
Choices, consequences of which - given how tightly everything is assembled in Factorio - can hit in really unexpected places at least appropriate time.
One of cornerstone choices to make being: which blend of Unicode to accept?
And is it really worth the effort to implement whole Unicode offered cheese (which - see in article above - is expected to cover all currently known languages and then some) for a game which have very finite list of localization packs AND very finite specific theme centric vocabulary used?
End users (read: players) probably shouldn't be bothered with such details.
But end users probably also shouldn't insist on specific solutions either.
And what's the benefit from implementing everything from maximally inclusive standard? Option to translate the game into klingon... klingonian? klingonese? how is this language even called?

[suzushiro]

As a Japanese player, I would like to thank you for releasing Space Age in a wonderful localization. As a non-English speaking gamer, I don't need to explain the joy of playing in my native language.

Games with special Japanese language support are really rare (even those made in Japan), so this announcement is surprising: with support for two kana, it is great to be able to concentrate on the search window and just type B-E to start searching for a belt. So what about Copper ore, D-O-U-K-O-U-S-E-K-I, and then the Space key... a list of Kanji combinations appears, my Google IME listed 13 possible candidates. To find copper ore, you must first find the correct Kanji combination in the IME list (it is actually at the top of the list because it is a major word, but not necessarily at the top for all words).
This is the troublesome part of Japanese language support that several people have already mentioned.

The problem continues. When I typed in “石”(stone) to look for a stone patch because I ran out of stones, factorio displayed a list that also included patches for iron ore, copper ore, coal, and uranium ore. This is because there is no difference between “ore” and “stone” in the Japanese localization. When it comes to coal (石炭), the first character contains Stone.
I tried to search only for “石” using double quotation marks, but there is no such function, and I am at a loss.

After all, I feel it would be better if we could search in English regardless of the language setting of the game. even a child can remember how to spell STONE.
Besides, it's a familiar method in Minecraft with factory mods (GregTech, etc.) installed.

Even with full Japanese support, some players will still play in English because Factorio is not Skyrim, there is no dialogue or flavor text, and they are willing to suffer a little inconvenience in order to be able to search for “stone” in “stone”.

[bmmtstb]

Love the "Hello" at the beginning of the post switching through different languages. "Grüßli Müsli" for the German one is hilarious

I was wondering whether I was the only one noticing that. Who put in "Grüßli Müsli" that has to be a native.


---

For the other stuff:

+1 for multi language search, e.g. German as main language, and english always - searching for "belt" should work, because its way easier than most translations.

As someone said, in German "ä" is officially "ae", but on my laptop I use a qwery keyboard, and I mostly type "a", do both versions map to the same word? So the "Grüßli" from above could also be a "Grusli", a "Gruessli" or something similar...

multiple people mentioned that there are other words for some items e.g. "iron rod" vs "iron stick", especially in other languages there will be more words, is it possible to have a search that links "rod" and "stick" to the same item? And even for "Stange" and "Stab" (the German equivalents) to match one item? I'm not good with names, and in most cases I just roughly search something and maybe I don't use the right word...

But thanks for all the work!

[USN484259]

Appreciate your efforts. I think searching in Chinese could be further improved.

Some Chinese users tend to turn off IME during gaming to avoid possible input troubles, and want to use Chinese pinyin (something like Chinese annotations in English letters) to search in the game. However, it's not trivial to map pinyin to Chinese words, it's equivalent to embedding the whole Chinese IME engine to the game. Below are possible solutions I came up with.

1. Annotate every word in the game with corresponding pinyin so searching can match against the annotation. However this needs some extra translation work, and not so friendly to mods.
2. Try to dynamically link to IME engine, e.g. dlopen("libpinyin.so") on Linux/Ubuntu, but this is OS dependent, I don't know how to do on Windows and if this is possible with 3rd-party/custom IMEs.
3. Embed lightweight IME engine into the game, but this add unnecessary game sizes to users of other languages.
4. For people knowing another language, allow to search in that language or by game internal name, maybe add a dropbox in settings to configure this feature.

Localization is not easy, but is vital to user experiences. Thanks you for the nice translations.

[éclair]

Thank you for the support in Japanese.
I know it's a lot of work, but I would be grateful if you could make it possible to search in English as well as Japanese.
(To search in English when kana conversion is too much of a hassle.)

日本語の対応有難う御座います。
大変だと思いますが、日本語で検索出来るのと共に英語でも検索出来る様にして頂ければ幸いです。
（かな変換が面倒臭い時に英語でも検索するため）

[tiktalik]

It would be nice if when you search, all the items show up on the same page. Having to look between multiple pages when you're using a text input search defeats the purpose a bit.

[tinker9]

Out-of-the-box solutions are usually bloated to accommodate edge cases for many scripts and languages we can't even render in Factorio (Hieroglyphs, Sumerian Cuneiform, Emoji...).

You can build ICU with just the locales (languages) you support:
https://unicode-org.github.io/icu/userg ... dtool.html

[Colin969]

The search system is great, but it definitely feels a bit lacking still. The lack of keyboard navigation being the worst offender, it's almost pointless to search in recipe selection when you're forced to take your hands off and click on it with the mouse. I constantly get frustrated by the recipe being deselected half the time too when changing rarities.

I'd also kill for support for item name aliases even if only through the modding api. (Red chips, rods, cogs, yellow science packs)

It's weird having to put very conscious thought into remembering what the game actually names things for the search system because it contradicts with common community names or instinctive names, it's often why I avoid using it. It's like the opposite of autocomplete lol.

[BlueTemplar]

It would be nice if when you search, all the items show up on the same page. Having to look between multiple pages when you're using a text input search defeats the purpose a bit.

Not when you consider that search is only there to guide you until you remember the actual location of the item ?
Which you won't (for the tab that it's in) if just gives it to you.
(It already automatically switches you to one of the relevant tabs when you do, somewhat hindering the learning process, but a good balance I guess.)

[nuhll]

WTF!!!!!!!! a working search? Only 10 years after release?

but i wönder if it älsö wörks with germän letters

[mmmPI]

I think the research is very important in the game, it's good to read it would work better for everyone.

I think it's a good news that FFF are still written even if this one is not really thrilling for its content if you only use english ^^ it shows the game is still being improved even after release

[ladavarga]

Přílíš žluťoučký kůň pěl dábelské ódy