A solution to searching prototypes in natural language

[Therenas]

There are many situations where you want to find items/recipes/etc. by their localised name, ie. the user entering the name of an item (in their own natural language and locale). As a mod, you can't currently search localised strings because of determinism. This can't be circumvented without a lot of effort, so it hasn't been done yet, for good reason. There are many threads in this subforum documenting these proposals and rejections.

Now I think there might be a way to add a way to make your items/recipes/etc searchable in a better way. The idea is to add an additional field to the prototypes, named search_strings or something, that contains the words the developer wants to let people find their item as. So, for an item with the name 'gear', and the localised name 'Iron gear', you could have search_strings like 'Iron', 'Gear', 'Eisen', 'Zahnrad' (German translation of those). So even if the internal name is different from the localised one, users would still find it if they search for 'Iron'.

Whether this is a csv or a lua table doesn't really matter. This list would need names in many different languages potentially, same as the locale does. But the mod author could just add them in English, and translation/locale mods could just append their language version of those strings to the prototype attribute, which would make it searchable in that language too.

Now how this is implemented exactly, and for which prototypes and such I don't know, but items/recipes/technologies seem like the prime contenders, maybe entities too.

[eradicator]

I'm all in for a translate() function, but i want a proper solution, and i want it to work for all locale strings. And as far as the old thread goes, this requires making locales deterministic in one way or the other. I.e. by synchronizing locales between players or by making locales part of the mod checksum in the first place.

What you propose has several issues:

1. You essentially introduce a second locale file that can be asynchronus with the actual locale file. Meaning double work for translators/modderes.
2. The serach keywords are an incomplete representation of the locale and as such won't do what the user expects them to do 90% of the time.
3. You're blindly mixing languages. For all i know "Iron" could be the Estonian word for "Bomb". Leading to completely wrong results for every user.
4. You can already implmenet this on your own via remote.call() so it doesn't improve the situation at all.

[Bilka]

I'm all in for a translate() function, but i want a proper solution, and i want it to work for all locale strings. And as far as the old thread goes, this requires making locales deterministic in one way or the other. I.e. by synchronizing locales between players or by making locales part of the mod checksum in the first place.

Something changed since that thread - all mods are now part of "game changing mods", so all players will have the same mods active and the right locale file on their system. My first reaction to that was: "great, locale is deterministic now". Except it's not: 1. Player locale is not game state - easily solvable, that's not a lot of data to send around. 2. You only ever load your own locale. Using a different one for Lua means also loading that. Now, loading like 30 different locales? That sounds insane. I still hope for translate(), but I want to explain why it's not a thing yet.

[eradicator]

Something changed since that thread - all mods are now part of "game changing mods", so all players will have the same mods active and the right locale file on their system.

As i suspected. So locale is also part of the mod checksum now?

Using a different one for Lua means also loading that. Now, loading like 30 different locales? That sounds insane.

That's the same reasoning as in the old thread. Let me try to hold something against it (again?) for arguments sake:

1. Is 30 locales really that big? A search over my factorio folder says there are 947 ".cfg" files with a total of 9MB. With RAM in the gigabytes this should be negligible.
2. If it's still too much anyway (table overhead or whatever) then why not load them dynamically? To keep the game deterministic it would be sufficient if only locales were loaded that *someone* is actually using. The players see only their own locale. And i suspect that multiplayer servers with players from more than 10 different countries are extremely rare, and most of the time it's less than 5.
3. There are benefits besides modding to this. It'd be possible to e.g. have a button that toggles between the players two/three favourite locales, making communication much easier when playing with friends in a different locale. (Ok, would probably require too much engine rewrite to actually happen...)

I still hope for translate(), but I want to explain why it's not a thing yet.

So do i. Just look at how often many of the posts in that other thread are mine. I also have a horrible voodoo hack on paper that could probably do this right now. But maybe a horrible hack is what this thing needs to "get rolling", after all they keep saying "stuff that mods demonstrate are more likely to be implemented" .

[Rseding91]

I could implement a

Code: Select all

player.translate(localised-string)

but it's not going to be instant and would only work when the player is connected to the game.

How it would work is:

• API call is made requesting that a translation for 'localised-string' by 'player' is done
• If the player is online the game sends an InputAction with the localised-string and translated result + a bool if it failed to translate
• Whenever the action gets to the game and is run it fires a "on_string_translated" event which contains the player it was translated by, the original localised string, and the result translation + if it failed to translate

In single player it would be instant or 1 tick later that results come back. In multiplayer it's N ticks later or never if the player disconnects before finishing the request. It's not perfect but I think for most cases it would work.

Is this something that would handle peoples needs?

[Optera]

I could implement a

Code: Select all

player.translate(localised-string)

but it's not going to be instant and would only work when the player is connected to the game.

How it would work is:

• API call is made requesting that a translation for 'localised-string' by 'player' is done
• If the player is online the game sends an InputAction with the localised-string and translated result + a bool if it failed to translate
• Whenever the action gets to the game and is run it fires a "on_string_translated" event which contains the player it was translated by, the original localised string, and the result translation + if it failed to translate

In single player it would be instant or 1 tick later that results come back. In multiplayer it's N ticks later or never if the player disconnects before finishing the request. It's not perfect but I think for most cases it would work.

Is this something that would handle peoples needs?

Just to be sure I understand this proposal.

Code: Select all

player.translate("Eisenplatte")

would raise

Code: Select all

on_string_translated(player, "iron-plate")

That would be a step towards searching by localized names. However that still requires players to search for exact matching strings.
Every other search in game supports fuzzy search. A search function should be able to return all item names where the localized string contains a certain string.
Searching for "Eisen" should return an array containing iron-ore, iron-plate, iron-chest, ...

[posila]

No, I think it's the other way around

Code: Select all

player.translate({ "iron-plate" })

would raise

Code: Select all

on_string_translated(player, "Eisenplatte")

[Rseding91]

No, I think it's the other way around

Code: Select all

player.translate({ "iron-plate" })

would raise

Code: Select all

on_string_translated(player, "Eisenplatte")

Correct.

[Optera]

That seems only useful to check if a certain translation exists.

Just going by the thread name the goal is to search for a prototype by translated name. Any mod trying to do a translated name> prototype name search would need to call this function for every prototype and player to build a translation dictionary.
Ideally that would then be used to create one function library shared between all mods wanting to use this feature, as if that would ever happen. We all know every larger mod created makes his own library because none of us trust the other to keep a library working with all mods depending on it.

[Rseding91]

That seems only useful to check if a certain translation exists.

Just going by the thread name the goal is to search for a prototype by translated name. Any mod trying to do a translated name> prototype name search would need to call this function for every prototype and player to build a translation dictionary.
Ideally that would then be used to create one function library shared between all mods wanting to use this feature, as if that would ever happen. We all know every larger mod created makes his own library because none of us trust the other to keep a library working with all mods depending on it.

Yes, that's what mods would have to do. Doing reverse translation lookups isn't something even the base game can do. Any time anything does searching/filtering it's "get all of the translations for every prototype i'm filtering and then match what the player typed against them".

[eradicator]

No, I think it's the other way around

Code: Select all

player.translate({ "iron-plate" })

would raise

Code: Select all

on_string_translated(player, "Eisenplatte")

I would very much prefer if it raised

Code: Select all

on_string_translated(player,{["iron-plate"]="Eisenplatte"})

I.e. if it a) allows multiple requests at once and b) is trivial to know what was translated (named key, or same numeric key as in request). Because if i have to build a dictionary one string at a time with a 1+ tick delay each it'd be really really slow.

Any time anything does searching/filtering it's "get all of the translations for every prototype i'm filtering and then match what the player typed against them".

And that is obviously what a mod would need to do to. For *every prototype*, all at once, preferably in on_player_joined_game. Or if that's too much traffic cut it down into a few packages.

As a slight addition it would be nice to have LuaPlayer.locale = 'string', i.e. "de" or "en", so i can skip the building of the dictionary if it already exists for the newly joined players language.

Apart from that it sounds good:)

[eradicator]

On second thought, is that really more efficient than just

Code: Select all

player.request_locale()

or even

Code: Select all

player.request_locale_filtered({types={'item-name','recipe-name','entity-name'})

that raises an event that contains the complete locale? Mods will have to build dictionaries anyway, so translating only a single keyword will be a rare edge case.

[Optera]

On second thought, is that really more efficient than just

Code: Select all

player.request_locale()

or even

Code: Select all

player.request_locale_filtered({types={'item-name','recipe-name','entity-name'})

that raises an event that contains the complete locale? Mods will have to build dictionaries anyway, so translating only a single keyword will be a rare edge case.

Good idea.
Call that function once in on_player_joined_game and receive a full dictionary like

Code: Select all

{["en"] = {["iron-plate] = "Iron plate", ...}

Delaying by 1 or more ticks would produce a window where not all players have a translation available which could cause desyncs.

[Pi-C]

The idea is to add an additional field to the prototypes, named search_strings or something, that contains the words the developer wants to let people find their item as. So, for an item with the name 'gear', and the localised name 'Iron gear', you could have search_strings like 'Iron', 'Gear', 'Eisen', 'Zahnrad' (German translation of those).

Bad example! What does "iron" mean, exactly? In English, the word "iron" can be used as noun (German: "Eisen") and as adjective (German: "eisern", or -- in the right context -- "ehern"). Would you expect people to search only for nouns? Would you include every alternative translation imaginable in the dictionary? Seems like opening a whole box of new problems to me.

[Rseding91]

A function which gives the full set of translations wouldn't be useful because translations aren't a 1-1 mapping of key<>value. They have substitutions/replacements and can even include each other.

The way I imagined it would work is you ask through a given player for some localised string to be translated.

Then later the event is fired and contains the player_index, localised_string, string, and boolean saying if it translated it or not. If you wanted multiple just call the event multiple times in what ever tick you want.

[eradicator]

A function which gives the full set of translations wouldn't be useful because translations aren't a 1-1 mapping of key<>value. They have substitutions/replacements and can even include each other.

I was just trying to be clear that for my and probably 90% of other mods the main usecase will be the creation of a dictionary of *all* item/tech/recipe/etc-names when the player joins. Then store the dictionary in global for later use. So a one-by-one interface might produce more overhead than needed.

But... if you want to go the extra steps and give us a full translation function that can translate any abitrary (concatenation of) localised strings i will be totally happy with that. :)

[Rseding91]

One at a time vs multiple at once won't have any significant difference in performance but it will make the backing logic easier if it's one at a time.

[eradicator]

One at a time vs multiple at once won't have any significant difference in performance but it will make the backing logic easier if it's one at a time.

Well then. Very much looking forward to this unexpected present. I'm guessing it'll happen after 0.17 stable?

[pleegwat]

Problems like this make me think of a different potential solution: Having a distinct UI stage which can read from and send commands to the control stage. Those commands would need to be synchronized across all players, but the UI itself could remain local to each player. Probably isn't feasible in this stage of factorio development though, and would break mods bigtime. Or not possible at all because I'm not seeing the entire picture.

[Rseding91]

Capture.PNG (14.36 KiB) Viewed 4753 times

Capture.PNG (7.87 KiB) Viewed 4753 times