Factorio Forums

I would like to remind all the newbs that factorio is written down to the CPP (c++) classes as a way of packing the "logic".
As a side effect of such decision, we get the woefully lacking "native" support of unicode.
Simply because cpp as a language kinda predates the era where unicode took over the world.

std::string might just not be aware that characters could be multi-byte sequences;

and even if it was, declaring a full unicode support is kinda tricky in general (even emojies support is "fun" when you start to look into country flags, for example; look up TW flag history, if you want some IRL drama);

basically, in the cpp land there is no simple-and-natural way out with unicode. Might have changed with the latest cpp revisions, not sure; but they are probably kinda tricky to follow either way.

collation kinda means you can compare strings A and B to decide if they are "the same".
flip-the-bit option proposed above is an example of lowercase-it-all approach (uppercase it all, for the fans of Microsoft, haha).

Simple example of dates collation: m/d/y vs d.m.y; could be solved with a perfect "isodate" yyyy-mm-dd form.
Harder example of numbers collation: 1.00 and 1,00 is probably what looks the same for you. but 1_00 and 1,00 can also mean the same as 100; could only be solved with the help of the "context" of what 1,00 *probably* means by-default.

Same kind of stuff exists for languages.
In English you get kinda close to what "canonical form" of a grammar could be. Programming then dances around it even more, using a subset with a number of strict rules (such as with JSON) that make the "context-free parsing" part of it easy.
And that largely influences what ppl consider languages to be in the first place.

Yet, with, say, Russian, all the words are mutable. Some have to, some depend on your taste preference. You can say change the "copper-plate" depending on how or where you want to use it. ("plateys", "coppah" etc is kinda allowed, if you are willing to) You can even say "iron-gears" in a few different popular ways, still meaning "gears" for any native speaker (since the same "root", think "gearz"). Maybe not something one would consider for collation, but even the "plurals formula" is fun; declension; wiki is what you can look at for some more fun.

But CJK ("glyphy langs") can get you further than that.
There, you can write the stuff that sounds the same, means the same ...yet is written not the same way.
And there is more than one way to do that!
Not just "capital" letters but the "sets" of same-meaning glyphs.
And don't get me started on how they "mutate", compact or extend.
As such, imagine the collation that allows you to, say, have:
LDS -> Low density structure, or even GC -> (eh) Electronic-Circuit (yep, no Green in ANY form here, haha).

the ability to have the latter WOULD have been nice. Especially if user could control the mapping.
But I suppose that would have some fun effects on determination and overall system complexity, hehe.
Yet, for the CJK languages, having such an extensible vocab would be a killer feature in more ways than one.

And who knows, maybe there is even a goto-way to collate all that stuff already in the cpp world; but sorry, I do not speak cpp fluently ;>

Well, even Hebrew lets you "omit" the vowels as you go. German has ẞ/ß (for "ss", see eszett; wiki).
And most languages there are have some sort of irregular yet popular and widely used stuff.
And there is a whole another issue of motors-vs-engines like collation, often seen with players reading the non-english locale try to communicate in English.

so in short, making a perfect collation even for a known subset of world grammar is tough. Even for any "reasonable" limit on what is "the same".
thus having a take of "latin only, and the rest we do not care" is "valid" and "understandable".
Definitely not something of "call me a manager here!", I would say.

Most "duplicates" of the matter that I've seen so far refer to Cyrillic family.
Skipping the strong-and-complex lib-that-does-it-all option, it is kinda hard to grasp why it just "does not work" for `К==к` and `М==м` as it does for `K==k` with `M==m`.

In fact, looking at the unicode Cyrillic blocks (wiki) does give a suggestion that it would be as easy as adding a Titillium style font.
Well, the font was added only with v1.1.38, @@2021-08-18 (post-release), but now we have it :>

Nicely enough, unicode blocks are mostly linear, which in turn allows for these kinds of "hacks" that do the "flip-a-bit" approach with a range-check. Without any sizeable hash-maps, dicts or the like. (still, a few more checks)
Please look into it if you have the time to, as it introduces pretty much a crapton chunk of value for quite a reasonable buck. Even though not directly.

There are some not-so-trivial details as when the "lowecase bit" bleeds out to the first byte of the pair that represent the codepoint; but overall it is on the "easily manageable by one person" scale.

Naturally, both Python and JS have a native way around with `"Привет Мир!".lower()` or `"Привет Мир!".toLowerCase()`.
And Lua does not really. Because Lua is Lua. Nice and *simple*.

Having a game-engine implementation bindings to a "proper" lowercase in a form of `scripts.lowercase(str)` would be fun and neat, right.
Having it overload the string.lower might even be more fun (and *probably* won't cause too much problems, as far as I can imagine).

If we are talking static content (searching over pre-defined strings) one can probably even build-and-bake a radix tree right away, but well, to hell with it. Let's speak "real scale" effort where ppl do not even wrap the critical routines with pcalls. Really, what can possibly go wrong. Just mark it as incompatible if any problems arrive and that's it. Hehehe. (yet, less complex often also means easier to reason about! which is ok)

Then, say, "static" content allows us to have a "canonical" form of a string stored alongside for indexing purposes. These can be made with either Python or JS with quite a small scripting effort. As long as the canonical representation is uniform, it lifts up half the trouble (at the cost of memory usage). And works for even more complex languages where the "canonical" form could be manually tweaked to whichever is the norm.

But since we are in the land of naive contraptions,
here's a workaround that you can do for your own script on Cyrillic (with the naive hashmap-ing approach). A proper "lowercase" of sorts, baked with your own alphabet-like collation of choice.
All it really allows for is to have the case-insensitive search over the strings that have some Cyrillic inside in the Lua land. Dynamically.
Maybe not much, but, well, it does mean a lot for some.
Would prefer if it "worked" right away somehow, but this workaround is also an option; if you are aware of the issue, that is.