[0.12.1] Right-To-Left languages are presented Left-To-Right

[sillyfly]

Do you know if Allegro supports Unicode control characters? I really don't know how widespread the support for these is.

Anyway, if the chosen method of support will indeed be deploy scripts, they can always use fribidi which does a good job even without such characters: http://fribidi.org/ (LGPL)

[jockeril]

Do you know if Allegro supports Unicode control characters? I really don't know how widespread the support for these is.

Anyway, if the chosen method of support will indeed be deploy scripts, they can always use fribidi which does a good job even without such characters: http://fribidi.org/ (LGPL)

I don't know what "allegro" or "fribidi" are ( I will check the link you provided, though). I just inverted the text in the .cfg files that are in the locale folders of the "core" & "base" folders in the game and that seams to do the job - for now.

[sillyfly]

Allegro is the gaming/graphics engine that Factorio uses. If it doesn't support Unicode direction markers then MF-'s suggestion probably won't work.

[jockeril]

If I understand you correctly, then it's the other way around - the text in the locale files is visually ok - but the game re-arranges it logically and that messes up the visual...

I thought factorio simply doesn't support RTL / BiDi.
So a "logical" RTL format will appear allright in the text editor, but will be presented "in byte order" in-game.
note: You know that technically there is a "RTL" unicode symbol, followed by character is reading order, right?

You workaround currently seems to be converting to "visual" ordering, where byte order is "left to right"
so there is no RTL marker and text might appear (but edit weirdly) right both in the text editor and in game.

Obviously a step forward would be using "logical" for the editing step, but then convert it to "visual" for the game.
That way the odd "visual" ordering will be limited to the internals of the game.
And I belive the factorio deploy scripts should be able to take care of that conversion.

I appreciate your explanation but all it's done, since I'm not a programmer, is confuse me about the difference between "logical" and "visual" representation of the text, and again - using scripts is not something I can do.

I hope the #dev team will sometime soon allocate the resources needed to make the changes so I don't have to continue translating & inverting the locale files text for the game.

FYI (1) - it turned out to not just be a matter of inverting text but also adjusting some of the original translation and the combination of hebrew & English in a sentence to appear correctly on screen. This is the reason it's not finished yet.

FYI (2) - Trying to use GitHub to publicly deliver the mod to the masses , I'm having problems uploading my local files to the repository... still learning how to work with that

[MF-]

Allegro is the gaming/graphics engine that Factorio uses. If it doesn't support Unicode direction markers then MF-'s suggestion probably won't work.

I thought the whole point of "logical ordering" is not having to support unicode direction markers
Note: That's abbreviated in the former message as "RTL/BiDi".

[MF-]

I appreciate your explanation but all it's done, since I'm not a programmer, is confuse me about the difference between "logical" and "visual" representation of the text, and again - using scripts is not something I can do.

I hope the #dev team will sometime soon allocate the resources needed to make the changes so I don't have to continue translating & inverting the locale files text for the game.

FYI (1) - it turned out to not just be a matter of inverting text but also adjusting some of the original translation and the combination of hebrew & English in a sentence to appear correctly on screen. This is the reason it's not finished yet.

FYI (2) - Trying to use GitHub to publicly deliver the mod to the masses , I'm having problems uploading my local files to the repository... still learning how to work with that

Ooops, I guess I assumed every right-to-left "speaking" person who ran into RTL problems will know.
In retrostpect, how silly of me.
I wrote this wall of text to explain, so I won't be offended if you read a wikipedia article instead.


Let me explain the unicode ordering methods.
You probably know, that a file in a computer is a series of bytes, usually read from the beginning to the end.
The same goes for text strings, they are represented by a series of bytes.

In the early days it would be too costy to support both text directions at once,
Those are the days of ASCII (and its upper-half extensions), where one byte equaled one character
and those were printed left-to-right onto the screen as they were read from the file/source.
All/most of? text drawing code did that. But then someone needed to diplay something Right-to-Left.
But the code was drawing characters from left-to-right as they arrived on the input.
Heck, there were some "electronic typerwriters", that wouldn't even technically be able to print in the other direction.
The obvious workaround to that problem is what you're doing now. That became known as "visual ordering".
In that the first data byte of a text string is the last character of the right-to-left text.
When the program then draws the string, then it reads this first data byte and puts it at the leftmost position.
That worked. Had it's flaws - the text printed this way would be obviously left-aligned and look odd I guess.

The point of "visual ordering" thus is, that it requires no program change to display seemingly R-t-L text.
You just "change the font" and invert the order of characters in the string so the code encounters them in right L-t-R order.


Even thought it looked (mostly) right, it sure had a lot of limitations.
For example text-processing tools (seeing the data bytes) had now no clue where the text should be appended.
If you would be to write a long text file in R-t-L (assuming no newlines), the file to store it would kinda have to be written from the end to the beginning to create a logical ordering. And implementing "end-to-start" filesystems proved to be too complicated or too awkward I guess. Even more awkward for mixed text OFC.

Then, the "logical" ordering was conceived.
In logical ordering the first data byte of a text string is the special LTR character, which merely notes,
that the following data bytes should actually be printed from Right-to-Left.
If the program drawing such text doesn't support that, the text will appear to be drawn in reverse order.
However, it does solve the storage issue very nicely. Now both LTR and RTL text files can be amended by simply appending bytes.

I'm suspect that talented programmers did something to display text "right" in your region in the early days, but no idea.

Glossary:
"visual ordering"
- text data bytes contain the leftmost="last" RTL text character first.
- Compatible with old code but awkward to edit or text-process.
"logical ordering
- text data bytes start with "RTL" character, then follows the first-char-to-be-read-by-human=first RTL text character.
- This needs the code to understand RTL and be able to draw in RTL direction. Editing works nicely. Preferred.
"RTL" - Right-to-Left unicode marker - data bytes read after this one are printed from the right
"LTR" - Left-to-Right unicode marker - data bytes read after this one are printed from the left.



Does my former message make sense now?
What you do now: in-cfg-file: "visual", in-game-data: "visual"
Fix that I propose: in-cfg-file: "logical", in-game-data: "visual" (automatic conversion while adding translations to game)
Once the game gains support for drawing the proper "logical" format by itself, the conversion will simply be removed.

[jockeril]

@MF- thank you for that - that's exactly the explanation I needed to understand the difference. I'm old enough to remember the original word processors in DOS days and the problems we had with integrated Hebrew-English mixed text... those where the days that brought the Hi-Tech revolution to Israel and maybe the reason companies like Intel & Microsoft decided to invest in Israel...

This explanation gave me an idea - I'll try to add the RTL Unicode char. to a line I see at the start menu ! and see how it goes. I will update on this later today.

now I have a new problem - it seams that not all the text to convert is present in the core & base files ! there are two (as far as I can find) lines that where not converted by me because of this. here is the screenshot:

Graphics options not found in the locale files (marked in red)

graphics options hebrew still inverted.JPG (76.09 KiB) Viewed 7907 times

The lines marked in red are the ones I'm referring to (one of which is a sub-menu with four options). when I did a Windows text search - I couldn't find them ! this is stranger then strange and I need the #dev team to give me that sub-section of the translation matrix to convert them so they also will appear visually correct on screen (even if the RTL char works - I need them)

[MF-]

remark: implementing a Hebrew text search could also be much harder because of this.
The inner disk-search algorithm will likely be set to look for a sequence of bytes,
but if you have both "logical" and "visual" - formatted files, finding one byte pattern simply won't work.
All becomes much worse if a file contains both formats mixed (well, because why not? can't tell by looking, really).
Another good reason for the wolrd to unite in using only the most-practical Logical format.

PS: the RTL character will likely be added automatically by any text editor you use.

[Dev-iL]

So I wrote a very basic python script that inverts all strings after "=" in a core.cfg file located in the same folder as the script:

Code: Select all

#!/usr/bin/python
# coding=utf-8

import codecs


def main():
    f_out = codecs.open('core_out.cfg', 'a+', encoding='utf8')
    with codecs.open('core.cfg', 'r', encoding='utf8') as f:
        for line in f:
            if line[0] in ["[", "\n"]:
                f_out.write(line)
                continue

            split_ln = line.split('=')
            eng_part = split_ln[0]
            heb_part = split_ln[1].strip()  # removing the newline at the end 
            tokens = list(reversed(heb_part.split(' ')))
            for ind, token in enumerate(tokens, 0):
                if not token:  # skip empty lines
                    continue
                if not all(ord(c) < 128 for c in token):  # (supposed to) skip non-heb tokens 
                    token = token[::-1]
                tokens[ind] = token
            f_out.write(eng_part + u"=\u200F" + ' '.join(tokens) + '\n')  # the RTL character is added after the '='
    f_out.close()

if __name__ == "__main__":
    main()
    print "\n!!! Don't forget to fix the font names @ line ~222 !!!"

Then I placed it in the mods folder as needed and got the following result:


For some reason "Multisampling" got inverted too .

In any case, I'm attaching my mod-attempt to this post.

[jockeril]

So I wrote a very basic python script that inverts all strings after "=" in a core.cfg file located in the same folder as the script:

Code: Select all

#!/usr/bin/python
# coding=utf-8

import codecs


def main():
    f_out = codecs.open('core_out.cfg', 'a+', encoding='utf8')
    with codecs.open('core.cfg', 'r', encoding='utf8') as f:
        for line in f:
            if line[0] in ["[", "\n"]:
                f_out.write(line)
                continue

            split_ln = line.split('=')
            eng_part = split_ln[0]
            heb_part = split_ln[1].strip()  # removing the newline at the end 
            tokens = list(reversed(heb_part.split(' ')))
            for ind, token in enumerate(tokens, 0):
                if not token:  # skip empty lines
                    continue
                if not all(ord(c) < 128 for c in token):  # (supposed to) skip non-heb tokens 
                    token = token[::-1]
                tokens[ind] = token
            f_out.write(eng_part + u"=\u200F" + ' '.join(tokens) + '\n')  # the RTL character is added after the '='
    f_out.close()

if __name__ == "__main__":
    main()
    print "\n!!! Don't forget to fix the font names @ line ~222 !!!"

Then I placed it in the mods folder as needed and got the following result:


For some reason "Multisampling" got inverted too .

In any case, I'm attaching my mod-attempt to this post.

And in that, you've just solved the whole RTL issue ! thank you very much

two questions:
1. what's with the text about line 222 font names ?
2. can you expand this to all the locale files in the game ?

also, I've noticed a problem with my "mod" that makes the game crash (and I reported it) for some unknown reason. I'll have the devs look at it anyway to insure that it's not a problem to other mods

[Dev-iL]

And in that, you've just solved the whole RTL issue ! thank you very much

two questions:
1. what's with the text about line 222 font names ?
2. can you expand this to all the locale files in the game ?

also, I've noticed a problem with my "mod" that makes the game crash (and I reported it) for some unknown reason. I'll have the devs look at it anyway to insure that it's not a problem to other mods

Wouldn't say that I solved it quite yet, since it still looks horrible and let's not forget it's just a temporary fix...
1. The following section also gets inverted (due to a bug), which makes the game not able to recognize the fonts (and there's an error on load).

Code: Select all

[font]
default=locale/he/Arimo.ttf
default-semibold=locale/he/Arimo.ttf
default-bold=locale/he/Arimo-Bold.ttf

2. Edit: see my post below.

[sillyfly]

Why re-invent the wheel?
https://pypi.python.org/pypi/python-bidi

[Dev-iL]

Why re-invent the wheel?
https://pypi.python.org/pypi/python-bidi

Thanks for the link! To answer your question - because of 2 main reasons:
1) I'm learning python so it's beneficial for me to do it myself.
2) I'm far from familiar with the multitude of available python libraries.

Having said that, here's a new version of my script and my mod (this time I used python 3.4.1 just because ):

Code: Select all

#!/usr/bin/python
# coding=utf-8

import codecs
import os
import glob
import sys

OUTPUT_FOLDER_NAME = "output"
DEFAULT_FILE_EXT = "cfg"


def main(input_file_ext=DEFAULT_FILE_EXT):
    # Initialize output folder:
    if not os.path.exists(OUTPUT_FOLDER_NAME):
        os.makedirs(OUTPUT_FOLDER_NAME)
    else:
        clear_folder(OUTPUT_FOLDER_NAME)

    # Obtain list of all files that need processing:
    for filename in glob.glob('*.' + input_file_ext):
        f_in = codecs.open(filename, 'r', encoding='utf8')
        f_out = codecs.open(os.path.join(OUTPUT_FOLDER_NAME, ''.join([filename[:-3], DEFAULT_FILE_EXT])),
                            'a+', encoding='utf8')
        for line in f_in:
            # Skip empty lines or lines where sections begin:
            if line[0] in ["[", "\n", "\r"]:
                f_out.write(line)
                continue

            split_ln = line.split('=')
            eng_part = split_ln[0]
            heb_part = split_ln[1].strip()  # removing the newline at the end
            tokens = list(reversed(heb_part.split(' ')))
            for ind, token in enumerate(tokens, 0):
                if not token:  # skip empty lines
                    continue
                if not all(ord(c) < 128 for c in token):  # (supposed to) skip non-heb tokens
                    # rotate parentheses
                    if token[0] is '(':
                        token = ')' + token[1:]
                    if token[-1] is ')':
                        token = token[:-1] + '('
                    token = token[::-1]
                tokens[ind] = token
            if all(ord(c) < 128 for c in line):
                f_out.write(line)  # Supposed to fix the inverted fonts issue
            else:
                f_out.write(eng_part + u"=\u200F" + ' '.join(tokens) + '\r\n')
                # Note the RTL character, \u200F, that is added after the '='
        # Close files
        f_in.close()
        f_out.close()

def clear_folder(path):
    for root, dirs, files in os.walk(path, topdown=False):
        for name in files:
            os.remove(os.path.join(root, name))
        for name in dirs:
            os.rmdir(os.path.join(root, name))

if __name__ == "__main__":
    if len(sys.argv) == 2 and len(sys.argv[1]) == 3:  # The 0th input is the script name, so we ignore it
        main(input_file_ext=sys.argv[1])
    else:
        main()  # read .cfg files by default

This time it operates on all files of a certain extension (e.g. ".cfg" or ".ini"). To use just put the script in a folder with some text files and run (assuming your default python is v3.x):

Code: Select all

python nameOfScript.py "ini"
-- OR --
python nameOfScript.py   <= by default assumes "cfg"

Known bugs \ incompatibilities with the game:
- Parentheses in the middle of words are not inverted (e.g. האם את)ה) בטוח)ה) שברצונך )
- Multi-line strings appear in the reverse order, that is the last line appears first etc.
- Strings displayed in orange popups (e.g. msg-introduction) appear reversed (no idea if they were ok before inverting the strings).
- Whenever a "/" is part of the string (e.g. description:durability in core.ini), the RTL character seems to confuse the game and makes it think that it's "\" or "//" which ultimately makes the string display wrong.
- There's a strange arrow-like character on the left side of all strings (I guess this is how the game renders the RTL character)

Preview:

[jockeril]

so even after running the script, there are problems, strange characters, and the need to put the converted file in a mods folder with a .json file like a mod to be recognized by the game as strings to replace the ones that where put by the devs...

and after all that, the test mod I made still doesn't work - it throws an error and crashes the game ( I think that happens when a research is finished) but i'm not sure

see this link

[jockeril]

edit: the crash is unrelated to this thread, as mentioned above.

@dev-il - Trying your mod on my laptop crashes the game right after loading the mod. log files follow:

[MF-]

So it might be crashing only for you?
Did you try it on a fresh factorio copy of both factorio data and configuration?

EDIT: Oh, you already did that in the Laptop thread.

[jockeril]

So it might be crashing only for you?
Did you try it on a fresh factorio copy of both factorio data and configuration?

EDIT: Oh, you already did that in the Laptop thread.

did you try just putting the mod in "mods" directory without enabling it ? what OS are you running ? and factorio - 32 or 64 bit ?

[Dev-iL]

I've uploaded my mod in the mods section of the forum. Here's a link to its thread.

[cube]

I added a functionality very simillar to Dev-iL's script to 0.12.1. It's definitely not perfect, and there will be issues with line wrapping strings, but it's better than nothing.

[jockeril]

I added a functionality very simillar to Dev-iL's script to 0.12.1. It's definitely not perfect, and there will be issues with line wrapping strings, but it's better than nothing.

Thank you very much ! you guys are the $#!7 !! I love you

I've uploaded my mod in the mods section of the forum. Here's a link to its thread.

and a special thanks to @dev-il for all his help in getting this going. I will now stop working on the translation mod and concentrate on helping @dev-il with the Hebrew localizing.

once we see a working R-T-L hebrew in the game we can archive this thread