Accessing text in GUI elements, tooltips

[Amonwilde]

Hi all,

I'm creating a mod that will (hopefully) speak GUI elements aloud using TTS, it's designed for low vision or maybe dyslexic players. I've got the mod talking for chat messages, so that's a start. I'm having some difficulty accessing the text exposed in GUI elements such as tooltips, however. Can someone provide a Lua statement that will access the current tooltip object and/or the text in that object? On a higher level, how are the tooltips implemented, i.e. what GUI type are they?

In the event that it's not possible to access tooltips, is there a way to access the properties of entities currently under the mouse? I'm not seeing much by way of events for on hover, unfortunately. Thanks for any help with this, it's for a good cause

[Deadlock989]

Unfortunately it's not possible to directly access the tooltips themselves. They are not exposed to the API (not for want of asking).

However, many/most entity properties can be accessed at runtime via the entity's prototype. The mouse-over event you are looking for is on_selected_entity_changed. Things like machine energy usage, recipe ingredients and results, etc. etc. can all be got at through those, so you could possibly recreate most or all of the info exposed by entity tooltips that way. I don't think there is an event which only catches mouse-over of things like crafting menu buttons though, I may be wrong.

There is the added problem of localisation (names, descriptions, units, categories, etc. etc.) which are client-side - remember that people use Factorio in over 40 languages. That means tangling with the on_string_translated event, however, which is a bit of a nightmare. That one is difficult to use without killing UPS stone cold dead, other people have more experience with it than me if you want to handle multiple languages. If your intended users only speak one language, though, start digging in entity prototypes.

[eradicator]

I don't think there is an event which only catches mouse-over of things like crafting menu buttons though, I may be wrong.

There is no event at all for gui-hover. Especially not for vanilla guis.

That means tangling with the on_string_translated event, however, which is a bit of a nightmare. That one is difficult to use without killing UPS stone cold dead,

I've never heared of UPS issues with translate_string. My first naive implementation translated+cached all recipe/item/etc names and descriptions in the first tick after loading. Worked fine in singleplayer. The "tiny" problem with that approach was that in multiplayer that's equivalent to a DOS-attack and made joining any game impossible due to instant timeout. So you have to do tick-distributed translation etcpp. Not easy for sure.


@OP: Have you considered using OCR? Factorio uses only a a few different fonts, so hopefully that should make it sufficiently precise? Sure won't be as fast as just reading text files, but can at least react to what's actually on the screen. I guess icons would be a problem... :/

[Amonwilde]

Thanks for the quick response on this. It's disappointing that there is no event for hover. While it looks like I could implement getting info using the select event, that would only work on objects already in the world...not useless but only of limited utility without some other features first. Unfortunately I think this means the project can't move forward.

I'm low vision myself and have gotten to the fourth science (purple?) using OCR. But it took a lot of memorization and was a strain. The main issue is inventory management, though getting certain kinds of info from objects in the world can also be a headache. The text is fine for OCR, but the issue is intelligently extracting text. Running OCR on the whole screen gives a lot of extra junk, and though I have something for drawing boxes around specific text to read, it doesn't work well on the tooltips.