[jiri][0.16.12] Unintuitive search term parsing

[eradicator]

What?

When using the search function in the crafting or technology menu the parser seems to consider every letter of a typed word as a seperate search term, increasing the amount of false positive results.
Examples (pointed out by @DaveMcW in this thread):

The spellchecker is very aggressive, results for "Heat" give:
Logistics -> Arithmetic combinator
Production -> Chemical plant
Intermediate products -> Heavy oil cracking to light oil

Also " " (space) is inconsisently considered either part of the search term if not followed by anything else:
e.g "exchanger " (space at the end) -> no results
or used as a delimiter if between two words:
e.g. "exchanger heat" -> heat exchanger (one result)

Some funny examples of current behavior

Expected behavior:

Most search features i know that are target at "the average end user" use " " (space) as the sole delimiter and treat it as a logic AND. Everything not delimited by spaces is considered a single word and must therefore be present in each search result as a whole word.

[posila]

I am not going to sort this report out, I am going to throw my two cents here. The change was intentional - for consequtive characters if searches those characters in that order. It is quite powerfull, for example you can filter out everything but heat exchanger by just typing "hx" whearas "h x" would find also kovarex enrichment and explosive shells. I agree it is confusing if you don't know how it works.

[kovarex]

I also thought it was a bug until it was explained to me by the guys who implemented it, because this is something they are used to in their software.

I would say, that we should have an option for this kind of search that should be off by default.

[Aprillion]

now that I read multiple times about the explanation, I still think it's a bug - sounds like 99% of times non-dev users search something, there are too many false negative results without knowing enough about the feature to use it with benefits...

BTW which software works this way? if I imagine some tool (IntelliJ, VS Code, Excel, browser's find on page, spotlight, glob file name expressions, ...) changed their search behaviour into this, that would be the worst nightmare

what I usually expect is:
hx => match(/hx/i)
heat ex == ex heat => match(/heat/i) and match(/ex/i)
heat*ex => match(/heat.*?ex/i)
h*x => match(/h.*?x/i)

and sometimes advanced parsing is supported that looks intuitive to users but very tough to specify correctly:
3 copper => amount:3 and ingredients_or_products_match(/copper/)

but hx => match(/h.*?x/i) was very unexpected indeed

[eradicator]

I also thought it was a bug until it was explained to me by the guys who implemented it, because this is something they are used to in their software.

Factorio does not do any such explaining though. And personally i never understood why there were all those "odd" search results until i read the post by @DaveMcW because i've never encountered this particular search term parsing pattern.

It is quite powerfull, for example you can filter out everything but heat exchanger by just typing "hx" whearas "h x" would find also kovarex enrichment and explosive shells. I agree it is confusing if you don't know how it works.

Even though i bet there are some situations (large code bases maybe?) in which this sort of search behavior can be benefitial i don't think it is benefitial to most of factorio players.
The "powerful" behavior you describe honestly sounds to me like a sort of "dynamic shortcut" that works by remembering specific search words. I.e. if $some_person wants to find a "heat exchanger" for the first time it is neither intuitive nor faster for that person to come up with a sequence of letters that happens to uniquely identify the result they want in the given set of item names (e.g. "hx"), and much faster to just type four letters "heat". So the system is only benefitial to the minority of users who take the time to remeber the optimal character sequence for items that they search for very often.
Another drawback (imho) of the current system is that the false positives are mostly completely unrelated items. Examlple:
With "normal" search "heat" gets you "heat exchanger" and "heat pipe", two items directly related to each other.
With "sequential letters" (is there a proper name for this type of search?) if you were in a modded environment searching for your preferred term "hx" will find you "heat exchanger" but possibly also "sulfur hexaflouride" and "short logistic extender", both of which are probably completely unrelated to what you want to do at the time.

I would say, that we should have an option for this kind of search that should be off by default.

That would be the obvious "best of both" solution .

[DDDGamer]

Perhaps this is related, but i would agree, im searching for iron trains with keyword "iron" and i get extraneous results with stone and circuits.
If i search for "iron" i expect only results with "iron" showing.

train-search-bug.png (934.14 KiB) Viewed 6470 times

I i think it would be nice to be able to search train station by *tags*:
for example if you name a station [Smelting][Export][Iron] that would parce into tags "Smelting", "Export" and "Iron" that you can just use to quickly filter results by.
(Think newegg.com where i can narrow down my shopping search results based on specific tags like RAM, CPU, Motherboard, etc..)

[DDDGamer]

On the other hand this search performs as expected

train-search-station-ok.png (51.79 KiB) Viewed 6468 times

[eradicator]

[CIRcuit][ExpOrt-GreeN]
[MEF] ArtIlleRy LOadiNg
[MEF][ImpoRt][StONe]

I have to say that even though i know how it works now i find it quite difficult to locate the matching sequence. Interesting that the station name search seems to use normal search, nice find ;).

[DDDGamer]

Why would anyone even want this. I am a programmer myself, and i see no use for that type of search.

[Bilka]

Why would anyone even want this. I am a programmer myself, and i see no use for that type of search.

I'm sometimes still playing 0.15 and dearly miss the fixed search there, so I noticed what it makes better. Here's an example that I personally use:


It also still finds things even if you typo the search word:

[DDDGamer]

I think the benefit of having cool shortcut tricks /misspelled words does not outweigh not having standard search functions.
If there is both i see no problem, use whatever search terms you want sort of thing, but if the expectation is to display only iron trains when you search for iron and get superfulous/irrelevant information, thats is a a failure for the search algorithm.
Basically it becomes a "Norman Door" where you have to explain to the user how a door works, instead of the user just using the door. (Think those doors that have a label "push" or "pull", those are frustrating because that design is very bad and unintuitive, that to compensate for the bad design has to instruct the user on how to use it.)

[Jap2.0]

I agree that it should be an option. I personally rarely use the search, but from reading this I've found that it's a feature that is cool and useful but also very unintuitive.

[eradicator]

@Bilka:
Your examples demonstrate well what i criticised about the unintuitiveness of the current algorithm:

"LDS": This is a typical memorized shortcut. No new player will use it. It produces false positives (you have 3 tabs of results). And with a "normal" search algorithm you could reach the same result precision using "low" or "den".

"SCINC": So you found a typo that happens to have the same result as your intended search term. Had you typoed "since", "sciene", "scienv", etc this would not be the case. So it's more of a random side effect than an intended feature. There are methods to match "similar" strings when searching, which would then work for all other typos as well. I'm not sure i'd want that in factorio, but it would still be a better feature than coincidentally working or not depending on how one typoes.

Conclusion: I'm still in favor of defaulting to a normal algorythm. And no objections to obtionally include the current one ofc.

[Jap2.0]

I believe that "since" and "sciene" would work.

[Caine]

Why would anyone even want this. I am a programmer myself, and i see no use for that type of search.

For example, it is pretty great for matching camelcased identifiers (by just typing the capital letters).

[jiri]

Hi,

I'm the guy who wrote the new search function. The way it works is actually really simple, it just checks that the search pattern and the searched terms have the same letters in the same order and treats each word of the search pattern separately. What that means is that if you get a lot of results at first, you can just hit space and refine the search.

Why would anyone even want this. I am a programmer myself, and i see no use for that type of search.

I find this very useful when coding, because most of the time, I don't know what I'm looking for and this enables me to refine the search very fast. With the larger number of false positives, you get a better overview of the searched space and you're less likely to miss something. Come to think of it, that's probably not the best approximation of how people use the search function in Factorio, as the "exploration" phase happens through the GUI and people mostly use the search function to get something they already know they want.

The main reason I did this was to make the search function a bit more forgiving for typos. After reading this thread, I do agree that it can be significantly improved, so I'll look into making it work more like the original one, while still trying to make it more forgiving for clumsy people like me

[DaveMcW]

make it more forgiving for clumsy people like me

https://en.wikipedia.org/wiki/Damerau%E ... n_distance

P.S. I made this suggestion in a different topic, but it would be really nice to remove item groups in the search results so everything is in one tab.

[Zavian]

make it more forgiving for clumsy people like me

https://en.wikipedia.org/wiki/Damerau%E ... n_distance

P.S. I made this suggestion in a different topic, but it would be really nice to remove item groups in the search results so everything is in one tab.

I'm not sure I agree. Showing the tabs helps players learn where to find things, so that in future, they don't need to search, and can just go to the correct tab.

[Caine]

I think the false positives of using the edit distance may be even less intuitive than the current search pattern, although it will strongly depend on the number of edit operations you allow.

[eradicator]

I find this very useful when coding, because most of the time, I don't know what I'm looking for and this enables me to refine the search very fast. With the larger number of false positives, you get a better overview of the searched space and you're less likely to miss something.

That pretty much what i suspected this was used for. But as you already noticed yourself the inventory search is a quite different usecase :D. I wonder if it would be feasible to actually highlight the matched pattern in the search result, i.e. searching for heat resulting in a tooltip of "Heat pipe" when hovering heat pipe in the search results. But i have a feeling that that would require a largish rewrite of that part of the GUI.

The main reason I did this was to make the search function a bit more forgiving for typos. After reading this thread, I do agree that it can be significantly improved, so I'll look into making it work more like the original one, while still trying to make it more forgiving for clumsy people like me :)

Wikipedia apparently has a whole list page for different edit-distance alrotythms: Category:String_similarity_measures. Personally for that kind of "typo aware" search function i'd prefer a two-staged results page. On the top show only exact results, and then below have a clearly marked list of "similar results"/"did you mean?" results that include possible typos. Because not everything that looks like a typo nessecarily is a typo.

P.S. I made this suggestion in a different topic, but it would be really nice to remove item groups in the search results so everything is in one tab.

I'm not sure I agree. Showing the tabs helps players learn where to find things, so that in future, they don't need to search, and can just go to the correct tab.

Personally i've played enough factorio to know all the items. So i use the search functionality exclusively to get where i want faster. Because for me it's faster to just ctrl-f -> "heat" -> click heat pipe than to remember which category heatpipe belongs to and then find the symbol in a heap of other symbols. (My "finding symbols in heaps of other symbols"-skill might be below average. Which is not helped by the fact that most building icons are tiny "kinda-grey-blackish-blobs". :P) The problem with category tabs for the results in my usecase is that they would (often) require me to click several buttons before i can see the result i want. I.e. instead of one click i'd need to do 4-5 clicks again, which nullifies all the speed benefits from using the search function in the first place (for me). Which is a major reason why i have completely turned of categories (and wasn't too happy that the options menu entry got removed :x, for the rare cases where i did enable categories temporarily. For example when explaining things to someone who has them enabled. Or debugging my mods.). Though i don't play with any mods that add large amounts of recipes, otherwise turning of categories completely would probably not be feasible anymore.

PS: Also just because people sometimes misinterpret my posts. I'm not angry or something. I'm just trying to explain how i play the game ;). And thanks @jiri for the effort to help find a more universal solution.