[MOD 0.13] EfficienSee : optimise your factory !

[Litcube]

"And I will soon add a recipe and item list to pickup diretly from the list."

Yeah, I think this was what I was thinking of. Being able to select an ingredient from a list, and then it would show you all recipes that use that ingredient. Is this what you mean?

Invaluable for planning a bus.

[binbinhfr]

"And I will soon add a recipe and item list to pickup diretly from the list."

Yeah, I think this was what I was thinking of. Being able to select an ingredient from a list, and then it would show you all recipes that use that ingredient. Is this what you mean?

Invaluable for planning a bus.

Yes that's what I mean, but you can already do that, by selecting an item :
pick up the item you want from your bag and click with it on the item icon in the mod window.
The item will be loaded and its linked recipes will all be shown (if you checked "unused").
Use also key B and N (click Help button to have explanations)

Then you can click a recipe in the "ingredient of" list. And the recipe will be shown. Then click whatever recipe or item and you will navigate thru the recipes tree.

[Litcube]

Nice! Really good work!

[binbinhfr]

For those who did not give a look at the interface since the beginning, there have been changes :

screen11.jpg (317.43 KiB) Viewed 6890 times

screen12.jpg (397.61 KiB) Viewed 6890 times

[Jos]

First, I'll report a simple bug of locale text. The 2 invalid lines in en.cfg should be rewrite as:

Code: Select all

effic_hotkey_on_map = EfficienSee machines stats on/off

1.png (21.95 KiB) Viewed 6881 times

Second, I have a suggestion about the removed main list for other use.
Obviously, the main list is useless for the whole production chain and occupies a large amount of system resources. However, it is of great use for local modular designs (like the following examples), which suits copying by blueprint. It is best to select the specified machines to give stats, and you can never worry about the consumption of CPU & memory. What's more, except for production stat, addtional stats can also be added, such as the construction cost, energy consumption, pollution ...

Examples of local modular designs

2.png (742.99 KiB) Viewed 6881 times

3.png (452.32 KiB) Viewed 6881 times

4.png (541.69 KiB) Viewed 6881 times

Besides, I have to say, your mod is what I've dreamed to make for a long time, but having no time to work on it. Your work is excellent! Thank you!

[binbinhfr]

Hi Jos,

thx for your return and appreciation, it's always welcome

I corrected the locale typo, thx. New version published.

For your suggestion, the problem is that factorio API does not provide any production information per machine. So I cannot compute the production of a subset of machines. Factorio API only gives information about the overall number of items produced and consumed per item type since the beginnig of the game, ignoring where they are constructed... I wish there was this kind of precise information. On a machine I can just say if it works or not, and test if its output are blocked or input are starved... (what i'm using for the icons on the machine, and rough employement rates)

[Qon]

For your suggestion, the problem is that factorio API does not provide any production information per machine.

... but you could calculate it if you know which recipies and modules are used. Since mods can manipulate modules installed that part should be possible to read. Beacons in proximity can also be included. Aren't recipies readable from the API?

[binbinhfr]

For your suggestion, the problem is that factorio API does not provide any production information per machine.

... but you could calculate it if you know which recipies and modules are used. Since mods can manipulate modules installed that part should be possible to read. Beacons in proximity can also be included. Aren't recipies readable from the API?

If you mean calculating the theorical normalised output and optimised inputs of a subset of machines, yes it is possible.
But it's not the purpose of my mod that is already quite complex.
I think that Helfima is working on the type of mod you are talking about.

[Qon]

For your suggestion, the problem is that factorio API does not provide any production information per machine.

... but you could calculate it if you know which recipies and modules are used. Since mods can manipulate modules installed that part should be possible to read. Beacons in proximity can also be included. Aren't recipies readable from the API?

If you mean calculating the theorical normalised output and optimised inputs of a subset of machines, yes it is possible.
But it's not the purpose of my mod that is already quite complex.

Once you know the theoretical production of a machine you can just multiply that with the utilisation % that you already have for effective production stats of a limited subset of machines and not just the global stats. So it is within the scope of your mod if you want to support stats of a subset of machines, which many have requested. It's to go around the limitation of the API only prividing global stats. I didn't suggest calculating that only for knowing the theoretical production limits. But if it's too complex I guess we will have to be satisfied with what we have, which is still good enough for my use cases since I test my factories in individual worlds.

[binbinhfr]

I already thought about this, because I had this idea before, when preparing this mod.
BUT I always bump into the same problems :
- One small (but painful to solve) problem is that not every machine has the same setup (modules, speed of assembling level, beacon presence around, or whatever other mod modifies the assembling performances). So the calculation would be good only if all machines are setup the same way.
- More than that, it assumes that all machines are perfectly fed and perfectly freed of their production. It is not the case, and I have no way to quantify this. (it depends on the setup of the inserters/belt/number of robots/place of roboports, trains traffic...)
- And your assumption supposes that each item is used only in one recipe which is not the case (ex: iron-plate). So what should I do with this overall iron-plate production (used in several recipes) in your machine-subset that uses only one iron-plate-base recipe ? How can I know what part of this overall production is dedicated to this precise recipe, in order to scale it ? I can't because there are no per-recipe or per-machine stats provided by factorio api.

For me, these are problems that are impossible to solve. We do not have all the necessary data.

Infact, playing day to day with my big-factory, I am quite happy with the tool as it is. You must give it a real try. I already could optimise my factory by a top to bottom analysis. And I detected a lot of things that were invisible in this forest of thousands of machines.
The very useful thing for me is these on-map icons. They are the way to correlate the window overall numbers with the reality of the map. WIth keys N and B, you can quickly swap from one recipe to another, of from one family of machines to another, just hovering on them on the map and clicking N or B.

I am actually adding a recipes list to pick up and study any recipe or item. This is what I'm lacking for the moment.

For the rest, I'm quite happy with the CPU stress. Even with all icons on, there is only a little imperceptible lag with my 5500 machines factory.

I'd like to have more returns of testers, especially on cpu stress / lags / freeze, etc...

Anyway, I will think about other interesting features once the actual tool will be stable, which is not totally the case right now, because I (and some courageous testers) always point out some strange exceptions related to factorio or numerous other mods, which explains the high number of subversions I published...

[Qon]

- One small (but painful to solve) problem is that not every machine has the same setup (modules, speed of assembling level, beacon presence around, or whatever other mod modifies the assembling performances). So the calculation would be good only if all machines are setup the same way

Or if you do the calculation per machine. The machines machines theoretic input/output only has to be qalculated when you select them of course.

- More than that, it assumes that all machines are perfectly fed and perfectly freed of their production. It is not the case, and I have no way to quantify this. (it depends on the setup of the inserters/belt/number of robots/place of roboports, trains traffic...)

But you already do quantify that. Isn't that exactly what the % usage marker and the input starved/output blocked icons are for?

- And your assumption supposes that each item is used only in one recipe which is not the case (ex: iron-plate). So what should I do with this overall iron-plate production (used in several recipes) in your machine-subset that uses only one iron-plate-base recipe ? How can I know what part of this overall production is dedicated to this precise recipe, in order to scale it ? I can't because there are no per-recipe or per-machine stats provided by factorio api.

I don't suppose that. If I know the theroretical properties of a certain machine and that it is used 50% then I just multiply those numbers for my answer. And then I do this for all machines. The logistics is not a part of the equation if you already know how much % of the time the machines are having input/output problems.

For me, these are problems that are impossible to solve. We do not have all the necessary data.

Infact, playing day to day with my big-factory, I am quite happy with the tool as it is. You must give it a real try. I already could optimise my factory by a top to bottom analysis. And I detected a lot of things that were invisible in this forest of thousands of machines.

What data is missing? What part of the mod is it that I've misunderstood?

I haven't tried this on my mega factory WIP yet but I will soon. As I said the way I build my factories at the moment non-global analysis isn't necessary for me. I'm just not agreeing with that it is impossible to implementing it since all data seems to be there even if it's some work to extract it all. If it's too much work then I understand, but first I want us both to understand what is possible and what is not. Seems we are misunderstanding eachother here.

I'd like to have more returns of testers, especially on cpu stress / lags / freeze, etc...

I'll build something big quickly in a cheat map and see how it performs

Anyway, I will think about other interesting features once the actual tool will be stable, which is not totally the case right now, because I (and some courageous testers) always point out some strange exceptions related to factorio or numerous other mods, which explains the high number of subversions I published...

Of course. You are doing the factorio community a great service with your long list of fantastic mods !

And this is what happened when I updated ES through in game auto mod update to latest version of the mod:

ERROR

EfficienSeeError.png (3.3 MiB) Viewed 7116 times

[binbinhfr]

But you already do quantify that. Isn't that exactly what the % usage marker and the input starved/output blocked icons are for?

Oh no it's not a quantification but more a "qualification", that's why I erased the % and only show a color now. Infact during testings, I saw that these values are rough estimates and are sometimes locally false. Because I'm polling every 1sec, the result depends when I do the measure. For example, a machine seems to craft in loop, so you would say it is 100% employed. But my tool will probably not give this value, but something less, because when I poll, it can be just on the moment between 2 crafts. Factorio has little hickups, especially when craft is quick. The same with input / output : depending on the moment, I can catch a bad situation. So you will notice that for a single machine, the local information is not always stable, even when the machine seems stable. I just reconstruct this information from this periodic innacurate polling. That's why I put a warning somewhere saying that those infos are rough estimates. But statistically and qualitatively, they still show very interesting results. Only the production per items is accurate (coming from the factorio api)

Note that a mod like Bottleneck_0.0.1 is doing this polling every tick, which is more accurate, but it's not viable. The cpu is quickly crashing down...

And this is what happened when I updated ES through in game auto mod update to latest version of the mod:

arf, did you try very last version ?
because i think it's a stupid bug I already corrected following a user report.

Coming back to our need of precision measuring, I had another idea, which will give accurate results :
Production stats of the API are per-force. So a solution to analysis a subset of machines is to transfer them temporarly into another dedicated force with the player-tester. Then all prod/cons data per item will concern only the isolated subset. But testings must be made, because energy must follow, and robots won't work for the enemy... It's just an idea like that. With its own problems too.

[Qon]

Note that a mod like Bottleneck_0.0.1 is doing this polling every tick, which is more accurate, but it's not viable. The cpu is quickly crashing down...

Well since it's a limited subset of your factory we are talking about it might actually be viable.

arf, did you try very last version ?
because i think it's a stupid bug I already corrected following a user report.

Think it was 1.0.32. Didn't get a chance to report it immediatly. And now it's on 1.0.35 so maybe it is fixed. Can't really know now though because I've saved without the mod so the problem would probably just disappear anyway.

Coming back to our need of precision measuring, I had another idea, which will give accurate results :
Production stats of the API are per-force. So a solution to analysis a subset of machines is to transfer them temporarly into another dedicated force with the player-tester. Then all prod/cons data per item will concern only the isolated subset. But testings must be made, because energy must follow, and robots won't work for the enemy... It's just an idea like that. With its own problems too.

Sounds awesome. As long as you don't have turrets or robot army close by that suddenly decides to attack your own factory xD
But with a bit of a warning or if there's some ally force feature it would be nice.

[binbinhfr]

Think it was 1.0.32. Didn't get a chance to report it immediatly. And now it's on 1.0.35 so maybe it is fixed. Can't really know now though because I've saved without the mod so the problem would probably just disappear anyway.

Yes it is fixed. It was a problem with hidden recipes and technos that are not always managd the same way by all mods.
The problem with mods like mine is that they can interact with a lot of other mods, and I cannot test them all, or know how all of them are made and use turnarounds...

Sounds awesome. As long as you don't have turrets or robot army close by that suddenly decides to attack your own factory xD
But with a bit of a warning or if there's some ally force feature it would be nice.

Oh, I can make the temporary force ally. I made some tests like this with my SpaceBook mod. It seems that forces can even share/steal electricity sometimes, but I don't know if it's a factorio bug or intended...

[binbinhfr]

You just had me concerned when you said my save my be dirty with older version data structures/indexes since I don't have a pure pre-efficiensee mod save at this point.

Peppe, coming back to your fear :
memory problems should now be gone. I never encountered them again, and my big test factory has passed safely thru this long list of subversions !
The problem with deserialisation came from the very first huge items table that I indexed directly by entites.
It was recently clearly identified as a major problem by Rseding here :
https://github.com/troelsbjerre/Bottleneck/issues/1
Now I don't do this anymore, only using integer or string indexed tables, so no more problem to fear ! (well unless maybe some little classical bugs... )

[Jos]

Hi Jos,

thx for your return and appreciation, it's always welcome

I corrected the locale typo, thx. New version published.

For your suggestion, the problem is that factorio API does not provide any production information per machine. So I cannot compute the production of a subset of machines. Factorio API only gives information about the overall number of items produced and consumed per item type since the beginnig of the game, ignoring where they are constructed... I wish there was this kind of precise information. On a machine I can just say if it works or not, and test if its output are blocked or input are starved... (what i'm using for the icons on the machine, and rough employement rates)

Well, I think it's unnecessary to acquire the exact amount of each machine's produced items counted only if it is created, which we have no measure to get by API. Instead, a "rough" estimation of production rate of selected machines on map still has great significance for efficiency optimization.

As what you have said and what you have done in EfficenSee, the following information of each machine can certainlly be get from Factorio API:

• Machine itself
  Recipe selected
  Module inserted
  Machine work status

With the information of machine, recipe and module, for each machine, it's enough to calculate the exact maximum value of each product's dP, each ingredient's dC, energy dC and production dP. What's more, there is no need to calculate them every polling until machine, recipe or module changed.
With the machine work status, you have got the rough employement rate.
Then, the maximum value plus the employement rate, we can get the "rough" estimation of each machine's dPs & dCs. They can also be added up with the same family. Further more, the total employement rate of one family (with same production) can be calculated by max-dP-weighted meaning the employement rate of each machine.

You can see the above thought as a mod-based and built-in Factoratio, but no simulator is better than Factorio itself. Thus, this function can completely replace all other quantitative design tools and be easier than any of them.

[binbinhfr]

Hi Jos, I see what you mean, and it should be feasible, even if it's a huge work to take everything into account, considering the huge variety of settings, influencers, but also some tweaking mods and their underlying subtilities.

So you would like on each machine a rough estimation of its local production ?

But I wonder what you are waiting from such a measurement ? Not precise numbers, because my employement rates would always be rough estimates (most of the times, on a perfectly working machine, I measure a rate of 80-90%... because of the inacurate polling)... So trying to optimise these inaccurate numbers seems strange to me. And if it's not to precisely tweak a subset of machines, what is it for ? How would you use it ? I do not understand what it would really bring in terms of optimisation, that I can already do with EffSee.

Maybe could you please describe the step by step process on how you would analysis a subset of machines with such a tool, and what conclusion/diagnosis you would get from it ?

[Jos]

Hi Jos, I see what you mean, and it should be feasible, even if it's a huge work to take everything into account, considering the huge variety of settings, influencers, but also some tweaking mods and their underlying subtilities.

So you would like on each machine a rough estimation of its local production ?

But I wonder what you are waiting from such a measurement ? Not precise numbers, because my employement rates would always be rough estimates (most of the times, on a perfectly working machine, I measure a rate of 80-90%... because of the inacurate polling)... So trying to optimise these inaccurate numbers seems strange to me. And if it's not to precisely tweak a subset of machines, what is it for ? How would you use it ? I do not understand what it would really bring in terms of optimisation, that I can already do with EffSee.

Maybe could you please describe the step by step process on how you would analysis a subset of machines with such a tool, and what conclusion/diagnosis you would get from it ?

I'm wrong, because I never imagine the employment rate is so "rough". Maybe it's truely feasible when according APIs are added, such as the events of starting & finishing producing an item for the selected machines.

[binbinhfr]

I'm wrong, because I never imagine the employment rate is so "rough". Maybe it's truely feasible when according APIs are added, such as the events of starting & finishing producing an item for the selected machines.

Yes it is rough. Because of the 1 sec polling that does not always the right information at the right time... On my 5000+ machines factory (which is small in comparison of what I saw), it already check every machine every second to test its crafting status, inputs and outputs (whether they are items or fluids, etc...). It's already a bunch of computation if you display icons on all machines...

But even if rough, these informations are quite enough to detect problems on machines. When you zoom out and you see that everything is square green, it's a good sign. When you have yellow or red and arrows, you have to zoom in and use your brain

I doubt they would ever create a item_produced event. When you have more than 5000 machines producing more than 1 item per second, can you imagine the number of event fired and that a mod should filter ?
What we could hope for is a production/consumption per machine, and even a recipe_changed event to follow more precisely what happens. It would be less heavy.

[binbinhfr]

For those who want to test new important features, please update :

1.0.38 - add a recipes and items browsers to open any recipe/item.
click on the main icon of recipe or object pane to open it.
- separate assemblers/furnaces/drills computation on 6 different ticks to release cpu stress
+ 1 tick to refresh window + 1 desync tick for polling selected machine.
- when resetting data (after install or mod change), reset counters to current time,
to show consistent data (total numbers are disturbing beginners).
- use machine mem recipe to toggle furnaces on-map infos per recipe.
- use machine mem mined entity to toggle drills on-map infos per ore.
- now update machine buttons color (double arrow) depending on the display status of on-map infos.
- add a show hidden recipe checkbox, with hidden recipes names in orange
- create a real help window