Mod quality & performance stats

[thelordodin]

Mod quality & performance stats:
- Timespan a mod takes in game loading
Why: my Factorio loads 5-7 minutes on Ryzen 3900x. How do I know which mods should I disable to make it faster?

- Overall RAM usage per mod in game
Why: i had times when game crushed on deathworlds having more than 32Gb RAM in use. I don't know which mod caused this.

- Overall CPU usage per mod in game

This statistics should be gathered and sent to mod portal, so users and modders could know if specific mod is or isn't "hungry" for RAM/CPU/Game start time.

How to measure: in % compared to base mod, average between all modded games, show all 3+1 specs (RAM/CPU/Game start time + number of issues solved on portal) on mod portal.

THE WHY:
- Just measuring this will increase the quality of all mods.
- Can have an aggregated stat "quality" composed of all other stats
- I want to see when a new good mod appears, even if it's small and not yet reviewed/trending. For now I just guided by authors (Klonan makes good mods:))

[Koub]

This statistics should be gathered and sent to mod portal

This is tricky. Despite almost everyone doing so, automatic data collection and sending is increasingly frowned upon - and illegal in some places of the world. I remember the shitstorm when the devs activated by default the option to send logs upon crash.
There should be an option, ideally opt-in, and the devs would have to ensure no personal data is collected in the process.

All that being said, I don't think many people would activate the option themselves.

[mrvn]

Timinig statistics can be shown F4 -> show-time-usage with scripts broken down per mod. Just send the mod author a screenshot and savegame if it consumes too much.

[thelordodin]

This statistics should be gathered and sent to mod portal

This is tricky. Despite almost everyone doing so, automatic data collection and sending is increasingly frowned upon - and illegal in some places of the world. I remember the shitstorm when the devs activated by default the option to send logs upon crash.
There should be an option, ideally opt-in, and the devs would have to ensure no personal data is collected in the process.

All that being said, I don't think many people would activate the option themselves.

You can just segmentate users: the ones who enable will see the marks, the ones who dont - wont see them.
The choise should be done when user opens "mods".
Also if he picked "Don't send" - ask him again just 3 times, once a month. This way most users will chooser "send".

[jodokus31]

Mod quality & performance stats:
- Timespan a mod takes in game loading
Why: my Factorio loads 5-7 minutes on Ryzen 3900x. How do I know which mods should I disable to make it faster?

You can actually see, which mod takes time in the progress bar.
Maybe also have a look into the log-file.

The other way is disable mods and see, how fast it gets. And enable one by one.

Just for reference, My seablock mod pack (50 mods), loads in 20 sec. from a SSD,. with 5-7 min. something is really screwed.

BTW: If you got the mod merging-chests or Wide chests by chance, the default settings are(or were at least) insane. You have to lower chest size

[thelordodin]

Just for reference, My seablock mod pack (50 mods), loads in 20 sec. from a SSD,. with 5-7 min. something is really screwed.

Strange... I have really fast SSD (Samsung 970evo plus) and still have this issue. I'll try looking into logs as you suggested.

Disabling one by one isnt an option, because each will cost about 5 minutes x 50 mods is 250 minutes = 2.5 hours just to find reason...

[UkcsAlias]

All that being said, I don't think many people would activate the option themselves.

It would usualy get activated when players run into trouble, which is exactly the moment when this info would be welcome. For those who never run into issues, there is no need to activate it as to them the mod works fine. Sure, they can face the same issue and have an effectively slower game, but they might simply not notice it. As in the end, what is an acceptable framerate depends on the player.

Automaticly sending the info to the dev isnt always needed for that, as its most of the time the mod itself being the cause (and the log should reveal that), and usualy just deactivating that mod would be enough. And adding a button/console command for that might be enough to generate a file that can be sent (a logfile someone can post on the forums). You can put an automated system for this, but i think that issues in this game are rare enough to not need that.

[jodokus31]

Just for reference, My seablock mod pack (50 mods), loads in 20 sec. from a SSD,. with 5-7 min. something is really screwed.

Strange... I have really fast SSD (Samsung 970evo plus) and still have this issue. I'll try looking into logs as you suggested.

Disabling one by one isnt an option, because each will cost about 5 minutes x 50 mods is 250 minutes = 2.5 hours just to find reason...

It would help, if you'd disable all mods and check how long that goes. This should not work for 5 min. If it's still that long, then it's not a problem with mods...

[Koub]

Just for reference, My seablock mod pack (50 mods), loads in 20 sec. from a SSD,. with 5-7 min. something is really screwed.

Strange... I have really fast SSD (Samsung 970evo plus) and still have this issue. I'll try looking into logs as you suggested.

Disabling one by one isnt an option, because each will cost about 5 minutes x 50 mods is 250 minutes = 2.5 hours just to find reason...

If you think the issue might be caused by a single mod, dichotomy is the king. Instead of doing it mod by mod, do it with half the mods. If the issue persists, the culprit is in the 50% still active mods. if not, it's in the 50% deactivated mods. Rince and repeat. You're guaranteed to find the problematic mod in 6 tries for a 50 mods pack.

[mrvn]

Just for reference, My seablock mod pack (50 mods), loads in 20 sec. from a SSD,. with 5-7 min. something is really screwed.

Strange... I have really fast SSD (Samsung 970evo plus) and still have this issue. I'll try looking into logs as you suggested.

Disabling one by one isnt an option, because each will cost about 5 minutes x 50 mods is 250 minutes = 2.5 hours just to find reason...

Disable half. That only takes 5 minutes * 6 = 30 minutes. And that's assuming it's the last mod costing all the time and every run takes 5 minutes.

With 50 mods and 5 minutes loading time each mod should take 6 seconds average on the loading screen. Now that is split by data, data-fixes, data-final-fixes and loading sprites and audio. And most mods should flicker just past. Isn't there any that remain listed on the loading screen longer than others?

[SoShootMe]

If you think the issue might be caused by a single mod, dichotomy is the king. Instead of doing it mod by mod, do it with half the mods. If the issue persists, the culprit is in the 50% still active mods. if not, it's in the 50% deactivated mods. Rince and repeat. You're guaranteed to find the problematic mod in 6 tries for a 50 mods pack.

Much better. Each doubling of mods need one extra try. On the other hand, if it's a combination of mods, there are 2^50 (~1e15) of them to try... Each extra mod needs double the tries!

But the first thing to do is to look in the log file, which has timestamps for mod loading events.

[Koub]

But the first thing to do is to look in the log file, which has timestamps for mod loading events.

Naaah too mainstream. Who reads logs nowadays ?
Now I feel stupid not to have thought about this.

[ssilk]

Mod quality & performance stats:
- Timespan a mod takes in game loading
Why: my Factorio loads 5-7 minutes on Ryzen 3900x. How do I know which mods should I disable to make it faster?

As already mentioned you can look into the factorio current log.

https://wiki.factorio.com/Log_file

- Overall RAM usage per mod in game
Why: i had times when game crushed on deathworlds having more than 32Gb RAM in use. I don't know which mod caused this.

This is hard to tell. Because if you store for example some extra information in a tile (a piece of ground) you cannot clearly distinguish if it belongs to the mod or the game.

- Overall CPU usage per mod in game

Can be seen in debug view. https://wiki.factorio.com/Debug_mode

This statistics should be gathered and sent to mod portal, so users and modders could know if specific mod is or isn't "hungry" for RAM/CPU/Game start time.

Bad idea. Because that is extremely dependent on how you use the game and how the concept of the mod is.

If the mod adds just a new entity (a new type of solar cell for example), it don’t uses any cpu, but you can place it down million of times and then it might use more cpu.

I tend to move this to “implemented”.

- Just measuring this will increase the quality of all mods.

no, you can’t measure this for comparisons, the differences between the games are too high.

- Can have an aggregated stat "quality" composed of all other stats

That’s not a quality. The quality of a mod is measured by how fun it is to play with it. Or how this integrates into other mods. And many, many more.

[jodokus31]

- Overall RAM usage per mod in game
Why: i had times when game crushed on deathworlds having more than 32Gb RAM in use. I don't know which mod caused this.

This is hard to tell. Because if you store for example some extra information in a tile (a piece of ground) you cannot clearly distinguish if it belongs to the mod or the game.

I think, the game engine could be able to distinguish, which mod adds which field in a prototype and what's the result in RAM usage. But I'm not sure, if it's worth. It might introduce more overhead

[mrvn]

- Overall RAM usage per mod in game
Why: i had times when game crushed on deathworlds having more than 32Gb RAM in use. I don't know which mod caused this.

This is hard to tell. Because if you store for example some extra information in a tile (a piece of ground) you cannot clearly distinguish if it belongs to the mod or the game.

I think, the game engine could be able to distinguish, which mod adds which field in a prototype and what's the result in RAM usage. But I'm not sure, if it's worth. It might introduce more overhead

Since when can a mod add anything to the prototype? They can change the value of existing properties (some of which might be nil and not in the lua table). But I doubt the C++ structure it all gets parsed into is flexible in size.

[jodokus31]

- Overall RAM usage per mod in game
Why: i had times when game crushed on deathworlds having more than 32Gb RAM in use. I don't know which mod caused this.

This is hard to tell. Because if you store for example some extra information in a tile (a piece of ground) you cannot clearly distinguish if it belongs to the mod or the game.

I think, the game engine could be able to distinguish, which mod adds which field in a prototype and what's the result in RAM usage. But I'm not sure, if it's worth. It might introduce more overhead

Since when can a mod add anything to the prototype? They can change the value of existing properties (some of which might be nil and not in the lua table). But I doubt the C++ structure it all gets parsed into is flexible in size.

Not sure, I would at least assume, that some properties have dynamic size, like strings, images or arrays of something, etc

[mrvn]

- Overall RAM usage per mod in game
Why: i had times when game crushed on deathworlds having more than 32Gb RAM in use. I don't know which mod caused this.

This is hard to tell. Because if you store for example some extra information in a tile (a piece of ground) you cannot clearly distinguish if it belongs to the mod or the game.

I think, the game engine could be able to distinguish, which mod adds which field in a prototype and what's the result in RAM usage. But I'm not sure, if it's worth. It might introduce more overhead

Since when can a mod add anything to the prototype? They can change the value of existing properties (some of which might be nil and not in the lua table). But I doubt the C++ structure it all gets parsed into is flexible in size.

Not sure, I would at least assume, that some properties have dynamic size, like strings, images or arrays of something, etc

Totally, those will be pointers to the data eventually. I assume std::string is used for strings and std::vector for arrays and such.

Another thing: prototypes exist only once. So their memory footprint is not really anything to judge by. The is one inserter prototype but a million inserters.

I think what would be interesting would be the global table for each mod. If it is tagging on extra data to entities then that is where it gets stored. If every inserter gets put into a mods global table then you would notice that in the size.

[ssilk]

What I meant is, that I have in my mod a hash-table, the hash points to the ID of the entity. So seen it’s clear, that this memory is used by the mod, and the thought that it could be counted to the game is stupid.

And it’s also clear, that it is easy to count, how much memory a mod needs. I would distinguish between local storage and memory that needs to be included into the save.

[mrvn]

What I meant is, that I have in my mod a hash-table, the hash points to the ID of the entity. So seen it’s clear, that this memory is used by the mod, and the thought that it could be counted to the game is stupid.

And it’s also clear, that it is easy to count, how much memory a mod needs. I would distinguish between local storage and memory that needs to be included into the save.

Oh yeah, you can have memory needed temporarily while processing an event too. Although any noticeable memory consumption there would also have a noticable time consumption. Your game would freeze if the mod fills 1GB of memory with data in some event.

Or memory used longer time but not saved and reconstructed on load. Stuff to speed up running the mod.

[FuryoftheStars]

Personal usage statistics is good, but sharing it up online I feel as though would not have the intended result. Different hardware will have different results, even if you could reliably tell every time how much of the usage was base vs mod (the mod may add 10% to computer A, but computer B it adds 20%). Then you also have the interactions between mods that may inflate these (they’re not necessarily either mod’s fault).