[MOD 1.1] Will-o'-the-Wisps updated

[mk-fg]

Save and mod list would do the same btw, but it's a bit of work installing all that, and I hope that maybe running such command on your end might be simplier.

[Bloodymight]

Hi,

I replaced the mod with your "debug" one

game didn't start, got an error at start up: https://imgur.com/a/8FWBsRL
...so I removed "-debug"(essentially same name as the replaced one) from it's name and it worked

but using your "/wisp zone export" command gave me this error:

"Error while running command "wisp": __Will-o-the-Wisps_updated__/libs/zones.lua:349: bad argument #1 to 'format' (not a number in proper range)"

feels like I "installed" it wrong but not sure

kinda off-topic... currently running 88 mods on my biggest map yet, and the game hasn't crashed after 12 hours. but eats like 16GB RAM

[mk-fg]

Forgot that zip files can be used in mod dir as-is, so didn't think that its name would matter, my bad.

You did rename it correctly though, but quick and dirty serialization code for that export command couldn't handle chunk offsets used in your game.
(used %d serialization for json keys, which can't handle 31bit+ integers, but didn't cause a problem in a relatively early save test)

Should be fixed it now, please try same stuff with the attached version (except zip should be named correctly now).

[mk-fg]

kinda off-topic... currently running 88 mods on my biggest map yet, and the game hasn't crashed after 12 hours. but eats like 16GB RAM

I think that's likely to be a memory leak in one of these mods, as I doubt anything in lua code can actually use 16G of data and don't lag horribly while doing that
("memory leak" is when code stores values somewhere but never cleans them up later, so memory usage grows indefinitely)

One quick test for that might be loading specific save before you'll go off somewhere, leaving it running for a few hours (unpaused) while doing absolutely nothing.
After these few hours, make a save and check memory usage.
If memory usage grows significantly (e.g. 2x, 3x or such) - almost certainly a leak in one of the mods.
Comparing disk space used by pre- and post-idle-time savegames, you can also check if this data ends up stored by some mod there, which might be even more troubling.

There are more sophisticated tools in factorio to debug this issue though, like F-something key to show mod resource usage, which should probably show which mod it is exactly, but don't quite remember how to read or use these off the top of my head, you might want to check more dedicated forum threads or wiki pages on that.

[mk-fg]

Also, to be fair, loading an 8-hour game here with a more modest collection of 37 mostly quality-of-life mods, factorio takes 1.7G actual and 3.2G virtual memory (allocated, but not necessarily ever used) here, so guess 4-8G might be normal for more extensive and more content-heavy mod collection.

[Bloodymight]

game eats 13GB in main menu and 16GB after loading savegame... so yeah

btw I tried it again and got the exact same error

didn't cause a problem in a relatively early save test

I'm currently at 170 hours on my map

[mk-fg]

btw I tried it again and got the exact same error

Ah, damn, guess these integers might be too big for any printf() format, sorry
Guess I'll test it better locally this time and come up with something more likely to work in your game.

game eats 13GB in main menu and 16GB after loading savegame... so yeah
I'm currently at 170 hours on my map

Ah, guess 16G might indeed be normal for that kind of game
Mine never got anywhere near that for sure, but I tend to play with reduced research costs and generally less grindy stuff, so probably not a good judge of factorio game length either.

[Bloodymight]

interesting thing is, I still didnt have any crashes since I reinstalled your mod, so not sure what to make out of that

I packed all mods and 2 saves together(latest main save(test 2) we use and one where it all started (test))
Save file where it all started uses less mods but both crashed before I reinstalled wisps

reduced research costs and generally less grindy stuff

wanted to start with bobs+angels but that was a bit too much

bob is fine atm


Edit: apparently I cant upload a 374MB file here

[mk-fg]

interesting thing is, I still didnt have any crashes since I reinstalled your mod, so not sure what to make out of that

I packed all mods and 2 saves together(latest main save(test 2) we use and one where it all started (test))
Save file where it all started uses less mods but both crashed before I reinstalled wisps

With my earlier theory, I just thought you might have some mod that does something with the map.
Can be any map-generation mod, though not entirely sure how they'd unload previously-generated map chunks, but didn't look into it much either.

Did just try to reproduce the issue with serialization, and couldn't do it with any large integer, so guess it might be some other bug after all - maybe some integer keys overflowing into floats randomly.
So currently was fixing serialization to properly represent either in json as strings, to see what ended up happening in your game with them.

Edit: apparently I cant upload a 374MB file here

You can maybe cheat the system and upload a bunch of smaller files, if limit is anywhere close to that.
Otherwise I'd probably try something like dropbox, onedrive, google drive, etc - "cloud" things, whichever one you already have an account with.

Will probably end up running exactly same routine on that save though, just need to get it working for weirder values which I didn't expect to be there at all (and they probably shouldn't be there).

[Bloodymight]

I used a mod to remove unused chunks so map adds newly enabled resources maybe that did something?


well nvm you said there were other people with the same error

currently uploading on onedrive, my inet is pretty slow should be done in 15 minutes ._.

[mk-fg]

I used a mod to remove unused chunks so map adds newly enabled resources maybe that did something?


well nvm you said there were other people with the same error

No no, that'd probably do it, if it does this by re-generating map chunks, which it probably does.

Other people had error in same function due to unrelated cache invalidation issue, which doesn't seem to be the case here at all, I just thought at first it might be, due to how easy it is to mess up with caches.

There is even a rather common programming proverb/mantra for that

There are only two hard things in Computer Science: cache invalidation and naming things.

[mk-fg]

I did fix that "/wisp zone export" command once again, if you want to give it another try, should definitely work now.
Though if you'll upload the save anyway, maybe not much need to bother.

[Bloodymight]

https://1drv.ms/u/s!AniVyk3veOIicVZw3pv8nMsRYcM

uploaded save and modlist

looks like it worked, this time I got another error

"Error while running event Will-o-the-Wisps_updated::on_tick (ID 0)
__Will-o-the-Wisps_updated__/libs/zones.lua:75: attempt to call field 'get_pollution' (a nil value)"

I attached the files, hopefully

Edit: I made a .rar instead of .zip because I didnt really pay attention to that, I hope thats not a problem

[mk-fg]

uploaded save and modlist

looks like it worked, this time I got another error

"Error while running event Will-o-the-Wisps_updated::on_tick (ID 0)
__Will-o-the-Wisps_updated__/libs/zones.lua:75: attempt to call field 'get_pollution' (a nil value)"

Yeah, this is totally normal after the command, as this quick-n-dirty data-mangling for export is destructive.
Thanks a lot for sticking with the issue, would almost be a shame if ends up being just the thing I suspected in the second reply, which it seem to be so far.
Will check/confirm, report back and upload fixed version soon.

[mk-fg]

Edit: I made a .rar instead of .zip because I didnt really pay attention to that, I hope thats not a problem

Definitely not, besides who doesn't have 7zip these days which opens any archive format ayway

[Bloodymight]

I did the command on both save games and got another error this time

"Error while running event Will-o-the-Wisps_updated::on_tick (ID 0)
__Will-o-the-Wisps_updated__/libs/zones.lua:150: attempt to index local 'chunk' (a nil value)"

pretty close to the error which started this all, second one is something something pollution for some reason, and this one(the third) is about chunks again with another lua(attached the files again)

[mk-fg]

I did the command on both save games and got another error this time

"Error while running event Will-o-the-Wisps_updated::on_tick (ID 0)
__Will-o-the-Wisps_updated__/libs/zones.lua:150: attempt to index local 'chunk' (a nil value)"

pretty close to the error which started this all, second one is something something pollution for some reason, and this one(the third) is about chunks again with another lua(attached the files again)

It's also due to export, same as above - mod will inevitably crash after that console command.
(as that export overrides all big-int keys with json-friendly strings, any lookup later will fail, and pollution thing above is due to it replacing chunk.surface userdata reference with "s.1" string representation)

[mk-fg]

Oh dear, I just realized that I don't actually have 16G of RAM on my 6-year old machine while trying to load your set of mods and save... will see how it goes

[Bloodymight]

I think the biggest RAM eater is Alien Biomes

didn't really pay attention to that ._.

you could remove "lightorio", "minime", "Bot-Servicing", "actual craft time" and "Alien Biomes" since they're all later additions

[mk-fg]

It's definitely something with a ton of sprites - when loading game, it passes lua init with something like 3-4G iirc, but when it gets to loading sprites, quickly goes over 11G and segfaults due to lack of memory.
No worries though, wisps.rar is all I need from there, and aside from definitely having that mismatch issue I mentioned earlier, seem to have two other minor bugs (non-critical - some stuff gets duplicated needlessly, probably due to those chunk replacements), which I'll also look into.