Autosaving while in combat is DUMB

[garath]

I'm sure this has been suggested before. Please don't save while in combat. In my honest opinion, saving in combat versus not saving in combat is one of those little check box items for games that are 'in development' versus 'production'. So, sure, with a 0.12 version number, it is something that probably won't be changed for a long time. But down the road I hope it is something that is seriously looked at. It is just so annoying to die in combat and then click to load the last autosave. Oh, but that one occurred *while* you were dying. So, you load the autosave. Oops. You died again. You aren't paying attention and click to load the autosave, and you die again. Oops. Oh, and of course, the main reason you died was because the game was autosaving while you were in combat!

[Muchaszewski]

Thats why autosave has 3 copy of itself, one is overwriten at the time, so simply go back to Load Game menu and choose one of the previuse one

[garath]

Thank you for the post.

Yes, I know about the three autosaves. That does not change at all anything I said in my post. Sure, if there was only a single Autosave, then it would be an unmitigated disaster to save in combat as you are dying. Still, I think it is a dumb idea to save while you are in combat. I do realize there may be times when you are "generally" in combat for hours on end. In such a scenario, it would be much worse to not save during those hours.

Even with breaks in combat, I'm not sure how you would code it. What's the definition of "not in combat"? And if the definition were too broad then how do you handle it? Do you force a save on the player in combat if it has been longer than <x> minutes? What is the definition of "not in combat"?

1. Player is <x> distance from a biter/spitter spawner?
2. Player is <x> distance from enemy?

I read the posts discussing why the save process takes so long--that it has to save a 100% accurate picture of the state of everything in the game. That being said, I still wish there were a magical way they could write the game so there wasn't the very noticeable pause during the auto saves. When you are in the "thick of battle" and have that very noticeable pause, it is quite distracting.

Hmm.... I wonder if the AutoSave settings are accessible for modding. I think it would be incredibly cool if I could turn on "Combat Mode" so that AutoSaving is disabled while I'm running around. I would want some very visible indicator on the screen so I know it is "Play at your own risk". It might be cool if a button on the mod would let me initiate a save using a default name with a button press while in combat.

This sounds workable if it is possible. Where is the list of LUA APIs?

[Rseding91]

I read the posts discussing why the save process takes so long--that it has to save a 100% accurate picture of the state of everything in the game. That being said, I still wish there were a magical way they could write the game so there wasn't the very noticeable pause during the auto saves.

If you find a way to perform magic do let us know.

[Zeblote]

I read the posts discussing why the save process takes so long--that it has to save a 100% accurate picture of the state of everything in the game. That being said, I still wish there were a magical way they could write the game so there wasn't the very noticeable pause during the auto saves.

If you find a way to perform magic do let us know.

Does it really take that long, to just copy the entire game state elsewhere? Surely it is much faster than processing it?

[Rseding91]

Does it really take that long, to just copy the entire game state elsewhere? Surely it is much faster than processing it?

Yes, 90% of the time is spent copying the game state. 10% is spent compressing it and writing it to disk.

[Zeblote]

Does it really take that long, to just copy the entire game state elsewhere? Surely it is much faster than processing it?

Yes, 90% of the time is spent copying the game state. 10% is spent compressing it and writing it to disk.

What does "copying the game state" mean here? Copying individual values around or just duplicating the entire block of memory at once? Would that be faster?

[bobingabout]

I honestly have had the game autosave before ticks before death, so you die pretty much instantly after loading the game again.

there's not a lot anyone can do about it, it saves the game on a certain tick multiplier, that's it.

All you can do is can load the previous one.

[Rseding91]

What does "copying the game state" mean here? Copying individual values around or just duplicating the entire block of memory at once? Would that be faster?

Copying individual chunks of memory out so the write-to-disk thread can compress and save that to the hard drive. You can't copy entire blocks of memory because the program is not in a contiguous block of memory. It's spread all over and arranged in all kinds of random patterns based off how the computer decided to allocate memory at the time. Also, you don't want to copy the entire state out because that means your save file would be the size of your save when loaded into RAM: 6.8 GB in my save file instance.

[Zeblote]

What does "copying the game state" mean here? Copying individual values around or just duplicating the entire block of memory at once? Would that be faster?

Copying individual chunks of memory out so the write-to-disk thread can compress and save that to the hard drive. You can't copy entire blocks of memory because the program is not in a contiguous block of memory. It's spread all over and arranged in all kinds of random patterns based off how the computer decided to allocate memory at the time. Also, you don't want to copy the entire state out because that means your save file would be the size of your save when loaded into RAM: 6.8 GB in my save file instance.

Well, saving that to disk makes no sense, I was more thinking duplicate the memory somewhere and then start a new thread to write and compress it like usual.

[ratchetfreak]

Or a better idea change allocation patterns/allcoators so that there is less randomness to the memory layout and perhaps allow some previously random bits to be copied out as a single large chunk.

[garath]

Or a better idea change allocation patterns/allcoators so that there is less randomness to the memory layout and perhaps allow some previously random bits to be copied out as a single large chunk.

Doing that would probably mean the ENTIRE game experience is sluggish and slow instead of just the Auto Saves.

[Zeblote]

Or a better idea change allocation patterns/allcoators so that there is less randomness to the memory layout and perhaps allow some previously random bits to be copied out as a single large chunk.

Doing that would probably mean the ENTIRE game experience is sluggish and slow instead of just the Auto Saves.

Not really, why do you think that? They're most likely already using custom allocators so there isn't a new allocation for every thing you create.

[ratchetfreak]

Doing that would probably mean the ENTIRE game experience is sluggish and slow instead of just the Auto Saves.

Not really, why do you think that? They're most likely already using custom allocators so there isn't a new allocation for every thing you create.

Exactly.

For example biters, each biter has a fixed sized struct that holds all the data it needs (position, what type of biter it is (large, small, medium, behemoth), a pointer to it's current path is, a pointer to it current target, ...). Lets call that BiterEntity.

In the naive method each time a biter spawns the game code will do "new BiterEntity(...)" which will invoke malloc and put the struct in a random spot in memory after that it will put the pointer in a list of all biters to keep track of it. When a biter gets killed the pointer is removed from the list and "delete killedBiter;" will get executed. This will then free the memory.

A much better method is to create a linked list with BiterEntity[CHUNK_SIZE] so that each CHUNK_SIZE biters are allocated at a time and when a biter spawns the code will look through one of the lists and select a free slot to store the biter's data. When the biter dies the slot can be marked free again. You can speed up finding empty slots by using a freelist. When the code has to do an operation on all biters it can then just run over all the lists and do the operation on each alive biter in it.

With this second method the saving code can blindly copy the BiterEntity[CHUNK_SIZE] arrays and know that all biters are in there even if some slots are unused but you can filter those out in the compression step. If you add a map of original-pointer-to-copied-pointer map then things like the biter groups can find the biters they contain.

[Rseding91]

Doing that would probably mean the ENTIRE game experience is sluggish and slow instead of just the Auto Saves.

Not really, why do you think that? They're most likely already using custom allocators so there isn't a new allocation for every thing you create.

Exactly.

For example biters, each biter has a fixed sized struct that holds all the data it needs (position, what type of biter it is (large, small, medium, behemoth), a pointer to it's current path is, a pointer to it current target, ...). Lets call that BiterEntity.

In the naive method each time a biter spawns the game code will do "new BiterEntity(...)" which will invoke malloc and put the struct in a random spot in memory after that it will put the pointer in a list of all biters to keep track of it. When a biter gets killed the pointer is removed from the list and "delete killedBiter;" will get executed. This will then free the memory.

A much better method is to create a linked list with BiterEntity[CHUNK_SIZE] so that each CHUNK_SIZE biters are allocated at a time and when a biter spawns the code will look through one of the lists and select a free slot to store the biter's data. When the biter dies the slot can be marked free again. You can speed up finding empty slots by using a freelist. When the code has to do an operation on all biters it can then just run over all the lists and do the operation on each alive biter in it.

With this second method the saving code can blindly copy the BiterEntity[CHUNK_SIZE] arrays and know that all biters are in there even if some slots are unused but you can filter those out in the compression step. If you add a map of original-pointer-to-copied-pointer map then things like the biter groups can find the biters they contain.

And then your memory use is 50 GB instead of 6GB. That doesn't work due to the dynamic nature of Factorio. The only place we have a custom allocator is sprite drawing.

[ssilk]

Hm. Quite interesting.

I think the only way to improve save speed is implementing this: viewtopic.php?f=6&t=25640#p161886

I think for v0.14 the team should try to improve the save-speed.

[garath]

It is very easy for me as a novice to say something dumb like: Just use more cores or make it more multi-threaded. But reading what one of the developers said above:

1. Game code memory allocation places objects all over the place.
2. Saving the game requires a significant amount of time to consolidate all this data in one location prior to compressing it and then saving it.
3. In order to "copy" the data, you cannot be simultaneously change that data.
4. As a result, the game must pause while anything at all is happening to the game data.

So, speeding up the save game process would not be a change *only* to the save game operation. Rather, the developers would have to rewrite the entire game to create a way of storing a second copy of all the game data--which presumably means the entire game takes essentially DOUBLE the memory and some increase in the processing power. Once there are two copies of all the data that are updated at all times, then you get super fast saves simply by stopping updates to one of the copies of the data.

I dislike the pause when you save the game. But I have a TON more stuff I want the developers to work on instead of rewriting the entire memory allocation subsystem of the game...

For me, I think I would ask the developers to expose to LUA the ability to turn on and off autosaves. Then, I could turn on a "combat" mode with autosaves disabled and run around pause free to my heart's content.

However, I can *definitely* see a number of downsides to such a mode. The first time someone's computer crashes after doing "combat" mode for two hours and loses two hours of game play, they won't blame the author of the mod, they will blame the developers of the game.

[Rseding91]

It takes less than 5 seconds to turn off autosaves.

Hit escape, press options, press the option sub button with the auto save option, slide the slider all the way to the right, click ok. Done.

[Muchaszewski]

It is very easy for me as a novice to say something dumb like: Just use more cores or make it more multi-threaded.

I would love if could use multiple threads to save but, world is cruel place and you cannot do that Operation that save stuff to disk is locking ALL threads. So for every chunk of bytes you write you have to wait unlill that finishes. So using multiple thread is no go.
But there is another problem Lets say we have somewhat cloned game state in memory, and we try to write this to disk on second Thread. As mentioned above saving data to disk locks All threads, so even doing this performance efficient you would have your fps dropped for (current time * magic number that will not make game run at 1fps and keep it at 30 or 60) or they would be doing saving stuff for more that autosave interval is set

But of course you can speed things up like, check what changed between saves and only overrites changed parts. But this would not speed things by mutch on bases that are 200x200 chunks of pure factory

[garath]

It takes less than 5 seconds to turn off autosaves.

Hit escape, press options, press the option sub button with the auto save option, slide the slider all the way to the right, click ok. Done.

And then when I forget to turn it back on again after I exit combat. No thanks. Terrible suggestion.