Friday Facts #270 - HR Substation & Save/Load overview

Regular reports on Factorio development.
Dixi
Inserter
Inserter
Posts: 31
Joined: Sat Nov 04, 2017 1:47 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by Dixi » Sun Nov 25, 2018 10:57 am

I always thought that "Entities", if biters fit in there, takes much more space in a save file.
Because, when everything is calm, my save file is around 40Mb. If I start to expand and clear a new map piece, even not so big one, or do some manual artillery shooting (both actions leads to intensive biters attack waves) save easily can grow to 60+Mb. If I delete all alive biters by console command save instantly shrinks to 25Mb. ( 15+20 )/ 60 is over 50% of save file!
I've tried to complain on forums before, that biters, while been just a side factor in a game about factories take too much save file space. I complain, because it's inconvenient to have very big save, because it freezes the game every several minutes for autosave. Maybe it's possible to review possibility to copy data, then save without game pause? I remember, it was rejected for some reason, but I still think it could be an important improvement to gameplay smoothness.

ske
Filter Inserter
Filter Inserter
Posts: 353
Joined: Sat Oct 17, 2015 8:00 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske » Sun Nov 25, 2018 2:25 pm

Dixi wrote:
Sun Nov 25, 2018 10:57 am
it was rejected for some reason
I'd like to know that reason.

Most likely running two server/simulation processes on most processors reduces maximum UPS considerably since factorio performance is said to be memory bound. Also doubling the background logic could be somewhat complicate things. But I think now it would be worth it to support massive multiplayer games without those pesky pauses every 10 seconds when somebody tries to (re)join.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 2604
Joined: Tue Jul 12, 2016 9:03 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by eradicator » Sun Nov 25, 2018 2:44 pm

ske wrote:
Sun Nov 25, 2018 2:25 pm
Dixi wrote:
Sun Nov 25, 2018 10:57 am
it was rejected for some reason
I'd like to know that reason.

Most likely running two server/simulation processes on most processors reduces maximum UPS considerably since factorio performance is said to be memory bound. Also doubling the background logic could be somewhat complicate things. But I think now it would be worth it to support massive multiplayer games without those pesky pauses every 10 seconds when somebody tries to (re)join.
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
nonblock.png (38.54 KiB) Viewed 462 times
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English

factoriouzr
Filter Inserter
Filter Inserter
Posts: 547
Joined: Sat Jun 06, 2015 2:23 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by factoriouzr » Sun Nov 25, 2018 2:55 pm

So can we have a release of 0.17 before the holidays this year? It's been a ridiculously long time since the last Factorio release; almost a year in fact. 0.16.0 was released Dec 13 2017. A lot of us will have some time to play over the holidays and a release of 0.17 before the holidays would be great.

Wube Software said that you would release more frequently, however this definitely doesn't seem to be the case.

I appreciate all the work you are doing, but what goes into each release is arbitrary. You can release the features you have ready now and have time to fix critical bugs before holiday vacation. Then you can fix the rest of the issues after holiday vacations (I don't expect people to not take time off during the holidays because of a release). Then you can release the rest of the planned feature for 0.17 in January as you mentioned you wanted to.

pleegwat
Fast Inserter
Fast Inserter
Posts: 107
Joined: Fri May 19, 2017 7:31 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by pleegwat » Sun Nov 25, 2018 3:54 pm

ske wrote:
Sun Nov 25, 2018 2:25 pm
Dixi wrote:
Sun Nov 25, 2018 10:57 am
it was rejected for some reason
I'd like to know that reason.

Most likely running two server/simulation processes on most processors reduces maximum UPS considerably since factorio performance is said to be memory bound. Also doubling the background logic could be somewhat complicate things. But I think now it would be worth it to support massive multiplayer games without those pesky pauses every 10 seconds when somebody tries to (re)join.
Not memory bandwidth, memory latency. So that's not the immediate problem.

However it's built single-threaded; an action taken earlier in the tick (EG an inserter picking up an item from the ground) can cause an action later in the same tick (EG another inserter attempting to pick up the same item) to fail.

And I think the pauses are caused by the server saving current game state, so I don't see how multithreading would mitigate that.

User avatar
5thHorseman
Filter Inserter
Filter Inserter
Posts: 478
Joined: Fri Jun 10, 2016 11:21 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by 5thHorseman » Sun Nov 25, 2018 4:36 pm

factoriouzr wrote:
Sun Nov 25, 2018 2:55 pm
So can we have a release of 0.17 before the holidays this year?
They answered that already.

No.
"So you completed the game with a spaghetti factory? Well I hand crafted a rocket and threw it into space with my bare hands!"

ske
Filter Inserter
Filter Inserter
Posts: 353
Joined: Sat Oct 17, 2015 8:00 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske » Sun Nov 25, 2018 6:05 pm

eradicator wrote:
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.

posila
Factorio Staff
Factorio Staff
Posts: 2755
Joined: Thu Jun 11, 2015 1:35 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila » Sun Nov 25, 2018 6:14 pm

ske wrote:
Sun Nov 25, 2018 6:05 pm
eradicator wrote:
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.
It's forking the factorio process, that's why it doesn't work on Windows (no official fork on Windows, and we didn't managed to make it work with the un-official undocumented one)

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 2604
Joined: Tue Jul 12, 2016 9:03 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by eradicator » Sun Nov 25, 2018 6:36 pm

posila wrote:
Sun Nov 25, 2018 6:14 pm
ske wrote:
Sun Nov 25, 2018 6:05 pm
eradicator wrote:
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.
It's forking the factorio process, that's why it doesn't work on Windows (no official fork on Windows, and we didn't managed to make it work with the un-official undocumented one)
Wouldn't hurt if there was a short tooltip and description in the config that says "Only on Mac/Unix.". Nothing more confusing than a checkbox that doesn't do anything :p. Thanks for the explanation ofc though.
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English

posila
Factorio Staff
Factorio Staff
Posts: 2755
Joined: Thu Jun 11, 2015 1:35 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila » Sun Nov 25, 2018 6:45 pm

Honestly, I thought the checkbox doesn't show up on Windows.

ske
Filter Inserter
Filter Inserter
Posts: 353
Joined: Sat Oct 17, 2015 8:00 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske » Sun Nov 25, 2018 7:29 pm

posila wrote:
Sun Nov 25, 2018 6:14 pm
ske wrote:
Sun Nov 25, 2018 6:05 pm
eradicator wrote:
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.
It's forking the factorio process, that's why it doesn't work on Windows (no official fork on Windows, and we didn't managed to make it work with the un-official undocumented one)
I see. Is copy-on-write quick enough to avoid significant lag when allocating and copying all that memory? (I assume that all pages are touched by some write access.)

If it is sufficient, could you rewrite the game logic in such a way, that it creates a separate data structure for each frame in a new memory area?

posila
Factorio Staff
Factorio Staff
Posts: 2755
Joined: Thu Jun 11, 2015 1:35 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila » Sun Nov 25, 2018 7:33 pm

ske wrote:
Sun Nov 25, 2018 7:29 pm
I see. Is copy-on-write quick enough to avoid significant lag when allocating and copying all that memory? (I assume that all pages are touched by some write access.)
I heard, the game stutters while saving kicks in.
ske wrote:
Sun Nov 25, 2018 7:29 pm
If it is sufficient, could you rewrite the game logic in such a way, that it creates a separate data structure for each frame in a new memory area?
So 0.17 sometime in 2020? :)

ske
Filter Inserter
Filter Inserter
Posts: 353
Joined: Sat Oct 17, 2015 8:00 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske » Sun Nov 25, 2018 7:41 pm

posila wrote:
Sun Nov 25, 2018 7:33 pm
ske wrote:
Sun Nov 25, 2018 7:29 pm
If it is sufficient, could you rewrite the game logic in such a way, that it creates a separate data structure for each frame in a new memory area?
So 0.17 sometime in 2020? :)
:o I hope I didn't give you any new ideas. I mean I think it could probably be done. What the operating system is doing during COW is not much more than a simple memory copy. If you do that manually you should be more efficient. You'd just have to use an allocator for everything relevant for the game state and then just copy all memory areas allocated by that allocator. That's a few memory areas. Sure you'd need to fix the pointers but that could be done by some lookup-tables or a simple offset if you only have one container. Sure, you'd need to apply that offset to all internal pointers but that could be covered by using templates that optimize away such an offset for the normal operation. After copying you'd only need to serialize the whole package and discard it.

FasterJump
Long Handed Inserter
Long Handed Inserter
Posts: 98
Joined: Sat Jul 09, 2016 11:43 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by FasterJump » Mon Nov 26, 2018 5:29 am

I know that chunks that are far away (and no polluted) are inactive (frozen) to save fps.

Would it be possible to delete specific chunks automatically, to save map size? Like a chunk where no nest was destroyed and no ore mined. The minimap stays updated, but the chunk is deleted as soon as the fog war comes back.

User avatar
bobingabout
Smart Inserter
Smart Inserter
Posts: 6050
Joined: Fri May 09, 2014 1:01 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by bobingabout » Mon Nov 26, 2018 9:19 am

Rseding91 wrote:
Fri Nov 23, 2018 7:19 pm
bobingabout wrote:
Fri Nov 23, 2018 6:58 pm
I don't really like the new substation graphic, but I'm sure it will grow on me.

The save file is more than a quarter decoratives? Most games don't even save those, just re-generate them on load, or when you re-enter an unloaded area. I don't see why you can't do the same. Is there any reason these need to remain perfectly in sync?
Determinism.
do "Decoratives" really matter though? I remember playing Terraria, and moving around, and when I come back the decoratives have changed their appearance, because they aren't actually saved. So two players in a multiplayer can be looking at the same spot, and see different things because the decoratives are different, it really doesn't matter.

So, my question is basically... is it REALLY important enough to save? would it break the game if they weren't perfectly in sync? 30% of a save seems like a lot of space to dedicate to something that most people just see as annoying (because it hides more important things from view, or look ugly EG grass coming through your concrete floor)

Don't get me wrong, I don't think these "annoying" doodads should be removed, they add character, I just don't see why they need to be synced in the name of determinism.
IronCartographer wrote:
Sat Nov 24, 2018 12:01 pm
Regarding the HR graphic update and complaints that it looks too small / noisy / similar to a power pole...
Image

I agree that it no longer looks like something I would call a substation, though the new graphic is nice in its own right. What if the mismatch can be resolved while using the new art?

How about a new name: Electric Transceiver! It fits because of the huge wireless coverage area, both sending and receiving power.

Parallel discussion on Reddit: https://www.reddit.com/r/factorio/comme ... ?context=2
Also it would seem the diagonal pole rotations are backwards.
Creator of Bob's mods. Expanding your gameplay since version 0.9.8.

mrvn
Smart Inserter
Smart Inserter
Posts: 2579
Joined: Mon Sep 05, 2016 9:10 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by mrvn » Mon Nov 26, 2018 10:36 am

eradicator wrote:
Fri Nov 23, 2018 8:23 pm
Oktokolo wrote:
Fri Nov 23, 2018 7:44 pm
eradicator wrote:
Fri Nov 23, 2018 7:21 pm
For example concrete has a random chance to destroy decoratives. Could probably be made deterministic.
Random numbers are already deterministic in Factorio. They are based on synced state.
I doubt you understand what you just said there. The results of random numbers generated during runtime are part of the savegame in the form of decorative positions. Not saving decoratives would mean recalculating all random numbers from the start of the save (i.e. running a full replay before every load) or completely changing how concrete determines which decoratives are removed.
______
Koub wrote:
Fri Nov 23, 2018 8:05 pm
Most of the time, I wish they weren't there, but I can't take them off without a mod, which would in turn prevent me from earning achievements on Steam.
Steam "achievements" the greatest non-reason ever invented to play a game you like less instead of a game you like more :roll:.
______
I was never very fond of the current substation, always looked too...flat? modern? Didn't have that "steampunk" feel written all over it. Not that the new one couldn't need some more of that too.
No, you would use a separate random number generator for decorations so they wouldn't affect the numbers of anything else.

But why do decoration need to be random at all? Why not make them noise instead? The "drawback" would be that when you have e.g. a leached animal skull in the desert, change that to grass and back to desert then the leached animal skill would be back. Because the noise function says there should be one. Weather that is a plus or minus is up to you.

PS: When you landfill an area you kind of see the stripes of landfill you placed because going up and down look different. Play Seablock for a while and you will notice it.

weaknespase
Manual Inserter
Manual Inserter
Posts: 4
Joined: Sat Mar 24, 2018 8:19 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by weaknespase » Mon Nov 26, 2018 4:53 pm

Nonblocking saving? Yeah, forking won't work most probably, due to the mechanics of how forking works. The process that modifies dataset will get all the usual goodness - page faults, copying of memory, thousands of page allocations, randomized page table and a lot of extra cache misses - all the usual causes of generic performance hits. Guess which process that would be.

Copying entire serializable data in one go (without gathering it over entire memory) between game updates may be one of the most performant ways to solve this problem, but would require tricky memory allocation.

On the other note, Factorio has pretty nice deterministic game mechanics, is it utilized anywhere besides saves? For example, to defer (or join several into one) updates on anything that is outside of (everyone's) screen, for example?

N35t0r
Burner Inserter
Burner Inserter
Posts: 12
Joined: Sat Aug 23, 2014 9:29 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by N35t0r » Mon Nov 26, 2018 7:59 pm

bobingabout wrote:
Mon Nov 26, 2018 9:19 am
Also it would seem the diagonal pole rotations are backwards.
This. Literally unplayable ;-)

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 2604
Joined: Tue Jul 12, 2016 9:03 am

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by eradicator » Mon Nov 26, 2018 8:09 pm

bobingabout wrote:
Mon Nov 26, 2018 9:19 am
Rseding91 wrote:
Fri Nov 23, 2018 7:19 pm
Determinism.
Don't get me wrong, I don't think these "annoying" doodads should be removed, they add character, I just don't see why they need to be synced in the name of determinism.
Just to put this into perspective: Suddenly you are the one asking to remove engine features that other modders might be using to hack around limitations. Also i personally am quite fond of the idea that i see the same things that other people do. There's a certain mission in Dead Space 3 coop where one person sees something completely different, and i can tell you it doesn't feel that great when the other person thinks you're suddenly insane ;). It was funny afterwards, but i wouldn't want that as the default state.
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English

posila
Factorio Staff
Factorio Staff
Posts: 2755
Joined: Thu Jun 11, 2015 1:35 pm

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila » Mon Nov 26, 2018 8:17 pm

ske wrote:
Sun Nov 25, 2018 7:41 pm
:o I hope I didn't give you any new ideas. I mean I think it could probably be done. What the operating system is doing during COW is not much more than a simple memory copy. If you do that manually you should be more efficient. You'd just have to use an allocator for everything relevant for the game state and then just copy all memory areas allocated by that allocator. That's a few memory areas. Sure you'd need to fix the pointers but that could be done by some lookup-tables or a simple offset if you only have one container. Sure, you'd need to apply that offset to all internal pointers but that could be covered by using templates that optimize away such an offset for the normal operation. After copying you'd only need to serialize the whole package and discard it.
Yes, it could be done, in unreasonable amount of time :). I mean, we are not lacking ideas, we are lacking time dilation device; or time travel device to send kovarex message to 2012 about how big EnergyCraft will get and what things he should plan for better ... and that he should really make it 3D (like Starcraft 2, not first-person) :D

Post Reply

Return to “News”