[0.17.42] Save of large map is extremely slow

[AngeloidBeta]

My current favorite map covers an area of 250x250 chunks (8000x8000 tiles), with the map generator set to produce extremely rich resources and equally rich biter bases, and all 62,500 chunks generated (some via exploration but mostly by console script). When dealing with this map, saving - in particular the autosave - is extremely slow. The very best case I've experienced was a save taking 15 seconds; the average is more like 40, with the slowest I've seen taking something on the order of three minutes. In most cases, including both the aforementioned extremes, my computer (a 15" 2015 MacBook Pro) was under approximately the same CPU, GPU, and I/O load. The amount of change in the map between any two saves doesn't seem to bear any strong correlation to how long the save takes. There are ten mods (not including base) active on the map, including my own unreleased mod (containing a great many prototypes but almost no control code).

I realize this is an enormous map and thus a huge amount of data to manage, but I still wonder if the average case is expected to be this bad. Please let me know what information I can provide to help investigating this (if any)- I've held off on attaching the (almost 200 MB) save (and a more complete system profile) in case this is an instance of "yes, this is what's expected for this situation", which would certainly be understandable.

[arrow in my gluteus]

The amount of change in the map between any two saves doesn't seem to bear any strong correlation to how long the save takes.

That's because saving always writes the whole map to disk, not just the difference.

[Rseding91]

62,500 chunks really isn't that big. I have save files with 95,000 chunks and it takes 5~ seconds for an autosave.

Can you please post the save file and your mods folder (if using mods)?

[AngeloidBeta]

62,500 chunks really isn't that big. I have save files with 95,000 chunks and it takes 5~ seconds for an autosave.

Can you please post the save file and your mods folder (if using mods)?

I tried attaching the files to this post, but your forum server is having some difficulty:

Code: Select all

<b>[phpBB Debug] PHP Warning</b>: in file <b>[ROOT]/phpbb/plupload/plupload.php</b> on line <b>329</b>: <b>move_uploaded_file(./files/plupload/cfd5f8e59121bb005a34a00a22e0e3b4_804532f4f32b39511b5a46ddd05f22f2): failed to open stream: Permission denied</b><br />
<b>[phpBB Debug] PHP Warning</b>: in file <b>[ROOT]/phpbb/plupload/plupload.php</b> on line <b>329</b>: <b>move_uploaded_file(): Unable to move '/tmp/phpzknBLx' to './files/plupload/cfd5f8e59121bb005a34a00a22e0e3b4_804532f4f32b39511b5a46ddd05f22f2'</b><br />
{"jsonrpc":"2.0","id":"id","error":{"code":103,"message":"Failed to move uploaded file."}}

Never a dull moment in PHP upload handling. I especially love how phpBB doesn't bother suppressing report_errors and ends up with a broken JSON response the client side can't parse anyway XD;;; Let me know when that's fixed or if there's an alternative method.

The amount of change in the map between any two saves doesn't seem to bear any strong correlation to how long the save takes.

That's because saving always writes the whole map to disk, not just the difference.

Yes, that is indeed the very obvious conclusion.

[Rseding91]

You'll need to upload it somewhere like dropbox/google drive/one drive.

[AngeloidBeta]

You'll need to upload it somewhere like dropbox/google drive/one drive.

Let me know if these links don't work:

https://www.icloud.com/iclouddrive/0cN_ ... OnAXA#mods
https://www.icloud.com/iclouddrive/0uOW ... dugUQ#save

[Rseding91]

Saving your map takes 8.4~ seconds for me.

Of that time, half is spent saving chunks/resources and half is spent compressing the resulting 400 MB of save data and writing it to disk. Both happen in parallel.

If it's taking considerably longer for you then your CPU and or hard drive are the limiting factors.

[AngeloidBeta]

Saving your map takes 8.4~ seconds for me.

Of that time, half is spent saving chunks/resources and half is spent compressing the resulting 400 MB of save data and writing it to disk. Both happen in parallel.

If it's taking considerably longer for you then your CPU and or hard drive are the limiting factors.

I tend to doubt my SSD is the limiting factor, but I've noticed Factorio rarely seems to use more than 1-2 CPU cores at a time. The process' CPU usage maxes out at two cores during saving, according to a test I just did. This machine has an 8-core 2.5GHz Intel Core i7 with turbo boost to 3.5GHz (which does happen during Factorio's save process). The write rate to the SSD maxes out at ~15MiB/sec, and the save is currently taking ~22 seconds with the game paused (autosaves that interrupt the game usually take a bit longer before the progress bar starts filling); the majority of the delay appears to be before the writes to disk start, according to my measurements. The SSD is the 512GB Apple SSD which shipped with this MBP. If this matches your expectations for this CPU and SSD, then that's pretty much what I figured anyway But if not, please let me know if there's more I can do to help you nail down whatever the issue may be.

[Rseding91]

I just remembered that the files you gave me all had .DS_STORE folders in them + you uploaded them through apple drive. Are you on a an apple laptop?

If so, that could explain your problem. Try bringing up a temperature monitoring software and see what the CPU temperatures are when autosave is running. I'm wondering if it's thermal throttling.

From what you describe your CPU is the limiting factor (at the moment). There's nothing to be done on our end since as I said it only takes 8 seconds for me to save that map.

[AngeloidBeta]

I just remembered that the files you gave me all had .DS_STORE folders in them + you uploaded them through apple drive. Are you on a an apple laptop?

If so, that could explain your problem. Try bringing up a temperature monitoring software and see what the CPU temperatures are when autosave is running. I'm wondering if it's thermal throttling.

From what you describe your CPU is the limiting factor (at the moment). There's nothing to be done on our end since as I said it only takes 8 seconds for me to save that map.

Er... I've mentioned this being a MacBook Pro in almost every single post in this thread so far In any event, it doesn't seem to be thermal throttling. After taking a spindump during the save process, it looks like the actual bottleneck here is memory usage; the kernel is spending a non-trivial amount of time blocking in the VM subsystem. This machine has 16GB of RAM, of which factorio's usage is averaging about 12GB - which when you consider the kernel's wired memory usage and the presence of other running apps, is enough to overrun physical memory by quite a bit. This also explains the variance in timing I've been seeing. Apparently at this map size (and/or the contents of the map) the working set for iterating the whole thing for saving is just too big for the amount of RAM you can fit in a laptop and still have other things running at the same time. I verified that quitting all other apps on the system (which were using no meaningful CPU time but significant amounts of RAM) dramatically improves save performance.

I suppose I could ask if there's a way you can reduce the memory usage, but since it's a matter of swapping in data already in memory, you'd have to reduce it for the entire runtime map state for the difference to matter to this problem - just using less at a time during save wouldn't change anything. That being said, given that you said 62,500 chunks isn't really that big, is 12GB of ("real") memory usage a reasonable expectation? Obviously under normal circumstances it's not a problem since you never keep all that in active memory at once, but the save process touches it all in sequence, effectively forcing a total cache flush all the way down to at least main memory if not also the VM backing store (depending on the availability of physical RAM and the semantics of the running kernel's dynamic pager - and it's no secret that xnu's pager has never been the most effective at handling this kind of VM load). Tinkering with my graphics settings a bit to reduce the overall memory usage also helped a bit, but 14 seconds (and that with some of my critical apps not running) is still a fairly long wait. I realize I'm not asking for something even remotely small here (by now it's probably obvious I have experience of my own in this arena!), but I do at least hope the info helps you out in some way.