Saving is highly inefficient wrt circular references

[emptyrivers]

Demonstration here: https://drive.google.com/open?id=13IZTW ... bypEkKoH_V
To reproduce, simply install and enable the mod linked above, and note that it takes a good deal longer to complete the save process than necessary, and in the process consumes an enormous amount of ram, and the save file itself is quite large, too. In extreme circumstances with 100 thousand or more circular references (yes, before you ask, I actually do have a use case for something that large), this can create a save file 10-100 times larger than the same file with the same tables, but with the circular references removed. Now, I can work around this, by only creating those references locally and not actually saving them, but I thought I would let you know that there is something very inefficient in the serialization process.

[Rseding91]

Thanks for the report. The issue comes from Lua using immutable strings and how it handles concatenation: the circular references generate a bunch of concatenated strings where the non-circular ones don't. There's not much I can do about that since it's the internals of Lua and the Serpent library used to serialize the Lua state that causes the problem.

Simply put: you won't get good performance when you have that much data you're trying to store.

[emptyrivers]

Yes, I figured it would be something like that, given Lua's insistence on internalized strings with no buffer. Restructuring the saved data to contain only enough information to reconstruct a local structure which has the proper references gave a marked improvement in performance during the save process, for the price of essentially zero increase in loading time. With the mod payload I'm using to test my algorithms, I have approximately 50 thousand circular references which are generated. Saving them directly (in a brand new game, mind) takes approximately two minutes, consumes 15 gigabytes of ram at peak (it would probably consume more if I had more than 16 gigabytes available), and generates a save file which is ~200 megabytes in size. Saving only the information needed to construct the references gives a near instant save time, with no perceptible spike in ram usage, and a 2 megabyte save file.

The moral of the story for anyone who happens to look at this post is that you should be careful of what you actually put in the global table.