[0.16.16] Tile prototype limit (255) too low

[Arcitos]

Dear devs,
i just installed the recently updated mod "Alien Biomes" which adds a bunch of new biome types. Together with Dectorio and Asphalt Roads (both are mods that add new tiles) the following error occured on startup:

"Error: Reached id limit for tile. The game can't contain more than 255 instances of this prototype due to hardcoded limits of the game engine"

This hardcoded limit was sufficiently large for most of the time. But, apparently, every hardcoded limit is determined to be exceeded at some point. Would you please consider to increase this limit up to something like 1023? I would appreciate that.

Regards
Arcitos

[posila]

This limit won't be ever increased. I am sorry.

[FuryoftheStars]

This limit won't be ever increased. I am sorry.

Well that's a bummer. I was hoping for it to be upped to. Would it take too much work to do or something?

[posila]

It would increase size of loaded map in RAM quite a lot, increase save size and reduce performance, which is not good tradeoff for adding possibility of more tile types, in my opinion.

[FuryoftheStars]

Understood, but this would only affect those of us using mods that add up like that, wouldn't it? It'd be a known trade off that we'd have to consider.

[posila]

Understood, but this would only affect those of us using mods that add up like that, wouldn't it? It'd be a known trade off that we'd have to consider.

Unfortunatelly not. We would have to increase tile ID type from one byte to two bytes, which would happen for everyone regardless of how many tile types has been loaded, and remove some optimizations which assume there won't ever be tile ID higher than 255.

[Arcitos]

Thanks for your fast reply, posila. This is indeed a real pity, but i fully understand your motivation in keeping the tile id limit at 255. I was just curious if lifting this limit would be feasible.

Regards
Arcitos

[FuryoftheStars]

Unfortunatelly not. We would have to increase tile ID type from one byte to two bytes, which would happen for everyone regardless of how many tile types has been loaded, and remove some optimizations which assume there won't ever be tile ID higher than 255.

Ok, yeah, sorry, I understood that and guess I didn't convey myself correctly or am not fully understanding it from your viewpoint.

I understand that increasing the ID limit will cause an additional byte of data to be stored for each tile and that this would affect everyone. But for those not loading mods that add tiles, shouldn't this only raise memory usage at 1 byte per vanilla tile? I don't know how many are currently in, but even if you were using the limit of 255, this should only raise memory usage by 255 bytes, right? And then the code you speak of that assumes tile IDs aren't greater than 255, couldn't that just be recoded for 1023 (rather than removed)? Sorry, while I do have some coding experience myself, it is fairly limited and obviously I have not seen the code you speak of.

Regardless, I do appreciate your taking the time to answer. I also hope that this could be something put on a list for reconsideration in the future as things change and maybe other optimizations are done.

Thank you.

[posila]

It would be 1 extra byte for each tile on map ... one chunk is 32x32 tiles, medium sized map has ~25000 chunks (about 5000x5000 uncovered tiles). Yes, I suppose it would be possible to keep optimizations if we limited max tile ID to 1023 instead of allowing whole range of 2 bytes.

Anyway 255 is very reasonable maximum number of tile types. Vanilla uses just 31, so mods can still add more than 200 tile types. Go complain to the mod creators to keep their usage of tiles reasonable

[FuryoftheStars]

It would be 1 extra byte for each tile on map

Ahhh, gotcha! Too bad there wasn't some way to optimize that!

Anyway 255 is very reasonable maximum number of tile types. Vanilla uses just 31, so mods can still add more than 200 tile types. Go complain to the mod creators to keep their usage of tiles reasonable

Yeah, in my case I'm using about 6 or 7 different mods that add and alter tiles, some of which want to add variations on the vanilla tiles (like Scorched Earth or Sparky's Smog), so they add up quick.

Thanks for the replies and explanations, though. If ever the opportunity arises, though, to somehow make this possible, it'd be great to see happen. Thank you!

[jinsediaoying]

It would be 1 extra byte for each tile on map ... one chunk is 32x32 tiles, medium sized map has ~25000 chunks (about 5000x5000 uncovered tiles). Yes, I suppose it would be possible to keep optimizations if we limited max tile ID to 1023 instead of allowing whole range of 2 bytes.

Anyway 255 is very reasonable maximum number of tile types. Vanilla uses just 31, so mods can still add more than 200 tile types. Go complain to the mod creators to keep their usage of tiles reasonable

Hi, here is an idea that may work.
In short, you create a "id-overflow-placeholder" tile.
And for each chunk, save an additional table of (tile-coord: long_tile_id) pairs.
Then when loading, whenever a placeholder id is detected, go to the long tile id table and search the long id to decide the tile.
For most part of the map, this long id table should be empty, so it won't affect save size much.
And you can also ask the modders to manually marks some of their tiles as "rare-use" for further optimization. (most tiles that require a manual placement is pretty rare).

This solution provides the possibility of infinite tile types, without affect save size much.

[pleegwat]

There's various ways in which you could compress the tile list. First that springs to my mind is to use different lookup tables for different chunks. This would probably need to be based on the biomes in each chunk, so it would not work if lots of tiles were added that aren't tied to a biome, and it would limit how many biomes can appear in a single chunk. This would not help at all with concrete/asphalt/etc mods, but it could potentially help with additional biome mods.

However, coming up with ideas on how to fix this is hardly the most difficult step to fixing it.

[jonharson]

err... dotnet number optimizations anyone? I assume the lowest random access level is by chunk? Use a value of 255 to mean there is another byte following for a tile using an extended id. infinite ids case solved...

[Rseding91]

err... dotnet number optimizations anyone? I assume the lowest random access level is by chunk? Use a value of 255 to mean there is another byte following for a tile using an extended id. infinite ids case solved...

That's not how compiled languages work

You can't do that with:

Code: Select all

std::array<32, std::array<32, TileID>>

Where the sizeof(...) that is 32 * 32 * 1. If TileID is > 1 byte that immediately doubles in size.

[jonharson]

"You can't do that..."

Sir you can do anything! pseudo slow buggy code...

data std::array<32, std::vector<byte>>

operator[](int x)
{
target = x % 32
index= 0
id = 0

for (std::vector<byte>::iterator it = data[int(x/32)].begin() ; it != data[int(x/32)].end(); ++it)
{
if index != target
{
if byte <> 255
index +=1
else
continue
}
else
if byte = 255
id += 254
continue
else
id += byte
}

return id
}

now to edit it you may need to shift part of vector ...
operator =() {}

}

[eradicator]

If the world is basically a bitmap of tiles, would it be possible to use image specific optimization techniques like huffman coding (possibly combined with a minecraft-style "region" approach) to reduce the size or would that be too slow? Opening up even larger images in a good imaging program is usually quite fast and not very ram intensive (20M for a 4k² png).

Just for reference: What is the amount of ram currently required by factorio to load a vanilla 10k² map?

(Theoretical question as this would likely cause major code changes?)

[betrok]

It would be 1 extra byte for each tile on map ... one chunk is 32x32 tiles, medium sized map has ~25000 chunks (about 5000x5000 uncovered tiles)

So, extra 24Mb? It does not look too big compared with current usage.

[jonharson]

It would be 1 extra byte for each tile on map ... one chunk is 32x32 tiles, medium sized map has ~25000 chunks (about 5000x5000 uncovered tiles)

So, extra 24Mb? It does not look too big compared with current usage.

Update to 0.16, load a "few" mods, start a new game, plop a radar, memory compression kicks in, 10 gb commit, Batman!
https://i.imgur.com/uDbJyzA.png

[Rseding91]

"You can't do that..."

Sir you can do anything! pseudo slow buggy code...

data std::array<32, std::vector<byte>>

operator[](int x)
{
target = x % 32
index= 0
id = 0

for (std::vector<byte>::iterator it = data[int(x/32)].begin() ; it != data[int(x/32)].end(); ++it)
{
if index != target
{
if byte <> 255
index +=1
else
continue
}
else
if byte = 255
id += 254
continue
else
id += byte
}

return id
}

now to edit it you may need to shift part of vector ...
operator =() {}

}

That changes the memory from one contiguous section into multiple fragmented heap-allocated blocks. It also removes the ability to have O(1) index time.

Any change can't negatively impact the base game where it would never use that feature.

[jonharson]

That changes the memory from one contiguous section into multiple fragmented heap-allocated blocks. It also removes the ability to have O(1) index time.

Any change can't negatively impact the base game where it would never use that feature.

Yes I probably would fragment the shit out of it... why do I need 6 gb of crap loader for this "small" map? cant you bake the chunks map and dynamically load what's active?