How to store 4 trillion square tiles of map in just 8 GB RAM

[Jupiter]

So apparently the max map size is 2000km by 2000km. 1km = actually 1000 tiles. So that is 2M * 2M tiles = 4000000000000 tiles

4*10^12 tiles!!

I have just 8 gigs of ram. That means 8 * 10^9 * 8 bits = 64 * 10^9 bits. That doesn't even come close to 4*10^12 bits you'll need if you store every tile using just one bit.

So, does anyone have an idea how this is stored in ram in such a way that it is accessible in realtime?

[m44v]

maps never get that big.

[Zeblote]

So apparently the max map size is 2000km by 2000km. 1km = actually 1000 tiles. So that is 2M * 2M tiles = 4000000000000 tiles

4*10^12 tiles!!

I have just 8 gigs of ram. That means 8 * 10^9 * 8 bits = 64 * 10^9 bits. That doesn't even come close to 4*10^12 bits you'll need if you store every tile using just one bit.

So, does anyone have an idea how this is stored in ram in such a way that it is accessible in realtime?

Only revealed chunks are stored in memory. You can't actually uncover the entire map.

[Jupiter]

So apparently the max map size is 2000km by 2000km. 1km = actually 1000 tiles. So that is 2M * 2M tiles = 4000000000000 tiles

4*10^12 tiles!!

I have just 8 gigs of ram. That means 8 * 10^9 * 8 bits = 64 * 10^9 bits. That doesn't even come close to 4*10^12 bits you'll need if you store every tile using just one bit.

So, does anyone have an idea how this is stored in ram in such a way that it is accessible in realtime?

Only revealed chunks are stored in memory. You can't actually uncover the entire map.

What if I tried... would I get an 'out of memory' error or something? Besides, they have set the limits to 2000x2000km. They just might as well have set that to something lower. They didn't

[Zeblote]

I'm assuming the limit is 2000km as that's the possible range for their fixed point positions, or something similar.

[Aatch]

They likely use floating-point numbers for positions in the game (because that's how you generally do this stuff). And single-precision floating point numbers have about 7 digits of precision. 2,000km is 2,000,000m, which is 7 digits of precision there. Assuming that 1 tile = 1m, then that gives you basically no precision loss even at the extremes of the map. It also nicely allows positions to fit into a 32-bit signed integer with millimetre accuracy (a 32-bit signed integer has a range of approximately +/-2,000,000,000).

[Koub]

That's the kind of "only limited by your hardware" thing The game engine can handle a 2Mmx2Mm map, provided you run it with 1 TB of RAM (500 gigs may even be enough)

[Tnarg]

I don't know, maybe it would crash maybe it wont. If it does not crash this is how it is likely to work: The world as you probably know is split up into chucks. All these chucks are saved to your hard disk. If they are using compression maybe they get 1 bit per tile, maybe it's 4 bytes. Lets be kind and say 4 bits per tile that means it takes up 200gb hard disk space. if you don't have this you will get an error. What the game might do (and what I know minecraft does) is only load some chunks. What chunks it loads I don't know maybe just the chunks near the player maybe it will load any chunks that have has a player building on any only update a few of those chunks so that enemy from the player will not move.

[PKing Zombie Spy]

So, does anyone have an idea how this is stored in ram in such a way that it is accessible in realtime?

This prior post details some aspects of the world generation, as does this wiki page. It is procedurally generated using Perlin noise. (In principle they could have used any parameterizable locally continuous function, but Perlin is a perfectly good choice.)

Now, still, despite this, the trouble is that parts you've "seen" are stored, which is why, if you save, then reveal a bunch of map using the console, then save again, the second save will be much larger. (In principle I guess, if they committed to never changing the map generation algo, or at least retained the code for backward compatibility, they could have conceivably just left everything purely procedural, and just stored the diff relative to the procedurally generated stuff, but this is not practical for various reasons.) So everything you've seen they store.

Then, one thing preventing you from "seeing" too much are the presence of biter nests. You know those nests with a hundred spawners, and hundreds of worms, where you eventually say, "screw it, not worth it" and go back to your base to build your factory and what have you? Biters aren't just protecting their planet, they're also protecting your computer!

But there is one more thing preventing you from seeing too much. Let's imagine you have no water, and no biters. I figure that with six exoskeletons (powered by two fusion reactors) I can cover about 30-40 squares per second, on unpaved terrain. The player's "view" appears to be 5 by 5 chunks (chunks being the 32x32 squares), so let's figure that in best case by running full-out I can reveal perhaps 5 chunks a second. (This is admittedly totally a SWAG, but not a terrible one I hope. ) 2M by 2M is 62.5k by 62.5k chunks, or roughly 3.9 billion chunks. So at 5 chunks per second, it would take you about 780 million seconds, or about ~25 years, to explore this whole area, without using the console or something.

Edit: Off by a factor of 100, somehow I miscalculated 62.5k * 62.5k as 390 billion not 3.9 billion. So not 2500 years but 25 years. Still a long time. Anyway still I wouldn't worry about it.

[ChurchOrganist]

So, does anyone have an idea how this is stored in ram in such a way that it is accessible in realtime?

Simple - you use virtual memory ie the swap drive.

Way way back in the mists of time when a PC had 640K hardware RAM that's how programmers did everything

Of course processors weren't fast enough yet in those days for it to be glaringly obvious that data was being pulled from the hard drive and not RAM.

In fact on the Acorn Electron I first learned to program on, I was using a floppy disk drive for virtual RAM - a revolution at the time as the basic means of data storage on that machine was cassette tape. Ah the heady days of going to make lunch while your game loaded

[Zeblote]

They likely use floating-point numbers for positions in the game (because that's how you generally do this stuff). And single-precision floating point numbers have about 7 digits of precision. 2,000km is 2,000,000m, which is 7 digits of precision there. Assuming that 1 tile = 1m, then that gives you basically no precision loss even at the extremes of the map. It also nicely allows positions to fit into a 32-bit signed integer with millimetre accuracy (a 32-bit signed integer has a range of approximately +/-2,000,000,000).

No, they used fixed points, see here: https://www.factorio.com/blog/post/fff-64

Though those don't actually explain the 2000km range. It's probably just something silly like the map generator breaks and noone knows why

[bobucles]

Once the floating point number loses precision, you have reached the effective end of the world. While you can just throw enough bits at the problem until it vanishes, floats simply aren't designed to store values that require consistent precision across all values. That's what integers are for!

A typical coder might think "aw damn, I need to represent a decimal at some point so I HAVE to use floats!". Not true. Shift the decimal point over so that the smallest value is the level of precision you require. 1000 can represent ten + 0/100ths and none's the wiser.

No one will reach the end of the world here unless they're deliberately running in one direction on hax speed for weeks on end with no biters.

[Jupiter]

So, does anyone have an idea how this is stored in ram in such a way that it is accessible in realtime?

This prior post details some aspects of the world generation, as does this wiki page. It is procedurally generated using Perlin noise. (In principle they could have used any parameterizable locally continuous function, but Perlin is a perfectly good choice.)

Now, still, despite this, the trouble is that parts you've "seen" are stored, which is why, if you save, then reveal a bunch of map using the console, then save again, the second save will be much larger. (In principle I guess, if they committed to never changing the map generation algo, or at least retained the code for backward compatibility, they could have conceivably just left everything purely procedural, and just stored the diff relative to the procedurally generated stuff, but this is not practical for various reasons.) So everything you've seen they store.

Then, one thing preventing you from "seeing" too much are the presence of biter nests. You know those nests with a hundred spawners, and hundreds of worms, where you eventually say, "screw it, not worth it" and go back to your base to build your factory and what have you? Biters aren't just protecting their planet, they're also protecting your computer!

But there is one more thing preventing you from seeing too much. Let's imagine you have no water, and no biters. I figure that with six exoskeletons (powered by two fusion reactors) I can cover about 30-40 squares per second, on unpaved terrain. The player's "view" appears to be 5 by 5 chunks (chunks being the 32x32 squares), so let's figure that in best case by running full-out I can reveal perhaps 5 chunks a second. (This is admittedly totally a SWAG, but not a terrible one I hope. ) 2M by 2M is 62.5k by 62.5k chunks, or roughly 390 billion chunks. So at 5 chunks per second, it would take you about 78 billion seconds, or about ~2500 years, to explore this whole area, without using the console or something. Forty years in the desert would have nothing on this playthrough.

Anyway I wouldn't worry about it.

This isn't entirely right. If the player travels at 40 tiles/s then it takes him 2M / 40 = 50000 seconds to go from one side to the other side of the map. If the player has a vision of 5x5 chunks (meaning, 2.5 chunks in all directions) and a chunk is 32 tiles then he has to run back and forth from the sides of the map 2M / (5*32) = 12500 times. Each trip takes him 50k seconds so that would be 50k * 12.5k = 625.000.000 seconds. This is 625M / 3600 second in an hour / 24 / 365 = 19.82 years. Not 2500 years. This is a rough estimate but only off by a couple of hours, maybe a day.

But there are ways around this (excluding console commands) using mods. Such as FARL and Satellite radar. But I guess that those are just other ways of invoking console commands....

[Jarin]

And again, just having the map revealed doesn't load it in memory. You'd have to make every single map chunk "active". Requiring thousands of radar units and some way to power them.

[PKing Zombie Spy]

This isn't entirely right. If the player travels at 40 tiles/s then it takes him 2M / 40 = 50000 seconds to go from one side to the other side of the map. If the player has a vision of 5x5 chunks (meaning, 2.5 chunks in all directions) and a chunk is 32 tiles then he has to run back and forth from the sides of the map 2M / (5*32) = 12500 times. Each trip takes him 50k seconds so that would be 50k * 12.5k = 625.000.000 seconds. This is 625M / 3600 second in an hour / 24 / 365 = 19.82 years. Not 2500 years. This is a rough estimate but only off by a couple of hours, maybe a day.

Whoops, I miscalculated 62.5k times 62.5k as 390 billion instead of correctly 3.9 billion. So ~25 years.

[ssilk]

I have updated the article-part: https://wiki.factorio.com/index.php?tit ... new_Chunks

I also want to mention, that I have currently a map, which needs 12 GB RAM. But I can play it also with my 8 GB Laptop. When I look into the memory usage I see, that 4 GB is swapped out to disk. Makes sense, cause a chunk will only change (need to write = need to swap it in), if the player modifies it. Or some other entity...

[Kayanor]

I have updated the article-part: https://wiki.factorio.com/index.php?tit ... new_Chunks

Maximum Map Size and used Memory
The map size is limited to 2000 x 2000 kilometers (2,000,000 tiles). This is larger than the earth land-surface. It would take around 8 game-hours by train to reach that border. So we can speak here from "endless".

2000 x 2000 km = 4,000,000 km² (= 4,000,000 tiles), not 2,000,000.
Or am I wrong?

[Zeblote]

I have updated the article-part: https://wiki.factorio.com/index.php?tit ... new_Chunks

Maximum Map Size and used Memory
The map size is limited to 2000 x 2000 kilometers (2,000,000 tiles). This is larger than the earth land-surface. It would take around 8 game-hours by train to reach that border. So we can speak here from "endless".

2000 x 2000 km = 4,000,000 km² (= 4,000,000 tiles), not 2,000,000.
Or am I wrong?

2000km is 2,000,000 tiles. The area would be 4,000,000,000,000 tiles.

[daniel34]

Maximum Map Size and used Memory
The map size is limited to 2000 x 2000 kilometers (2,000,000 tiles). This is larger than the earth land-surface. It would take around 8 game-hours by train to reach that border. So we can speak here from "endless".

The map size is correct although a little bit ambiguous, one could also read that 2000x2000km equal 2 million tiles.

But the other statements are false:

This is larger than the earth land-surface.

The earth's land surface is ~148 million km², Factorio's is only 4 million km².

It would take around 8 game-hours by train to reach that border.

These are not game-hours but realtime hours. (8 game-hours are only 2m20s realtime).
It would take around 4 hours as the distance to the border is only 1000km from the player spawn point. It should more precisely say "8 hours by train to get from one end of the map to the other."

[ssilk]

Fixed issues. I like this

This is between the size of India and Australia.

And I think that the term "game-hour" is quite ambiguous. I wrote that long ago: https://wiki.factorio.com/index.php?title=Game-second -> Game-Hour

My logic: A game-day is 416.66 game-seconds. A game-second is a second. So a game-minute is 60 game-seconds. And a game-hour is 60 game-minutes.

Yours: A game-day is 416.66 game-seconds. A day has 24 hours and so a game hour has 17.36 seconds.

Well, we are obviously both right (*). We have here two different hours I suggest to give the game hour two names.
- The game-day-hour (17 game-seconds)
- The game-hour (60 game-minutes)


* I still think it is not useful to have an hour, which is shorter than a minute, but well...