[Short article] Can X-Ypoint be described by only one coord?

Place to post guides, observations, things related to modding that are not mods themselves.
Post Reply
User avatar
Betep3akata
Inserter
Inserter
Posts: 40
Joined: Fri Sep 30, 2016 10:31 am
Contact:

[Short article] Can X-Ypoint be described by only one coord?

Post by Betep3akata »

Can every point on a surface be described by only one coordinate? - Yes!
One day I wondered how to prevent the simultaneous and redundant tasks on a single point (or tile) on the map. The short answer is the sets and the caching.

Sets in the Lua looks, for example, like:

Code: Select all

cached = { [ID1] = true, [ID3] = true, ... , [ID2] = true }
We can use such a cache to avoid the appearance of the multiple tasks for the same point:

Code: Select all

if not cached[ID] then
    add(x, y, action)
For the point on the map, the ID is its X-Y coordinates. To our disappointment is not so easy to create the set in Lua with a kind of compound key. But we can produce a single value that is uniquely associated with that pair of X-Y coordinates. It is called a bijective ZxZ -> N mapping. This conversion is performed by so-called pairing functions.

[EDIT]: Most likely I'm wrong in that I use the pairing functions to generate the hash from the two values. The main advantage of the pairing function is that the unpairing functions exist :) And in the above case, reversibility of the operation is not required. But I have not yet found a simple and fast hash function for Lua. :?: It is also very desirable that the computed IDxy will have not too many digits.

The easiest way is the concatenation of these two coordinates as strings with a delimiter between:

Code: Select all

IDxy = tostring(x).."/"..tostring(y)
But how much it will cost in terms of the performance? :?

Probably it is not a better way at all. And I managed to find on the web at least another two examples of the pairing functions:
  • 1. The Cantor pairing function;
    2. An Elegant Pairing Function by Matthew Szudzik.
These methods provide the pairing functions for NxN -> N (N means non-negative integers). But in the Factorio we have coordinates on the Z surface (Z means signed integers).

For the Z -> N mapping we can use such a way:

Code: Select all

n = z * 2, for n >= 0;
n = -z * 2 - 1, for n < 0. 
Let's compare these three functions by the simple script below.

profiler.lua

Code: Select all

local profiler = {}
require "socket"

local Timer

local function getTime()
    -- time in milliseconds instead of os.time() in seconds
    return socket.gettime()*1000
end

function profiler.start()
    Timer = getTime()
end

function profiler.stop()
    return tostring(os.difftime(getTime(), Timer))
end

return profiler
comparison.lua

Code: Select all

------------------------------------------------------
-- Test environment section
------------------------------------------------------
local profiler = require("profiler")
local p = 
{ 
x = {},
y = {}
}

local out1 = {}
local out2 = {}
local out3 = {}
local out4 = {}
local out5 = {}
local out6 = {}
local out7 = {}

local function genPolyline(n, max)
    for i = 1, n do
        p.x[i] = math.random(-max, max)
        p.y[i] = math.random(-max, max)
    end
end

------------------------------------------------------
-- The mappings
------------------------------------------------------

-- ZxZ -> {strings}
local function concatPair(x, y)
    return tostring(x).."/"..tostring(y)
end

-- Z -> N
local function NtoZ(x, y)
    return (x >= 0 and (x * 2) or (-x * 2 - 1)), (y >= 0 and (y * 2) or (-y * 2 - 1))
end

-- ZxZ -> N, i.e. (x, y) -> IDxy
local function cantorPair_v1(x, y)
    x,y = NtoZ(x, y)  
    return (x + y +1)*(x + y)/2 + x
end

local function cantorPair_v2(x, y)
    x,y = NtoZ(x, y) 
    -- other representation, same result
    return (x*x + 3*x + 2*x*y + y + y*y) / 2
end

local function cantorPair_v3(x, y)
    x,y = NtoZ(x, y)
    -- the same values * 2
    return x*x + 3*x + 2*x*y + y + y*y
end

local function elegantPair_v1(x, y)
  x,y = NtoZ(x, y)
  return (x >= y and (x * x + x + y) or (y * y + x))
end

local function elegantPair_v2(x, y)
    x,y = NtoZ(x, y)
    if x >= y then
        return x * x + x + y
    end
    return y * y + x
end

local function godelPair(x, y)
    x,y = NtoZ(x, y)
    return (2^x)*(3^y)
end

------------------------------------------------------
-- Profiling section
------------------------------------------------------
-- init test data
local n = 5000000
print("Generating test data..")
genPolyline(n, 12000)
print("Profiling:")

print("Test #1 - concatPair")
profiler.start()
for i = 1, n do
    out1[i] = concatPair(p.x[i], p.y[i])
end
local result = profiler.stop()
print(result.." ms")

print("Test #2 - cantorPair v1")
profiler.start()
for i = 1, n do
    out2[i] = cantorPair_v1(p.x[i], p.y[i])
end
local result = profiler.stop()
print(result.." ms")

print("Test #3 - cantorPair v2")
profiler.start()
for i = 1, n do
    out3[i] = cantorPair_v2(p.x[i], p.y[i])
end
local result = profiler.stop()
print(result.." ms")

print("Test #4 - cantorPair v3")
profiler.start()
for i = 1, n do
    out4[i] = cantorPair_v3(p.x[i], p.y[i])
end
local result = profiler.stop()
print(result.." ms")

print("Test #5 - elegantPair v1")
profiler.start()
for i = 1, n do
    out5[i] = elegantPair_v1(p.x[i], p.y[i])
end
local result = profiler.stop()
print(result.." ms")

print("Test #6 - elegantPair v2")
profiler.start()
for i = 1, n do
    out6[i] = elegantPair_v2(p.x[i], p.y[i])
end
local result = profiler.stop()
print(result.." ms")

print("Test #7 - godelPair")
profiler.start()
for i = 1, n do
    out7[i] = godelPair(p.x[i], p.y[i])
end
local result = profiler.stop()
print(result.." ms")

print("Done!")
Shortened script output (in average on my PC for 5 000 000 of operations):
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio
concatPair: 15199 ms
cantorPair v1: 1949 ms
cantorPair v2: 2312 ms
cantorPair v3: 2266 ms
elegantPair v1: 2017 ms
elegantPair v2: 1974 ms
[EDIT]: godelPair: ~3000-3500ms


As we can see, the way of multiple concatenations is dramatically slow. I was rooting for the elegantPair() function but I did not manage to implement it to be faster than the Cantor's method.

The more efficient ways are welcome! For me the current winner is:

Code: Select all

-- Z -> N
local function NtoZ(x, y)
    return (x >= 0 and (x * 2) or (-x * 2 - 1)), (y >= 0 and (y * 2) or (-y * 2 - 1))
end

-- ZxZ -> N, i.e. (x, y) -> IDxy
local function cantorPair_v1(x, y)
    x,y = NtoZ(x, y)  
    return (x + y +1)*(x + y)/2 + x
end
Perhaps this method for describing of X-Y coordinate by only one value will be useful for someone.

Materials used:
http://szudzik.com/ElegantPairing.pdf (an Elegant Pairing Function by Matthew Szudzik)
https://en.wikipedia.org/wiki/Pairing_function (The Cantor pairing function)
https://www.lua.org/gems/sample.pdf (Lua Performance Tips by Roberto Ierusalimschy)
https://www.lua.org/pil/11.5.html (Lua - Sets)
[EDIT]:https://hbfs.wordpress.com/2011/09/27/p ... functions/ (some more examples of the pairing functions)
My mods: Scorched Earth, Will-o-the-wisps
//My nickname is some kind of transliteration from Cyrillic to Latin. Betep3akata stands for WindOfSunset.

User avatar
Betep3akata
Inserter
Inserter
Posts: 40
Joined: Fri Sep 30, 2016 10:31 am
Contact:

get chunk id

Post by Betep3akata »

The code below produces the unique and numerically small id for the coordinates of the chunk:

Code: Select all

local function getChunkID(x,y)
    return cantorPair((x - x%32) / 32,  (y - y%32) / 32)
end
My mods: Scorched Earth, Will-o-the-wisps
//My nickname is some kind of transliteration from Cyrillic to Latin. Betep3akata stands for WindOfSunset.

User avatar
Adil
Filter Inserter
Filter Inserter
Posts: 945
Joined: Fri Aug 15, 2014 8:36 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Adil »

Good find.
I do mods. Modding wiki is friend, it teaches how to mod. Api docs is friend too...
I also update mods, some of them even work.
Recently I did a mod tutorial.

User avatar
Sirenfal
Long Handed Inserter
Long Handed Inserter
Posts: 50
Joined: Tue Jan 10, 2017 1:42 am
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Sirenfal »

Lua doesn't have bitwise operators, and Factorio doesn't seem to have Lua's bitwise library. You can still do the equivalent operation. These should be fairly fast, with the caveat that you can't store signed ints.

The actual code:

Code: Select all

-- Factorio uses 64-bit Lua. All numbers are stored as 64-bit doubles.
-- A double has 52 bits in the mantissa, so we can represent int values up to 52 bits
-- 52/2 == 26 bits per side in our x,y key
local base = math.pow(2,26)
local floor = math.floor

function pack(x, y)
        if(x % 1 ~= 0 or y % 1 ~= 0) then
                error('Cannot store floats')
        elseif(x < 0 or y < 0) then
                error('Cannot pack negative numbers')
        elseif(x > (base-1) or y > (base-1)) then
                error('Cannot pack values higher than ' .. (base-1))
        end

        -- x << 26 | y
        return (x*base)+y
end

function unpack(key)
        -- (key & (2**26-1), key >> 26)
        return {x=floor(key / base), y=key % base}
end
Testing it for correctness. I haven't actually tested this in game because I'm busy, but I did test it on 64-bit Lua. Should work in Factorio too

Code: Select all

local function test_val(x,y)
        local res = unpack(pack(x,y))

        if(res.x ~= x or res.y ~= y) then
                error('x: ' .. x .. ', y: ' .. y .. ' | x2: ' .. res.x .. ', y2: ' .. res.y .. ', key: ' .. pack(x,y))
        end
end

for x=0,base-1 do
        test_val(x,x)
end

User avatar
Betep3akata
Inserter
Inserter
Posts: 40
Joined: Fri Sep 30, 2016 10:31 am
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Betep3akata »

Hi, thanks for your addition to this topic.

I'm have tested the code below hurriedly.

Code: Select all

local base = math.pow(2,26)
local floor = math.floor

local function pack(x, y)
        x,y = NtoZ(x, y)
        -- x << 26 | y
        return (x*base)+y
end
The resulting durations on the same test:

Code: Select all

Test #2 - cantorPair v1
2164 ms
Test #6 - elegantPair v2
2255 ms
Test #8 - pack
2248 ms
And I don't know why it even slightly slower than the cantorPair function. Perhaps due to the fact that more digits are used on average.
My mods: Scorched Earth, Will-o-the-wisps
//My nickname is some kind of transliteration from Cyrillic to Latin. Betep3akata stands for WindOfSunset.

Exasperation
Long Handed Inserter
Long Handed Inserter
Posts: 88
Joined: Fri Apr 22, 2016 9:55 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Exasperation »

I did some playing around with this to see if I could manage some unneeded optimization, and came up with the following:

Code: Select all

local function NtoZ(x, y)
   return (x >= 0 and (x * 2) or (-x * 2 - 1)), (y >= 0 and (y * 2) or (-y * 2 - 1))
end

local function NtoZ_b(x, y)
   return (x >= 0 and (x + x) or (-1 - x - x)), (y >= 0 and (y + y) or (-1 - y - y))
end

local function cantorPair_v1(x, y)
   x, y = NtoZ(x, y)
   return (x + y + 1)*(x + y)/2 + x
end

local function cantorPair_v4(x, y)
   x, y = NtoZ(x, y)
   local s = x + y
   return s * (s + 1) / 2 + x
end

local function cantorPair_v5(x, y)
   x, y = NtoZ_b(x, y)
   local s = x + y
   return s * (s + 1) + x + x
end

local function cantorPair_v6(x, y)
   x, y = NtoZ_b(x, y)
   local s = x + y
   return s * s + s + x + x
end
I obviously can't compare my results directly to yours (running in a different environment on a different machine), but my results were:
v1 (included for comparison): ~89.5ms
v4: ~88ms
v5:~87.5ms
v6:~87ms

My methodology was to do a bunch of runs of each, keep a running total of the times for each version, and after each set of runs display (total run time)/(# of runs) for each version; that way any jitter in the results would hopefully cancel out over a large sample of runs. I'm guessing that the large difference in magnitudes between my results and yours is mostly due to using LuaJIT on my end. Would be interesting to see if the results work out similarly with your setup.

The margins are pretty slim (only about 2.8% improvement from v1 to v6), but I guess the moral is that multiplication is slower than addition and memoization is faster than addition - but not by very much.

Edit - Another bit of messing around, taking advantage of v6's doubling of the calculated index:

Code: Select all

local function NtoZ_c(x, y)
   return (x >= 0 and x or (-0.5 - x)), (y >= 0 and y or (-0.5 - y))
end

local function cantorPair_v7(x, y)
   x, y = NtoZ_c(x, y)
   local s = x + y
   return s * s + s + x + x
end
v7 gets ~86.5ms for me, but its presence inexplicably causes v4 and v5 to take longer than in the previous test. v1 and v6 times are unaffected. Commenting out the code for v7 restores the previous results for v4 and v5. I'm guessing there's some compiler optimization hijinks going on?

User avatar
Adil
Filter Inserter
Filter Inserter
Posts: 945
Joined: Fri Aug 15, 2014 8:36 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Adil »

LuaJIT is unlikely to give a faithful representation of performance of the pure lua. Not only it is unknown when the jit will kick in but also, there's no guarantee that the fastest compiled version will also be the fastest interpreted one.

There's little to no point in fiddling with operations over a pair of numbers, external fluctuations of the processor load will have greater effect than those. Not to mention that there's much more impactful elements in the code: function call. Inline the NtoZ function and you get 20% shorter execution time.

In my tests Sirenfal's pack function actually was the fastest. Another good thing about it is that it is linear transformation: having the hash value of one position, you can directly compute the hash of a shifted position nearby without carrying out the inverse transform.

I did test the pack function with lesser base, and that didn't have notable effect on the performance. (The code below is using values greater that it can correctly transform however.)
I did use the following code:
Code
Addition also, concatPair is probably slowest possible implementation second only to usage of `table.concat`. Using `string.concat` is notably faster albeit still inferior to all non-string methods:

Code: Select all

concatOptimized = (function() 
		local format=string.format
		return 
		function(x, y)
			return format('%d/%d',x,y)
		end 
	end)()
Here's final results of my tests:

Code: Select all

testing cantorPair_v1
Time 0.931985
testing pack
Time 0.845207
testing concatPair
Time 14.386504
testing pack10Contained
Time 0.639154
testing concatOptimized
Time 5.625936
testing packContained
Time 0.636068
testing cantorContained
Time 0.723276
I do mods. Modding wiki is friend, it teaches how to mod. Api docs is friend too...
I also update mods, some of them even work.
Recently I did a mod tutorial.

Exasperation
Long Handed Inserter
Long Handed Inserter
Posts: 88
Joined: Fri Apr 22, 2016 9:55 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Exasperation »

I did say at the beginning of the post it was unneeded optimization; I did it because it's fun. I also considered inlining NtoZ, but decided against it; you could get the same performance improvement across the board for every algorithm that includes a call to NtoZ, so it doesn't actually say anything about which version is faster.

While LuaJIT doesn't necessarily give the same performance results as pure Lua (see my previous comment about running in a different environment), I have had very consistent results using LuaJIT for these tests; my averaged times converge to the same values every time I run the program despite randomized input plus any external processor load issues. Also, the devs have been talking about the possibility of migrating to LuaJIT for a while (see for example viewtopic.php?f=3&t=922&p=6529&hilit=LuaJIT#p6529 or https://www.factorio.com/blog/post/fff-111), so it isn't necessarily pointless from a Factorio modding perspective to think about performance under LuaJIT.

Also, I skipped a step in v7 (I blame it on staying up coding when I should have been sleeping). It should be

Code: Select all

local function cantorPair_v7(x, y)
   x, y = NtoZ_c(x, y)
   local s = x + y
   local h = s * (s + 0.5) + x
   return h + h
end
which actually comes out slightly faster than the wrong version by about .25ms (and doesn't mess with the performance of v4/5 either; so weird).

I will try testing the pack function at some point and see if that comes out faster for me as well.

User avatar
Adil
Filter Inserter
Filter Inserter
Posts: 945
Joined: Fri Aug 15, 2014 8:36 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Adil »

By the way, factorio lua does have bitwise operations, albeit they only work with 32 bits.
They are implemented as functions from a library, and that involves a function calls, which makes it slower than other fast solutions.
Another function
Results
Lua 5.3 on the other hand, has bitwise operators , usage of those does make function about 10% faster than the pack function that uses multiplication.
I do mods. Modding wiki is friend, it teaches how to mod. Api docs is friend too...
I also update mods, some of them even work.
Recently I did a mod tutorial.

User avatar
y.petremann
Filter Inserter
Filter Inserter
Posts: 407
Joined: Mon Mar 17, 2014 4:24 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by y.petremann »

On my case I've done some rewrite to test, firstly I decided to work on already existing table and overwriting it,this permit to have a lot of test without suffering memory problems, also all function are referenced in a table but that don't affect profiling performance.
comparison.lua
mappings.lua
One thing I've seen is that each test depends on test data, which are never the same on two profiling session, and since all functions works on the basis of multiplication, multiplication performance need to be taken in account, in fact one thing we need to take into account is which number is multiplied by the other, if we don't know which one have the smaller number of one in binary format, this has no importance since we will take more time at detecting this than multiplying them directly, but when we have a number like 2 or 2^26, this is very important because we know that they are int with only one 1 in their binary format.

Also here is the result I got

Code: Select all

Generating test data..
Profiling: (unit is ms)
cantorPair_v1   1449    1509    1444    1443    1449    1503    1437    1465    1444
cantorPair_v1_2 1466    1632    1484    1463    1454    1457    1452    1455    1458
cantorPair_v2   1690    1674    1693    1671    1659    1660    1666    1658    1665
cantorPair_v3   1610    1612    1604    1610    1611    1614    1601    1603    1603
cantorPair_v4   1411    1407    1429    1412    1421    1457    1455    1414    1411
cantorPair_v5   1414    1405    1406    1402    1400    1405    1398    1398    1400
cantorPair_v6   1474    1476    1473    1475    1473    1473    1474    1473    1472
cantorPair_v7   1355    1361    1352    1352    1357    1355    1352    1352    1354
elegantPair_v1  1432    1450    1427    1435    1434    1455    1443    1468    1437
elegantPair_v2  1419    1416    1417    1415    1415    1413    1423    1426    1429
godelPair       2193    2203    2203    2204    2195    2177    2192    2234    2251
pack    1401    1369    1354    1353    1356    1357    1356    1352    1351
pack2   1355    1356    1353    1352    1360    1352    1350    1383    1398
Done!
Perf.png
Perf.png (59.17 KiB) Viewed 6575 times
WhAt I can say is that cantorPair_v7 looks as efficient as pack

User avatar
Afforess
Filter Inserter
Filter Inserter
Posts: 422
Joined: Tue May 05, 2015 6:07 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Afforess »

It looks like you have already solved the problem satisfactorily - but I've faced this exact question and also had a fairly similar solution. I too used the bit32 library, and packed the x,y coordinates into a 10 bit integer value. If you consider that each in-game chunk is 32x32, that means you can assign every tile in a chunk a value from 0-1023 and that is a unique index. The bitwise operations are pretty straightforward:

Code: Select all

return bit32.band(bit32.bor(bit32.lshift(bit32.band(x, 0x1F), 5), bit32.band(y, 0x1F)), 0x3FF)
First confine each x,y coordinate to the chunk's local range by bitwise and with 0x1F (32), and then shift the x coordinate into the 5 high bits, and the y coordinate into the 5 low bits.

This does mean that you only have a unique coordinate id per chunk though. I never did come up with a satisfactory method of generating stable indexing for chunks themselves, only a crude one. Always open to improvements there.

Exasperation
Long Handed Inserter
Long Handed Inserter
Posts: 88
Joined: Fri Apr 22, 2016 9:55 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by Exasperation »

Interesting to see those results laid out graphically. I'm guessing that the (significantly) different ordering of cantor pair versions you got is a JiT difference, but it's nice to see that my final version still holds up.
Afforess wrote:This does mean that you only have a unique coordinate id per chunk though. I never did come up with a satisfactory method of generating stable indexing for chunks themselves, only a crude one. Always open to improvements there.
Stable indexing for chunks (or rather groups of chunks) is actually what I've been using this method for. Given (x, y), pass (math.floor(x / 32), math.floor(y / 32)) to your preferred packing function to get a unique, stable index for the chunk that (x, y) resides in (in my case, I was using 512 instead of 32 since I was dividing up a surface into 16x16 chunk sections and only cared about what section I was inside).

extermeon
Burner Inserter
Burner Inserter
Posts: 7
Joined: Thu May 18, 2017 7:44 pm
Contact:

Re: [Short article] Can X-Ypoint be described by only one coord?

Post by extermeon »

Just bookmarked this useful post. Thank you!

Post Reply

Return to “Modding discussion”