Page 1 of 1

Do an operation each n tick not submitted with on_tick event

Posted: Tue Dec 03, 2019 3:50 pm
by Linver
Hi guy!
I'm trying to write a code in control for do a think like this:
  1. Game call an event different from on_tick, let's call this event A
  2. A call is associated to a function that put some values in a global table, like global.B
  3. Let's call on_tick each 60 tick and take one item from global.B do something and wait.
I have some problem with this, in global.B I put some value from an entity (taken from differents call of A), what I expect are there is inside is some like

Code: Select all

{
    ...
    {entity.surface, entity.position}
    ...
}
but when I call on_tick I'm not sure about that if take one element and after I remove it, the table will be iterated one item a time. For example:

Code: Select all


function doThings(event)
	game.print(#global.B)
	if event.tick % 60 == 0 and #global.B> 0 then
		local i, entry = next(global.B)	
		if entry then				
			doSomething(entry)
			table.remove(global.B, i)
		end
	end
end
...
script.event(...on_tick, doThings)
In this case game.print(#global.B) will print very huge number (if I expect 10 entities can be 100) and not all will be called.
When the game call 2 event in control of one mod, is a sequential execution? Can be some call ignored if too much? There's a limit on the space taken in global by one mod?
Any suggestion?

Re: Do an operation each n tick not submitted with on_tick event

Posted: Tue Dec 03, 2019 8:10 pm
by DaveMcW
Lua has two kinds of tables.

Array
Must be indexed 1, 2, 3, 4...
Uses # for table size
Must use table.insert()
Must use table.remove()

Associative
Index can be any value
Uses table_size() for table size
Must insert using brackets, data[key] = value
Must delete using brackets, data[key] = nil

If you do something wrong to an array-style table, it will become associative, and array functions will not work correctly.

Re: Do an operation each n tick not submitted with on_tick event

Posted: Wed Dec 04, 2019 1:49 am
by PyroFire
DaveMcW wrote: Tue Dec 03, 2019 8:10 pm If you do something wrong to an array-style table, it will become associative, and array functions will not work correctly.
This has nothing to do with the issue.

it's because you're calling it on tick, and the doThings() only iterates over one item
so doThings() once in on_tick does one entity
if you wanted 10 then its for i=1,10,1 do doThings() end for example
note when using next() for delayed iteration (1 at a time), you need to store the key last used, in this case its i
in the local i,val = next(table)
store i then do local next_i,next_val=next(table,previous_i) for e.g.
make sure the key exists etc
basically you're doing next() wrong. you need to store previous_i in global.

Currently each step is only acting on the first element in your tables because of this not being setup correctly.

Re: Do an operation each n tick not submitted with on_tick event

Posted: Thu Dec 05, 2019 12:56 am
by Linver
After some trying I understand that the problem is different, this call:

Code: Select all

script.on_event(defines.events.on_chunk_generated, functionA)
In function A i do something like:

Code: Select all

function functionA(event)
	local entities = event.surface.find_entities_filtered{...}
	...
	for to iter on entities
	 put some position in the famous global.B (in the first message of the post)
	...
	end
end
The functionA will be called more than one time for the same chuck, how is this possible?

Re: Do an operation each n tick not submitted with on_tick event

Posted: Thu Dec 05, 2019 5:04 pm
by Linver
Solved, thanks again to PyroFire

Re: Do an operation each n tick not submitted with on_tick event

Posted: Sat Dec 07, 2019 4:33 pm
by Honktown
Don't iterate over a table you're editing. It's undefined normally, don't know if Factorio's version of Lua changes it.

You should make a temp table of keys, where each key in pairs(first_table) is stored, using table.insert(temp_table, key). Then do for i in ipairs(temp_table) do... myfunction( first_table[temp_table[ i ]] )... first_table[temp_table[ i ]] = nil (spaces around i to avoid the italics tag)

Re: Do an operation each n tick not submitted with on_tick event

Posted: Sat Dec 07, 2019 11:48 pm
by quyxkh
If you're not going to use `script.on_nth_tick`, the easiest way to scan an entire table while potentially adding and removing elements is to not insist on any particular ordering:

Code: Select all

-- this is how you efficiently remove items from a numbered but not particularly ordered table named `zlist`:
n=1 while n <= #zlist
    if shouldremove(zlist[n])
        then zlist[n]=zlist[#zlist] zlist[#zlist]=nil 
        else n=n+1
        end
    end
-- adding to unordered tables like that is just table.insert however you want to spell it:
zlist[#zlist+1]=newvalue

Re: Do an operation each n tick not submitted with on_tick event

Posted: Wed Dec 11, 2019 1:30 pm
by Linver
Honktown,
in my implementation I don't iterate on a table that I modify constantly, I say to the on nth tick callback to take the first sure element and remove it if no one have compute it before. I test it with a constant input and unstable one. No errors.
My initial misunderstand was in the filter declaration on "on chuck generated" event (noticed by Pyrofire).
Thank u anyway for the suggestion.

quyxkh,
I'm using on_nth_tick, and that function take the first sure item, the other callbacks will do only some insert operation, and isn't important with what index they insert the item, so there's no reason to order the table, like is now already is processed like expected.
My initial misunderstand was in the filter declaration on "on chuck generated" event (noticed by Pyrofire).
Thank u too for the suggestion.

Re: Do an operation each n tick not submitted with on_tick event

Posted: Wed Dec 11, 2019 11:42 pm
by Honktown
Linver wrote: Wed Dec 11, 2019 1:30 pm Honktown,
in my implementation I don't iterate on a table that I modify constantly, I say to the on nth tick callback to take the first sure element and remove it if no one have compute it before. I test it with a constant input and unstable one. No errors.
My initial misunderstand was in the filter declaration on "on chuck generated" event (noticed by Pyrofire).
Thank u anyway for the suggestion.
A random performance optimization to consider after the mod is working: depending on how big the table is, removing from the end is more efficient than removing from the front. If you remove from the front, it has to re-index the table from 2-n and it can do not-great things with memory (fragmentation). If you remove from the end, there's no re-indexing, and all memory is available from the end. Would matter if you're following tons of entities and expect to do the removal operation every time. It changes the behavior, because you're operating on the last thing inserted, instead of the first thing.

Edit: I had to check, although Lua is dynamic with keys and things, it does index table objects. A for loop is faster than pairs/ipairs, obviously only works for indexed, hole-less tables

Re: Do an operation each n tick not submitted with on_tick event

Posted: Thu Dec 12, 2019 6:36 am
by Optera
My preferred way to spread entity iteration over multiple ticks would be to use a global associative array and use next.

Code: Select all

function OnTick(event)
  if global.Entity_Index and not global.MyEntities[global.Entity_Index] then
    -- the index is no longer valid, probably because the associated entity was removed
    global.Entity_Index = nil
    return
  end
  
  for i=1, i<updates_per_tick, 1 do
    local unit_number, entity= next(global.MyEntities, global.Entity_Index)
    if entity then
      global.Entity_Index = unit_number
      -- do stuff with your entity
    else
      -- we reached the end of global.MyEntities
      global.Entity_Index = nil
      return
  end  
end

Advantages:
+ the same code will work for 1 entity every n ticks, m entities every tick or m entities every n ticks
+ performance is much better than using modulo or some other form of calculation on the entire array to find the current pointer
+ adding and removing entities can be done at any time and is (somewhat) gracefully handled
+ adding, removing entities doesn't require loops

Disadvantages:
- If done wrong it will produce weird errors :P

Re: Do an operation each n tick not submitted with on_tick event

Posted: Thu Dec 12, 2019 6:46 am
by Honktown
Optera wrote: Thu Dec 12, 2019 6:36 am My preferred way to spread entity iteration over multiple ticks would be to use a global associative array and use next.

Code: Select all

function OnTick(event)
  if global.Entity_Index and not global.MyEntities[global.Entity_Index] then
    -- the index is no longer valid, probably because the associated entity was removed
    global.Entity_Index = nil
    return
  end
  
  for i=1, i<updates_per_tick, 1 do
    local unit_number, entity= next(global.MyEntities, global.Entity_Index)
    if entity then
      global.Entity_Index = unit_number
      -- do stuff with your entity
    else
      -- we reached the end of global.MyEntities
      global.Entity_Index = nil
      return
  end  
end

Advantages:
+ the same code will work for 1 entity every n ticks, m entities every tick or m entities every n ticks
+ performance is much better than using modulo or some other form of calculation on the entire array to find the current pointer
+ adding and removing entities can be done at any time and is (somewhat) gracefully handled
+ adding, removing entities doesn't require loops

Disadvantages:
- If done wrong it will produce weird errors :P
Using normal key'ing gives a weird advantage: you can set the key to nil, even if it already didn't exist. A convenient shortcut I found useful (I can remove something I don't want to track, even if I wasn't already tracking it).

Re: Do an operation each n tick not submitted with on_tick event

Posted: Thu Dec 12, 2019 6:09 pm
by Klonan
Optera wrote: Thu Dec 12, 2019 6:36 am + performance is much better than using modulo or some other form of calculation on the entire array to find the current pointer
With modulo, you can modulo by the update rate to find the 'bucket' that you want to update, instead of checking the modulo for each key value:

Code: Select all

local on_tick = function(event)

  local bucket = script_data.depots[event.tick % depot_update_rate]
  if bucket then
    for unit_number, depot in pairs (bucket) do
      if not (depot.entity.valid) then
        bucket[unit_number] = nil
      else
        depot:update()
      end
    end
  end

end

Re: Do an operation each n tick not submitted with on_tick event

Posted: Fri Dec 13, 2019 3:32 am
by Honktown
Klonan wrote: Thu Dec 12, 2019 6:09 pm
Optera wrote: Thu Dec 12, 2019 6:36 am + performance is much better than using modulo or some other form of calculation on the entire array to find the current pointer
With modulo, you can modulo by the update rate to find the 'bucket' that you want to update, instead of checking the modulo for each key value:

Code: Select all

local on_tick = function(event)

  local bucket = script_data.depots[event.tick % depot_update_rate]
  if bucket then
    for unit_number, depot in pairs (bucket) do
      if not (depot.entity.valid) then
        bucket[unit_number] = nil
      else
        depot:update()
      end
    end
  end

end
Just because that's literally how you do it :lol:

A side note, if someone could update the API to be a little clearer on on_nth_tick: it can have multiple functions registered for different nth ticks, even though it's in bootstrap, like on_init and on_load which can only be used once. It makes sense different nth ticks can have different functions, but it's not like they're a different bootstrappy command, you know?

Page in question~~ https://lua-api.factorio.com/latest/LuaBootstrap.html

Re: Do an operation each n tick not submitted with on_tick event

Posted: Fri Dec 13, 2019 6:57 am
by Optera
Klonan wrote: Thu Dec 12, 2019 6:09 pm
Optera wrote: Thu Dec 12, 2019 6:36 am + performance is much better than using modulo or some other form of calculation on the entire array to find the current pointer
With modulo, you can modulo by the update rate to find the 'bucket' that you want to update, instead of checking the modulo for each key value:

Code: Select all

local on_tick = function(event)

  local bucket = script_data.depots[event.tick % depot_update_rate]
  if bucket then
    for unit_number, depot in pairs (bucket) do
      if not (depot.entity.valid) then
        bucket[unit_number] = nil
      else
        depot:update()
      end
    end
  end

end
Not bad, but still more expensive than index, data = next(dataset, index)
Beside binding the modulo to tick means updates of a given entity are directly joined to a tick making it unreliable for updating m entities every n ticks where both m and n can fluctuate heavily.