read status of screenshot queue

Post Reply
L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

read status of screenshot queue

Post by L0laapk3 »

Currently, when you use a script to take a screenshot, there is no way to tell if the screenshot is actually done.

My current solution involves waiting 1 gametick and then calling game.writefile(). But I've receieved a few bug reports about missing images in factoriomaps, so I've come to the conclusion that this method does not actually work.

I'm looking for a solution that avoids:
- waiting an arbitrary amount of time
- checking the existence of every single image (there can be easely 10's of thousands of images) (edit: this wouldnt work anyways, see edit below)
- basically any solution that just decreases the chances of this happening instead of solving this for good.

Currently, the only solution I can come up with is sending a SIGINT to factorio and waiting until it quits. But I'm not even sure if that actually works, and also I can't do that as I have more work to do after the screenshotting.


Is there a solution to this problem already that I am overlooking? If there is no solution, I propose to include one in the API in the form of a blocking function that forces the processing of all the screenshots, or maybe game.check_screenshot_queue(), or some event.

That might not be the optimal solution, some of the things I said might be completely wrong, I am just guessing what is actually happening under the hood of factorio with the screenshot stuff.
Last edited by L0laapk3 on Sun Feb 24, 2019 12:43 am, edited 2 times in total.

L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

Re: read status of screenshot queue

Post by L0laapk3 »

Afterthought:

Would it be sufficient to check the existance of the last (screenshots-threads-count + screenshots-queue-size screenshots)? (from the config.ini)


edit: this wouldn't work as there files can be created but not completely written to.

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: read status of screenshot queue

Post by quyxkh »

On linux you can set up an inotifywait loop that gets told what to expect and shuts down when it's processed all the screenshots it was expecting and has been told to quit when done, here's a sketch

(in game)

Code: Select all

    shot={prefix='mapraw/'}
    repeat 
        --[[ prepare a screenshot here, then ]]
        game.write_file(shot.prefix..'expect.'..shot.uniquename,'')
        game.take_screenshot{..path=shot.prefix..shot.uniquename)
    until done
    game.write_file(shot.prefix..'alldone','')
 
in the shell, fire up both the shot processor and factorio, wait for the shot processor to get everything done and then drop factorio:

Code: Select all

    cd ~/.factorio # factorio's write-data dir
    declare -i pending=0 alldone=0
    inotifywait -qrm "script-output/mapraw" -e close_write --format %w%f \
    | while read; do case "$REPLY" in
            expect.*) pending+=1; rm "$REPLY" ;;
            alldone) alldone=1; [[ ! ((pending)) ]] && break ;;
            *) # actual screenshot
                process your screenshot here
                ((--pending==0 && alldone )) && break
                ;;
      esac; done & 
    pixprocessing=$!
    factorio >/dev/null &
    factorio=$!
    
    wait $pixprocessing
    kill $factorio
The shot processor shuts down as soon as it's seen the `alldone` file created and has processed as many screenshots as it was told to expect, when the loop sees that, it drops factorio.

L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

Re: read status of screenshot queue

Post by L0laapk3 »

I was doing exactly this but only writing the file 1 tick later. The problem is there exist some sort of screenshotting queue that attempts to spread out the saving of files or something along those lines. So it doesn't actually work. I never noticed that it didnt work because for me it always worked but I got some bug reports on my mods for it.

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: read status of screenshot queue

Post by quyxkh »

I'm surprised to hear creating an "expect" file via write-file when queuing the screenshot doesn't work, I can't imagine how the postprocessor could be getting the number of pending screenshots it's waiting for wrong.

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: read status of screenshot queue

Post by Rseding91 »

I've added LuaGameScript::set_wait_for_screenshots_to_finish() for 0.17. When set it causes the queued screenshots to be force flushed to disk at the end of the current tick.
If you want to get ahold of me I'm almost always on Discord.

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: read status of screenshot queue

Post by posila »

Rseding91 wrote:
Mon Feb 25, 2019 6:14 am
I've added LuaGameScript::set_wait_for_screenshots_to_finish() for 0.17. When set it causes the queued screenshots to be force flushed to disk at the end of the current tick.
I just quickly went through the change quickly ... It doesn't guarantee rendering will run for current tick.

L0laapk3, you should also know that capturing screenshot is not guaranteed to succeed and all errors that happen during taking screenshots are silently thrown away.

L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

Re: read status of screenshot queue

Post by L0laapk3 »

posila wrote:
Mon Feb 25, 2019 7:22 am
L0laapk3, you should also know that capturing screenshot is not guaranteed to succeed and all errors that happen during taking screenshots are silently thrown away.
How so? What are some of the errors that could potentially occur? Is it maybe an option to give some indication of those errors in the output logs? (together with the target image path maybe)

posila wrote:
Mon Feb 25, 2019 7:22 am
Rseding91 wrote:
Mon Feb 25, 2019 6:14 am
I've added LuaGameScript::set_wait_for_screenshots_to_finish() for 0.17. When set it causes the queued screenshots to be force flushed to disk at the end of the current tick.
I just quickly went through the change quickly ... It doesn't guarantee rendering will run for current tick.
Thanks for the work, what should I expect from this?

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: read status of screenshot queue

Post by posila »

L0laapk3 wrote:
Mon Feb 25, 2019 11:16 am
How so? What are some of the errors that could potentially occur? Is it maybe an option to give some indication of those errors in the output logs? (together with the target image path maybe)
Most commonly, running out of memory - either GPU, or even CPU failing to allocate memory to copy the image from GPU and compress it; filesystem errors, etc. Another common error could be trying to take larger screenshot than maximum supported render target size on given GPU. In 0.17 every GPU that will be able to run the game should handle 8192x8192px render targets though.

I believe all (or at least most) errors are being logged to the log file. If Rseding wants to make enhancements for screenshotting functionality, I won't stop him :)

L0laapk3 wrote:
Mon Feb 25, 2019 11:16 am
posila wrote:
Mon Feb 25, 2019 7:22 am
I just quickly went through the change quickly ... It doesn't guarantee rendering will run for current tick.
Thanks for the work, what should I expect from this?
The game may run multiple updates without rendering; in single player this happens only when rendering itself takes long time. In multiplayer the game will sacrifice rendering for trying to keep up with the server. It should be safe to assume that rendering will happen within 10 ticks.

L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

Re: read status of screenshot queue

Post by L0laapk3 »

posila wrote:
Mon Feb 25, 2019 2:13 pm
The game may run multiple updates without rendering; in single player this happens only when rendering itself takes long time. (...) It should be safe to assume that rendering will happen within 10 ticks.
Edit: The mod stalls the game for a very long time and is only suited for single player.

I'm taking all the screenshots in the same tick. Its important that the screenshots are tick-perfect (graphically). Is this maintained? How safe is it to assume that it will happen within 10 ticks? Is this with LuaGameScript::set_wait_for_screenshots_to_finish() or without?

User avatar
DaveMcW
Smart Inserter
Smart Inserter
Posts: 3699
Joined: Tue May 13, 2014 11:06 am
Contact:

Re: read status of screenshot queue

Post by DaveMcW »

L0laapk3 wrote:
Mon Feb 25, 2019 2:35 pm
Its important that the screenshots are tick-perfect (graphically). Is this maintained?
No.

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: read status of screenshot queue

Post by posila »

What do you mean tick perfect? All screenshots queued in the same tick will be captured at the same tick. But they are not guaranteed to be captured at the same tick you requested them.

I forgot about game speed ... yeah, considering game speed, even 10 ticks might not be long enough delay.

L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

Re: read status of screenshot queue

Post by L0laapk3 »

posila wrote:
Mon Feb 25, 2019 3:41 pm
What do you mean tick perfect? All screenshots queued in the same tick will be captured at the same tick. But they are not guaranteed to be captured at the same tick you requested them.
I just care if they are consistent with eachother.

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: read status of screenshot queue

Post by posila »

I believe they are

L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

Re: read status of screenshot queue

Post by L0laapk3 »

posila wrote:
Mon Feb 25, 2019 2:13 pm
The game may run multiple updates without rendering; in single player this happens only when rendering itself takes long time. In multiplayer the game will sacrifice rendering for trying to keep up with the server. It should be safe to assume that rendering will happen within 10 ticks.
Wait, do you mean, that when you call that function, if the game is not running at 60fps only then there is an extra delay and only if there are frames dropped that the screenshots arent handled in that tick?

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: read status of screenshot queue

Post by Rseding91 »

L0laapk3 wrote:
Mon Feb 25, 2019 7:50 pm
posila wrote:
Mon Feb 25, 2019 2:13 pm
The game may run multiple updates without rendering; in single player this happens only when rendering itself takes long time. In multiplayer the game will sacrifice rendering for trying to keep up with the server. It should be safe to assume that rendering will happen within 10 ticks.
Wait, do you mean, that when you call that function, if the game is not running at 60fps only then there is an extra delay and only if there are frames dropped that the screenshots arent handled in that tick?
Yes.

Screenshots can't be processed while the game tick is running - only when a frame is about to be rendered is it safe to do so. As such: all queued screenshots are only processed when the next frame is rendered and shown on screen.
If you want to get ahold of me I'm almost always on Discord.

L0laapk3
Inserter
Inserter
Posts: 47
Joined: Sun Mar 18, 2018 10:01 pm
Contact:

Re: read status of screenshot queue

Post by L0laapk3 »

Rseding91 wrote:
Mon Feb 25, 2019 9:29 pm
L0laapk3 wrote:
Mon Feb 25, 2019 7:50 pm
posila wrote:
Mon Feb 25, 2019 2:13 pm
The game may run multiple updates without rendering; in single player this happens only when rendering itself takes long time. In multiplayer the game will sacrifice rendering for trying to keep up with the server. It should be safe to assume that rendering will happen within 10 ticks.
Wait, do you mean, that when you call that function, if the game is not running at 60fps only then there is an extra delay and only if there are frames dropped that the screenshots arent handled in that tick?
Yes.

Screenshots can't be processed while the game tick is running - only when a frame is about to be rendered is it safe to do so. As such: all queued screenshots are only processed when the next frame is rendered and shown on screen.
So, to summarise: When LuaGameScript::set_wait_for_screenshots_to_finish() is called, and at least 1 screenshot file is found, THEN one tick later, I can be confident that all screenshots (from the same tick) are done. Correct?

Post Reply

Return to “Implemented mod requests”