Page 1 of 2

Run from ramdisk

Posted: Tue Feb 11, 2020 7:05 pm
by Camylarde
Hi,

is there a way to move factorio to ramdisk?
I have:
1)Created a ramdisk h:/
2) Moved the steam part of the game to the root.
3) Created a sub-folder config-copy to have a separate location for roaming/factorio config data.
4) Applied the article How to move your data folder from this forum technical section faq.
5) Found the exe in the /bin folder

I still get error
---------------------------
Error
---------------------------
weakly_canonical: Incorrect function.

: "h:/config-copy/config\config.ini"
path1: h:/config-copy/config\config.ini
---------------------------
OK
---------------------------

despite the file confirm.ini being there in the path. Notable is also that if the file isn't there the new one is not created as the article says it should be...

Any suggestions?

Re: Run from ramdisk

Posted: Sun Feb 16, 2020 8:44 am
by posila
Camylarde wrote:
Tue Feb 11, 2020 7:05 pm
weakly_canonical: Incorrect function.
Hello, the error you are seeing originates from std::filesystem standard C++ library (which might or might not get it from underlying calls to WinAPI).
Can you please
1) post exact build version of Windows you are running (Win + R and type in winver, or just post Factorio log).
2) decribe steps you used to create the ramdisk
3) post your modified config.ini and command line parameters you use to launch Factorio?

Re: Run from ramdisk

Posted: Sun Feb 16, 2020 9:38 am
by Optera
For such experiments I'd start by downloading the portable version directly from factorio.com instead of using the steam version.

Re: Run from ramdisk

Posted: Sat May 23, 2020 11:18 pm
by hhrhhr
Factorio 0.18.26 and 0.18.26, unzipped to ramdisk (created with OSFMount):

Code: Select all

I:\Factorio_0.18.22>ver

Microsoft Windows [Version 10.0.18363.657]

I:\Factorio_0.18.22>bin\x64\factorio.exe -v

I:\Factorio_0.18.22>   0.000 Error Util.cpp:83: weakly_canonical:  .
: "I:\Factorio_0.18.22\bin\x64/../../config\config.ini"
path1: I:\Factorio_0.18.22\bin\x64/../../config\config.ini
   0.000 Error MessageDialog.cpp:140: Failed to widen string: bad conversion
   0.000 Goodbye
No logs appear after launch.

P.S.
If you select the "Physical Drive Emulation" (instead "Logical") option when creating a virtual disk, then no errors occur. This creates an almost "real" disk, which is visible in the Windows "Disk management".

Re: Run from ramdisk

Posted: Sun May 24, 2020 5:28 am
by Rseding91
A little off topic: why do you want to run it from a ramdisk?

Re: Run from ramdisk

Posted: Sun May 24, 2020 9:47 am
by hhrhhr
In my case, the data directory contains almost 10 thousand files. Even after several launches and caching of files by the operating system, the launch of the game takes 10-12 seconds. And when using a RAM disk - only 2-3 seconds. When debugging mods / scripts, this is pretty significant.
Аnother example: the game.save_atlas() function works on a regular disk for 5-6 minutes (at maximum quality), on a RAM disk - no more than a couple of minutes.

Re: Run from ramdisk

Posted: Sun May 24, 2020 2:22 pm
by Rseding91
hhrhhr wrote:
Sun May 24, 2020 9:47 am
In my case, the data directory contains almost 10 thousand files. Even after several launches and caching of files by the operating system, the launch of the game takes 10-12 seconds. And when using a RAM disk - only 2-3 seconds. When debugging mods / scripts, this is pretty significant.
Аnother example: the game.save_atlas() function works on a regular disk for 5-6 minutes (at maximum quality), on a RAM disk - no more than a couple of minutes.
Can you give a log file after loading to the main menu with all those mods? I've never seen things run at that speed.

Also, do you have a spinning hard drive, solid-state, or M2 solid state?

Re: Run from ramdisk

Posted: Sun May 24, 2020 3:23 pm
by hhrhhr
I meant a clean game without mods. At least at 0.18.26, the data directory contains exactly 9999 files and 1503 directories.
Yes, on powerful desktops with fast SSDs, the game’s launch time is not so exciting. But sometimes I have to run the game on an old HP laptop (with Radeon HD4250 video and 256 MB of video memory ;). Installing an SSD is useless, the controller is slow. But I can put 8-16 GB of RAM and use half to create a virtual disk.

Here is the log of the launch of a clean game from RAM-disk with such settings:

Code: Select all

max-texture-size = 8192
graphics-quality = normal
high-quality-animations = false
high-quality-shadows = false
high-quality-terrain = false
texture-compression-level = low-quality

Code: Select all

...
   3.922 Custom mipmaps uploaded.
   3.922 Verbose AtlasSystem.cpp:910: Atlas memory size: 245.54MB; 0.00MB (virtual)
   3.922 Verbose AtlasSystem.cpp:911: Size of sprites outside of atlas: 0.13MB
   3.984 Factorio initialised
...
factorio-current.log
(5.62 KiB) Downloaded 109 times

Re: Run from ramdisk

Posted: Wed May 27, 2020 8:45 pm
by Guenni7
You might want to test the cache settings first, since I activated Atlas-cache and the other cache my game starts in 3 to 4 seconds (from a normal notebook harddrive).
You can access the Cache-settings with CTRL-ALT-LEFTCLICK on the settings-button.

Re: Run from ramdisk

Posted: Thu May 28, 2020 9:29 am
by hhrhhr
config:
cache-sprite-atlas-count=1
cache-sprite-atlas=true
compress-sprite-atlas-cache=true
graphics-quality=normal
high-quality-animations=false
high-quality-shadows=false
high-quality-terrain=false
video-memory-usage=low
texture-compression-level=low-quality
run from SSD, cold start:
...
14.977 Data stage cache loaded in: 14.573237 seconds.
...
16.319 Loading sounds...
...
26.620 Initial atlas bitmap size is 8192
...
27.540 Atlases loaded from disk cache.
...
46.164 Parallel Sprite Loader initialized (threads: 7)
48.464 Sprites loaded
...
48.629 Custom mipmaps uploaded.
48.821 Generated mipmaps for virtual atlas of size 16384x12288
49.342 Factorio initialised
not really helped...

Re: Run from ramdisk

Posted: Thu May 28, 2020 4:01 pm
by Rseding91
hhrhhr wrote:
Thu May 28, 2020 9:29 am
run from SSD, cold start:
not really helped...
That's basically useless. Cold-start anything isn't how you could ever launch from a ramdisk since you have to copy the thing into the ramdisk first anyway - then you could launch it.

Re: Run from ramdisk

Posted: Thu May 28, 2020 4:13 pm
by posila
cache-sprite-atlas=true with video-memory-usage=low won't have much of an effect since most sprites won't be included in the sprite atlases.

Re: Run from ramdisk

Posted: Thu May 28, 2020 5:22 pm
by hhrhhr
Rseding91 wrote:
Thu May 28, 2020 4:01 pm
Cold-start anything isn't how you could ever launch from a ramdisk since you have to copy the thing into the ramdisk first anyway - then you could launch it.
I need to copy the game to ramdisk only once, and a cold start, at least in Windows, does not occur only if the game is launched literally immediately after its release. After a few minutes, the file cache of the system is cleared or replaced and the game starts again by reading files from the disk, and not from the memory.

However, if the game cannot start from certain types of media, then I would like to know in advance which ones. Or this is "not-a-bug"?

Re: Run from ramdisk

Posted: Thu May 28, 2020 9:50 pm
by Rseding91
hhrhhr wrote:
Thu May 28, 2020 5:22 pm
However, if the game cannot start from certain types of media, then I would like to know in advance which ones. Or this is "not-a-bug"?
We don't impose any limits on what storage type files can be on. If you're getting errors they're coming from the C++ filesystem library and are outside of our control (most likely).

Re: Run from ramdisk

Posted: Thu May 28, 2020 10:18 pm
by hhrhhr
Could you provide an example of your code that opens a file using the standard C++ library?
No matter how I tried, but I could not find an example that could not access the file on any types of virtual disks, including file systems with utf8 and utf16 encodings.

Re: Run from ramdisk

Posted: Thu May 28, 2020 11:16 pm
by Rseding91
hhrhhr wrote:
Thu May 28, 2020 10:18 pm
Could you provide an example of your code that opens a file using the standard C++ library?
No matter how I tried, but I could not find an example that could not access the file on any types of virtual disks, including file systems with utf8 and utf16 encodings.
The path is created, and then normalized by calling https://en.cppreference.com/w/cpp/filesystem/canonical - specifically 'weakly_canonical'. It's failing and reports the error you're getting.

Re: Run from ramdisk

Posted: Fri May 29, 2020 9:18 am
by hhrhhr
With this example, everything is fine:

Code: Select all

#include <iostream>
#include <filesystem>
namespace fs = std::filesystem;
int main()
{
    fs::path p = fs::path("..") / ".." / "config-path.cfg";
    std::cout << "Current path is " << fs::current_path() << '\n'
              << "Canonical path for " << p << " is " << fs::canonical(p) << '\n';
}
/*
g++ -std=c++17 -o test.exe test.cpp 
g++ -v
...
Target: x86_64-w64-mingw32
Thread model: posix
gcc version 9.3.0 (MinGW-W64 x86_64-posix-seh)
*/
H: - my ramdrive (created with "Logical emulation")

Code: Select all

h:\Factorio\bin\x64>test.exe
Current path is "h:\\Factorio\\bin\\x64"
Canonical path for "..\\..\\config-path.cfg" is "h:\\Factorio\\config-path.cfg"

h:\Factorio\bin\x64>factorio.exe

h:\Factorio\bin\x64>   0.000 Error Util.cpp:83: weakly_canonical:  .
: "h:\Factorio\bin\x64/../../config\config.ini"
path1: h:\Factorio\bin\x64/../../config\config.ini
   0.000 Error MessageDialog.cpp:140: Failed to widen string: bad conversion
   0.000 Goodbye

Re: Run from ramdisk

Posted: Fri May 29, 2020 3:30 pm
by Rseding91
Rseding91 wrote:
Thu May 28, 2020 11:16 pm
... specifically 'weakly_canonical'.

Re: Run from ramdisk

Posted: Fri May 29, 2020 9:27 pm
by hhrhhr
absolutely no difference:

Code: Select all

#include <iostream>
#include <filesystem>
namespace fs = std::filesystem;
int main()
{
    fs::path p = fs::path("..") / ".." / "config-path.cfg";
    std::cout << "Current path is " << fs::current_path() << '\n'
              << "Canonical path for " << p << " is " << fs::weakly_canonical(p) << '\n';

    std::cout << "file size: " << fs::file_size(fs::weakly_canonical(p)) << " bytes\n";

    p = fs::current_path() / "../../config/config.ini";
    std::cout << "Canonical path for " << p << " is " << fs::weakly_canonical(p) << '\n';

    std::cout << "file size: " << fs::file_size(fs::weakly_canonical(p)) << " bytes\n";
}
reslit:

Code: Select all

h:\Factorio\bin\x64>g++ -std=c++17 -o test.exe -static -s test.c && test.exe
Current path is "h:\\Factorio\\bin\\x64"
Canonical path for "..\\..\\config-path.cfg" is "h:\\Factorio\\config-path.cfg"
file size: 998 bytes
Canonical path for "h:\\Factorio\\bin\\x64\\../../config/config.ini" is "h:\\Factorio\\config\\config.ini"
file size: 26353 bytes

h:\Factorio\bin\x64>factorio.exe -v

h:\Factorio\bin\x64>   0.000 Error Util.cpp:83: weakly_canonical:  .
: "h:\Factorio\bin\x64/../../config\config.ini"
path1: h:\Factorio\bin\x64/../../config\config.ini
   0.000 Error MessageDialog.cpp:140: Failed to widen string: bad conversion
   0.000 Goodbye

Re: Run from ramdisk

Posted: Tue Nov 24, 2020 4:26 pm
by willis936
I'm having the same issue. I originally tried just symlinking the saves directory but found posts saying that it was unsupported and a mess.

Q: imdisk 2.0.10 NTFS drive
Factorio 1.0 zip version.
Unzipped Factorio, launched once, copied directory to RAM drive, then attempted to launch from RAM drive.

imdisk creation:

Code: Select all

C:\Windows\System32\imdisk.exe -a -m Q: -s 4G -p "/fs:ntfs /q /y"
Factorio launch:

Code: Select all

C:\WINDOWS\system32>ver

Microsoft Windows [Version 10.0.18363.1198

C:\WINDOWS\system32>factorio -v

C:\WINDOWS\system32>   0.000 Error Util.cpp:83: weakly_canonical: Incorrect function.
: "Q:\Factorio\bin\x64/../../config\config.ini"
path1: Q:\Factorio\bin\x64/../../config\config.ini
   0.000 Goodbye
Why I want to run from a RAM disk:
I'm hosting a dedicated server so I can play with a friend across the country. I recently discovered that manual artillery shots can go much further than autofiring. I sounded a huge area and now the save file is nearly 100 MB. Autosaving takes several seconds and downloading takes a minute. I've hosted game servers (modded minecraft) on RAM drives in the past with great success. Scheduled tasks sync the files to disk at a regular interval.


Edit: reading above, it looks like I need software that emulates a physical drive. There are no FOSS solutions for this on Windows afaik. OSFMount is free and does work, but it does not have an automation interface (GUI only) and does not support persistent RAM drives. I have a linux VM, but I don't think Factorio performance would take kindly to the memory latency that introduces.

I can launch the server with a save file that is on a logical volume RAM drive, however the autosaving is still done to the game launch directory. The download (over LAN) and autosave is faster, even with the autosaves going to NVMe SSD. I believe this is because the disk only has to write and not read/write.

I tried modifying the [path]write-data option in the game config to be on a logical volume RAM drive, but this gives the same error (Util.cpp:83: weakly_canonical: Incorrect function). Is there another way to manually specify the autosave directory?

I do not think that supporting symlinks or logical volumes is a niche ask. What I’m attempting seems like standard best practice for people hosting game servers (even if they don't talk about it much). I suppose the non-standard thing I'm doing is hosting from Windows. This appears to be out of your hands (likely an OS API bug). If anyone has experience with this, let me know.