Run from ramdisk

[Camylarde]

Hi,

is there a way to move factorio to ramdisk?
I have:
1)Created a ramdisk h:/
2) Moved the steam part of the game to the root.
3) Created a sub-folder config-copy to have a separate location for roaming/factorio config data.
4) Applied the article How to move your data folder from this forum technical section faq.
5) Found the exe in the /bin folder

I still get error
---------------------------
Error
---------------------------
weakly_canonical: Incorrect function.

: "h:/config-copy/config\config.ini"
path1: h:/config-copy/config\config.ini
---------------------------
OK
---------------------------

despite the file confirm.ini being there in the path. Notable is also that if the file isn't there the new one is not created as the article says it should be...

Any suggestions?

[posila]

weakly_canonical: Incorrect function.

Hello, the error you are seeing originates from std::filesystem standard C++ library (which might or might not get it from underlying calls to WinAPI).
Can you please
1) post exact build version of Windows you are running (Win + R and type in winver, or just post Factorio log).
2) decribe steps you used to create the ramdisk
3) post your modified config.ini and command line parameters you use to launch Factorio?

[Optera]

For such experiments I'd start by downloading the portable version directly from factorio.com instead of using the steam version.

[hhrhhr]

Factorio 0.18.26 and 0.18.26, unzipped to ramdisk (created with OSFMount):

Code: Select all

I:\Factorio_0.18.22>ver

Microsoft Windows [Version 10.0.18363.657]

I:\Factorio_0.18.22>bin\x64\factorio.exe -v

I:\Factorio_0.18.22>   0.000 Error Util.cpp:83: weakly_canonical:  .
: "I:\Factorio_0.18.22\bin\x64/../../config\config.ini"
path1: I:\Factorio_0.18.22\bin\x64/../../config\config.ini
   0.000 Error MessageDialog.cpp:140: Failed to widen string: bad conversion
   0.000 Goodbye

No logs appear after launch.

P.S.
If you select the "Physical Drive Emulation" (instead "Logical") option when creating a virtual disk, then no errors occur. This creates an almost "real" disk, which is visible in the Windows "Disk management".

[Rseding91]

A little off topic: why do you want to run it from a ramdisk?

[hhrhhr]

In my case, the data directory contains almost 10 thousand files. Even after several launches and caching of files by the operating system, the launch of the game takes 10-12 seconds. And when using a RAM disk - only 2-3 seconds. When debugging mods / scripts, this is pretty significant.
Аnother example: the game.save_atlas() function works on a regular disk for 5-6 minutes (at maximum quality), on a RAM disk - no more than a couple of minutes.

[Rseding91]

In my case, the data directory contains almost 10 thousand files. Even after several launches and caching of files by the operating system, the launch of the game takes 10-12 seconds. And when using a RAM disk - only 2-3 seconds. When debugging mods / scripts, this is pretty significant.
Аnother example: the game.save_atlas() function works on a regular disk for 5-6 minutes (at maximum quality), on a RAM disk - no more than a couple of minutes.

Can you give a log file after loading to the main menu with all those mods? I've never seen things run at that speed.

Also, do you have a spinning hard drive, solid-state, or M2 solid state?

[hhrhhr]

I meant a clean game without mods. At least at 0.18.26, the data directory contains exactly 9999 files and 1503 directories.
Yes, on powerful desktops with fast SSDs, the game’s launch time is not so exciting. But sometimes I have to run the game on an old HP laptop (with Radeon HD4250 video and 256 MB of video memory ;). Installing an SSD is useless, the controller is slow. But I can put 8-16 GB of RAM and use half to create a virtual disk.

Here is the log of the launch of a clean game from RAM-disk with such settings:

Code: Select all

max-texture-size = 8192
graphics-quality = normal
high-quality-animations = false
high-quality-shadows = false
high-quality-terrain = false
texture-compression-level = low-quality

Code: Select all

...
   3.922 Custom mipmaps uploaded.
   3.922 Verbose AtlasSystem.cpp:910: Atlas memory size: 245.54MB; 0.00MB (virtual)
   3.922 Verbose AtlasSystem.cpp:911: Size of sprites outside of atlas: 0.13MB
   3.984 Factorio initialised
...

factorio-current.log

(5.62 KiB) Downloaded 138 times

[Guenni7]

You might want to test the cache settings first, since I activated Atlas-cache and the other cache my game starts in 3 to 4 seconds (from a normal notebook harddrive).
You can access the Cache-settings with CTRL-ALT-LEFTCLICK on the settings-button.

[hhrhhr]

config:

cache-sprite-atlas-count=1
cache-sprite-atlas=true
compress-sprite-atlas-cache=true
graphics-quality=normal
high-quality-animations=false
high-quality-shadows=false
high-quality-terrain=false
video-memory-usage=low
texture-compression-level=low-quality

run from SSD, cold start:

...
14.977 Data stage cache loaded in: 14.573237 seconds.
...
16.319 Loading sounds...
...
26.620 Initial atlas bitmap size is 8192
...
27.540 Atlases loaded from disk cache.
...
46.164 Parallel Sprite Loader initialized (threads: 7)
48.464 Sprites loaded
...
48.629 Custom mipmaps uploaded.
48.821 Generated mipmaps for virtual atlas of size 16384x12288
49.342 Factorio initialised

not really helped...

[Rseding91]

run from SSD, cold start:
not really helped...

That's basically useless. Cold-start anything isn't how you could ever launch from a ramdisk since you have to copy the thing into the ramdisk first anyway - then you could launch it.

[posila]

cache-sprite-atlas=true with video-memory-usage=low won't have much of an effect since most sprites won't be included in the sprite atlases.

[hhrhhr]

Cold-start anything isn't how you could ever launch from a ramdisk since you have to copy the thing into the ramdisk first anyway - then you could launch it.

I need to copy the game to ramdisk only once, and a cold start, at least in Windows, does not occur only if the game is launched literally immediately after its release. After a few minutes, the file cache of the system is cleared or replaced and the game starts again by reading files from the disk, and not from the memory.

However, if the game cannot start from certain types of media, then I would like to know in advance which ones. Or this is "not-a-bug"?

[Rseding91]

However, if the game cannot start from certain types of media, then I would like to know in advance which ones. Or this is "not-a-bug"?

We don't impose any limits on what storage type files can be on. If you're getting errors they're coming from the C++ filesystem library and are outside of our control (most likely).

[hhrhhr]

Could you provide an example of your code that opens a file using the standard C++ library?
No matter how I tried, but I could not find an example that could not access the file on any types of virtual disks, including file systems with utf8 and utf16 encodings.

[Rseding91]

Could you provide an example of your code that opens a file using the standard C++ library?
No matter how I tried, but I could not find an example that could not access the file on any types of virtual disks, including file systems with utf8 and utf16 encodings.

The path is created, and then normalized by calling https://en.cppreference.com/w/cpp/filesystem/canonical - specifically 'weakly_canonical'. It's failing and reports the error you're getting.

[hhrhhr]

With this example, everything is fine:

Code: Select all

#include <iostream>
#include <filesystem>
namespace fs = std::filesystem;
int main()
{
    fs::path p = fs::path("..") / ".." / "config-path.cfg";
    std::cout << "Current path is " << fs::current_path() << '\n'
              << "Canonical path for " << p << " is " << fs::canonical(p) << '\n';
}
/*
g++ -std=c++17 -o test.exe test.cpp 
g++ -v
...
Target: x86_64-w64-mingw32
Thread model: posix
gcc version 9.3.0 (MinGW-W64 x86_64-posix-seh)
*/

H: - my ramdrive (created with "Logical emulation")

Code: Select all

h:\Factorio\bin\x64>test.exe
Current path is "h:\\Factorio\\bin\\x64"
Canonical path for "..\\..\\config-path.cfg" is "h:\\Factorio\\config-path.cfg"

h:\Factorio\bin\x64>factorio.exe

h:\Factorio\bin\x64>   0.000 Error Util.cpp:83: weakly_canonical:  .
: "h:\Factorio\bin\x64/../../config\config.ini"
path1: h:\Factorio\bin\x64/../../config\config.ini
   0.000 Error MessageDialog.cpp:140: Failed to widen string: bad conversion
   0.000 Goodbye

[Rseding91]

... specifically 'weakly_canonical'.

[hhrhhr]

absolutely no difference:

Code: Select all

#include <iostream>
#include <filesystem>
namespace fs = std::filesystem;
int main()
{
    fs::path p = fs::path("..") / ".." / "config-path.cfg";
    std::cout << "Current path is " << fs::current_path() << '\n'
              << "Canonical path for " << p << " is " << fs::weakly_canonical(p) << '\n';

    std::cout << "file size: " << fs::file_size(fs::weakly_canonical(p)) << " bytes\n";

    p = fs::current_path() / "../../config/config.ini";
    std::cout << "Canonical path for " << p << " is " << fs::weakly_canonical(p) << '\n';

    std::cout << "file size: " << fs::file_size(fs::weakly_canonical(p)) << " bytes\n";
}

reslit:

Code: Select all

h:\Factorio\bin\x64>g++ -std=c++17 -o test.exe -static -s test.c && test.exe
Current path is "h:\\Factorio\\bin\\x64"
Canonical path for "..\\..\\config-path.cfg" is "h:\\Factorio\\config-path.cfg"
file size: 998 bytes
Canonical path for "h:\\Factorio\\bin\\x64\\../../config/config.ini" is "h:\\Factorio\\config\\config.ini"
file size: 26353 bytes

h:\Factorio\bin\x64>factorio.exe -v

h:\Factorio\bin\x64>   0.000 Error Util.cpp:83: weakly_canonical:  .
: "h:\Factorio\bin\x64/../../config\config.ini"
path1: h:\Factorio\bin\x64/../../config\config.ini
   0.000 Error MessageDialog.cpp:140: Failed to widen string: bad conversion
   0.000 Goodbye

[willis936]

I'm having the same issue. I originally tried just symlinking the saves directory but found posts saying that it was unsupported and a mess.

Q: imdisk 2.0.10 NTFS drive
Factorio 1.0 zip version.
Unzipped Factorio, launched once, copied directory to RAM drive, then attempted to launch from RAM drive.

imdisk creation:

Code: Select all

C:\Windows\System32\imdisk.exe -a -m Q: -s 4G -p "/fs:ntfs /q /y"

Factorio launch:

Code: Select all

C:\WINDOWS\system32>ver

Microsoft Windows [Version 10.0.18363.1198

C:\WINDOWS\system32>factorio -v

C:\WINDOWS\system32>   0.000 Error Util.cpp:83: weakly_canonical: Incorrect function.
: "Q:\Factorio\bin\x64/../../config\config.ini"
path1: Q:\Factorio\bin\x64/../../config\config.ini
   0.000 Goodbye

Why I want to run from a RAM disk:
I'm hosting a dedicated server so I can play with a friend across the country. I recently discovered that manual artillery shots can go much further than autofiring. I sounded a huge area and now the save file is nearly 100 MB. Autosaving takes several seconds and downloading takes a minute. I've hosted game servers (modded minecraft) on RAM drives in the past with great success. Scheduled tasks sync the files to disk at a regular interval.


Edit: reading above, it looks like I need software that emulates a physical drive. There are no FOSS solutions for this on Windows afaik. OSFMount is free and does work, but it does not have an automation interface (GUI only) and does not support persistent RAM drives. I have a linux VM, but I don't think Factorio performance would take kindly to the memory latency that introduces.

I can launch the server with a save file that is on a logical volume RAM drive, however the autosaving is still done to the game launch directory. The download (over LAN) and autosave is faster, even with the autosaves going to NVMe SSD. I believe this is because the disk only has to write and not read/write.

I tried modifying the [path]write-data option in the game config to be on a logical volume RAM drive, but this gives the same error (Util.cpp:83: weakly_canonical: Incorrect function). Is there another way to manually specify the autosave directory?

I do not think that supporting symlinks or logical volumes is a niche ask. What I’m attempting seems like standard best practice for people hosting game servers (even if they don't talk about it much). I suppose the non-standard thing I'm doing is hosting from Windows. This appears to be out of your hands (likely an OS API bug). If anyone has experience with this, let me know.