Friday Facts #390 - Noise expressions 2.0

[Tricorius]

Hou-hou-hou!
We really need that snowy sprites and Christmas Tree in-game!

I don’t know about the Christmas tree, BUT we got a lava planet. An ice planet isn’t exactly crazy anymore.

[CaseyFox]

fff-390-white-noise.png (295.06 KiB) Viewed 3833 times

fff-390-white-noise_Text.png (17.77 KiB) Viewed 3833 times

I thought I saw letters in this. I was right XD

[Earendel]

fff-390-white-noise_Text.png

I thought I saw letters in this. I was right XD

He he, you found it it.

the way i see it, there are three possibilities for generating things that can appear on multiple planets:

- things that autoplace can have multiple sets of placement rules, varying per surface
- things that can be autoplaced on multiple surfaces have an obscenely complicated autoplace system that takes into account every planet's environment, it's just that all the "what vulcanus biome is this" values are 0 when looking at nauvis and so on.
- nauvis coal deposits and vulcanus coal deposits are secretly different entities that look and behave identically once spawned but have different autoplace rules

any hint as to which it is, or is this a secret?

None of the above, but option 1 is closest. Each placable thing has a built-in autoplace specification that it will use if included on a planet. When a planet chooses to include a thing, it can also choose to override the autoplace specification with any noise expression it wants.

[continue2021]

A FFF that is not very suitable for this time, I have a question, will a radio device for transmitting signals be added in the future? Similar to the transmitter and receiver in the Space Exploration module

[Drury]

Snow covered belts is quite the paradox - unless they're perfectly stationary for the holidays

[whistlepark]

Love the technical c++ and compiler bits. Really curious about your debugging steps and how you found those bottlenecks. Great read!

[Qon]

As for tools for visualizing: are you aware of the existence of the Dear ImGui? If you are using it, that's great! If you are not — both you and all the modders could be interested!

It does look neat. But I'm not modding the game in C++, so I wont have a use fot it yet.

As for noise expression - I am not quite sure I understand the need for the "noise expression engine" altogether, the post is a bit hard to understand and it's not very clear what it can do that simply writing functions itself wouldn't. For example description of implementing things like "constant folding" raise some red flags for me — this sounds too familiar to what compilers already do to optimize, so why does it have to be reinvented

The noise expressions are compiled to run fast. The noise language is a Domain Specific Language (DSL) so they can't use an already existing compiler for another language, compilers are made for specific languages. "Simply writing that function" is what we can do now, now that they have a compiler for the language. Whatever language it is written in would need a compiler. C++ and C# compilers are already written but seem overkill to include with the game for a simple noise expression and the syntax and semantics for the DSL would have to match a subset of those languages.
(EDIT: The language is Lua, or a subset. So it is "just writing that function", (EDIT2: The language is not Lua, just defined in Lua with strings. Check for yourself))

Yes compilers do constant folding, but they aren't reinventing constant folding, they are reimplementing it. Which is typical for writing a new compiler. It's not wrong to write your own software.

And as for the AST and parser part — you might be also interested in Roslyn (C#) compiler and how they managed to make some good optimizations and design their compiler and syntax APIs — for example looking at the previous lua table syntax and "line_number=..." reminded me of some of the techniques they used to make their syntax trees more robust — using red-green trees for immutable trees (one, internal, constructed bottom-up, one, a facade, lazy evaluated with parent references and file offsets), using node-widths to compute file-offsets etc. They came up with some cool ideas, techniques and optimizations that you may find interesting.

Roslyn is for C#, which the noise expressions aren't written in. Maybe that would be a possibility, but if you want to add language features or not include other features of other languages like C# then you get issues. Roslyn giving you the syntax tree seems like it would make it convinient to remove some features, but that might still be tricky and a lot of work if you just want mathematical expressions + some more minor features.

As for the grammar part — recently a very nice tool came to my attention, that handles grammars, generating parsers etc, and people are generally very pleased with it: Tree-Sitter. It's grammars are supposedly very powerful and said to be working great, to the point that there is an open discussion whether VSCode should use tree-sitter grammars instead of what they use now. Although hand-written parser/lexers/tokenizers can be faster, they also may be a lot harder to maintain, test and fix.

I heard about Tree-Sitter in the context "Cursorless: A spoken language for editing code" by Pokey Rule (Strange Loop 2023). Seems like a cool tool.
I'm writing compilers for my own projects, and I've considered using a parser generator.
Tree-sitter uses Node.js to generate the parser, which is ok, though I recently switched to Deno for my own js programs.
But it outputs a C library or DLL, which I can't include in a website (Well, I'm pretty sure there are C compilers that outputs JavaScript, but I don't feel like setting that up now). But it sounds like it would be a good inclusion for Factorio then.

[Qon]

I have a VERY bold suggestion.
Can we use an AST parser for Arithmetic Combinators?

It would be super-nice to write the whole expressions like this:

Code: Select all

signal_EACH*2 - 3*signal_L

(where signal_EACH & signal_L use respective icons)

I'm currently writing an "assembly" style language, the output of the assembler is Factorio combinators. It's not quite that, but I'm considering also making other kinds of languages that would do things like that next...

[Svip]

As for the grammar part — recently a very nice tool came to my attention, that handles grammars, generating parsers etc, and people are generally very pleased with it: Tree-Sitter. It's grammars are supposedly very powerful and said to be working great, to the point that there is an open discussion whether VSCode should use tree-sitter grammars instead of what they use now. Although hand-written parser/lexers/tokenizers can be faster, they also may be a lot harder to maintain, test and fix.

I heard about Tree-Sitter in the context "Cursorless: A spoken language for editing code" by Pokey Rule (Strange Loop 2023). Seems like a cool tool.
I'm writing compilers for my own projects, and I've considered using a parser generator.
Tree-sitter uses Node.js to generate the parser, which is ok, though I recently switched to Deno for my own js programs.
But it outputs a C library or DLL, which I can't include in a website (Well, I'm pretty sure there are C compilers that outputs JavaScript, but I don't feel like setting that up now). But it sounds like it would be a good inclusion for Factorio then.

Except Tree-Sitter doesn't have C++ language bindings, meaning they would have to write their own or rely on an external library, which would make development difficult. Remember, Factorio is available on macOS, Linux and Windows, so a DLL solution is not a proper solution. And I am not sure how this works on the Switch either.

[Qon]

That looks incredible!

Creating a snow covered version for that many entities is quite a lot of work for a Christmas card.

So I guess this screenshot wasn't taken on Nauvis

It wasn't, it was hand painted by Earendel according to a redditor, (a website without proper search functionality).

[Qon]

But it outputs a C library or DLL,

Except Tree-Sitter doesn't have C++ language bindings, meaning they would have to write their own or rely on an external library, which would make development difficult. Remember, Factorio is available on macOS, Linux and Windows, so a DLL solution is not a proper solution. And I am not sure how this works on the Switch either.

I just skimmed through the tree-sitter. I didn't say it outputs a DLL, I said it outputs DLL or C library. Because I didn't know what it made.
It needs a C/C++ compiler

A C Compiler - Tree-sitter creates parsers that are written in C. In order to run and test these parsers with the tree-sitter parse or tree-sitter test commands, you must have a C/C++ compiler installed. Tree-sitter will try to look for these compilers in the standard places for each platform.

So I think they could just include the C output in their source and be done with it potentially? C is kind of a subset of C++, so maybe it would work? Or maybe it can output C++ also then?

[Justderpingalong]

This was an incredibly technical post and I can understand it's not for everyone: Most people just like to see the new goodies. I myself read it, still don't quite understand it though I got the general idea...

But I did want to point out one specific sentence that just... gave me joy.

My goal was to make the compiler aid modders with the optimisation step.

I just love that. I'm not a modder myself but this is a mindset I believe the entire gaming industry could learn from. This is why factorio is such an amazing game: Not just because the game itself is just amazingly designed; but because when redesigning and optimising the game, you think not just about the community that is playing your game, but also about the community that is modding your game, bringing people even more content to enjoy within your game. So thank you Wube, for thinking about everyone. Merry Christmas and a happy new year everyone!

(Also who else thinks a christmas reskin of the game would be incredibly funny to mess around with?)

[Mmmmmmorten]

Impact on 1.1 ... or is that 1.2?
Reading this wonderful post made me think of an earlier info that 2.0 dlc (much anticipated) will have to update core 1.1 game engine also.
Just wondering how much creating 2.0 will impact 1.1 performance? And this has to be a core change for the game and not just the dlc?
What decides what changes are done to core or just dlc?
M

[FuryoftheStars]

Impact on 1.1 ... or is that 1.2?
Reading this wonderful post made me think of an earlier info that 2.0 dlc (much anticipated) will have to update core 1.1 game engine also.
Just wondering how much creating 2.0 will impact 1.1 performance? And this has to be a core change for the game and not just the dlc?
What decides what changes are done to core or just dlc?
M

These types of changes are to their engine (anything C++ definitely is), so when 2.0 and the expansion are released, these changes will be a part of the free 2.0 update.

As for 1.1 performance, I'm not sure what you're asking? None of these changes will be for 1.1. We'll all be updated to 2.0 just like for any other update when it drops. Though like for any other major update, they'll probably leave the last 1.1.x on Steam beta for those that don't want to update.

[Ext3h]

I even considered using an external grammar tool which would generate the parser, but I didn't think it would be worth the time to learn using it and the produced code would likely be suboptimal. In the end, I decided to go for an in-house solution.

You really should have given the classic Flex+Bison pair a chance. It's like a 2 day learning curve to get lexer + grammar + C++ bindings down (including the necessary toggle from static variables to context object), and from then on it's easily maintainable.

I guess doing it manually must have cost you about a week or two - and you probably do NOT have user friendly debug output yet in case an expression was provided that doesn't compile due to syntax errors.

Illegal use of tokens is a pain in the ass to deal with manually.

[Svip]

Except Tree-Sitter doesn't have C++ language bindings, meaning they would have to write their own or rely on an external library, which would make development difficult. Remember, Factorio is available on macOS, Linux and Windows, so a DLL solution is not a proper solution. And I am not sure how this works on the Switch either.

I just skimmed through the tree-sitter. I didn't say it outputs a DLL, I said it outputs DLL or C library. Because I didn't know what it made.
It needs a C/C++ compiler

So I think they could just include the C output in their source and be done with it potentially? C is kind of a subset of C++, so maybe it would work? Or maybe it can output C++ also then?

I feel kind of silly now, I failed to recognise Tree-Sitter is written in pure C. No additional compiler is necessary, using a C library in C++ is trivial. C++ is generally speaking a superset of C, although over the decades the two have diverged, but this way they should be able to compile Tree-Sitter straight into Factorio.

[coppercoil]

I want to learn more about this noise:

noise.png (38.66 KiB) Viewed 3342 times

[FasterJump]

Snow covered belts is quite the paradox - unless they're perfectly stationary for the holidays

A snowstorm has stopped all belts from working and blown away all the items being transported?

[Ext3h]

I want to learn more about this noise:


noise.png

That had been explained in the post - it's the term "0.1 - abs(perlin() - 0.5) > 0" to form those rivers in the spots where the gradient in the original noise goes through 0. First constant controls the width of the pathways, second constant whether it becomes a connected web, or just a bunch of disconnected lines.

[Mathematician]

Merry Christmas