Friday Facts #390 - Noise expressions 2.0

Regular reports on Factorio development.
Tricorius
Filter Inserter
Filter Inserter
Posts: 279
Joined: Fri Jul 01, 2016 9:04 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Tricorius »

Shadow_Man wrote: ↑
Fri Dec 22, 2023 12:45 pm
Hou-hou-hou!
We really need that snowy sprites and Christmas Tree in-game!
I don’t know about the Christmas tree, BUT we got a lava planet. An ice planet isn’t exactly crazy anymore. ;)

User avatar
CaseyFox
Manual Inserter
Manual Inserter
Posts: 2
Joined: Sun Jan 05, 2020 6:11 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by CaseyFox »

fff-390-white-noise.png
fff-390-white-noise.png (295.06 KiB) Viewed 3320 times
fff-390-white-noise_Text.png
fff-390-white-noise_Text.png (17.77 KiB) Viewed 3320 times
I thought I saw letters in this. I was right XD

User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Earendel »

CaseyFox wrote: ↑
Sat Dec 23, 2023 1:58 am
fff-390-white-noise_Text.png

I thought I saw letters in this. I was right XD
He he, you found it it.
planetfall wrote: ↑
Fri Dec 22, 2023 11:56 pm
the way i see it, there are three possibilities for generating things that can appear on multiple planets:

- things that autoplace can have multiple sets of placement rules, varying per surface
- things that can be autoplaced on multiple surfaces have an obscenely complicated autoplace system that takes into account every planet's environment, it's just that all the "what vulcanus biome is this" values are 0 when looking at nauvis and so on.
- nauvis coal deposits and vulcanus coal deposits are secretly different entities that look and behave identically once spawned but have different autoplace rules

any hint as to which it is, or is this a secret?
None of the above, but option 1 is closest. Each placable thing has a built-in autoplace specification that it will use if included on a planet. When a planet chooses to include a thing, it can also choose to override the autoplace specification with any noise expression it wants.

continue2021
Manual Inserter
Manual Inserter
Posts: 2
Joined: Fri Nov 17, 2023 1:10 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by continue2021 »

A FFF that is not very suitable for this time, I have a question, will a radio device for transmitting signals be added in the future? Similar to the transmitter and receiver in the Space Exploration module

User avatar
Drury
Filter Inserter
Filter Inserter
Posts: 794
Joined: Tue Mar 25, 2014 8:01 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Drury »

Snow covered belts is quite the paradox - unless they're perfectly stationary for the holidays ;)

whistlepark
Manual Inserter
Manual Inserter
Posts: 1
Joined: Sat Dec 23, 2023 6:54 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by whistlepark »

Love the technical c++ and compiler bits. Really curious about your debugging steps and how you found those bottlenecks. Great read!

Qon
Smart Inserter
Smart Inserter
Posts: 2160
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Qon »

Tooster wrote: ↑
Fri Dec 22, 2023 12:58 pm
As for tools for visualizing: are you aware of the existence of the Dear ImGui? If you are using it, that's great! If you are not β€” both you and all the modders could be interested!
It does look neat. But I'm not modding the game in C++, so I wont have a use fot it yet.
Tooster wrote: ↑
Fri Dec 22, 2023 12:58 pm
As for noise expression - I am not quite sure I understand the need for the "noise expression engine" altogether, the post is a bit hard to understand and it's not very clear what it can do that simply writing functions itself wouldn't. For example description of implementing things like "constant folding" raise some red flags for me β€” this sounds too familiar to what compilers already do to optimize, so why does it have to be reinvented :?:
The noise expressions are compiled to run fast. The noise language is a Domain Specific Language (DSL) so they can't use an already existing compiler for another language, compilers are made for specific languages. "Simply writing that function" is what we can do now, now that they have a compiler for the language. Whatever language it is written in would need a compiler. C++ and C# compilers are already written but seem overkill to include with the game for a simple noise expression and the syntax and semantics for the DSL would have to match a subset of those languages.
(EDIT: The language is Lua, or a subset. So it is "just writing that function", (EDIT2: The language is not Lua, just defined in Lua with strings. Check for yourself))

Yes compilers do constant folding, but they aren't reinventing constant folding, they are reimplementing it. Which is typical for writing a new compiler. It's not wrong to write your own software.
Tooster wrote: ↑
Fri Dec 22, 2023 12:58 pm
And as for the AST and parser part β€” you might be also interested in Roslyn (C#) compiler and how they managed to make some good optimizations and design their compiler and syntax APIs β€” for example looking at the previous lua table syntax and "line_number=..." reminded me of some of the techniques they used to make their syntax trees more robust β€” using red-green trees for immutable trees (one, internal, constructed bottom-up, one, a facade, lazy evaluated with parent references and file offsets), using node-widths to compute file-offsets etc. They came up with some cool ideas, techniques and optimizations that you may find interesting.
Roslyn is for C#, which the noise expressions aren't written in. Maybe that would be a possibility, but if you want to add language features or not include other features of other languages like C# then you get issues. Roslyn giving you the syntax tree seems like it would make it convinient to remove some features, but that might still be tricky and a lot of work if you just want mathematical expressions + some more minor features.
Tooster wrote: ↑
Fri Dec 22, 2023 12:58 pm
As for the grammar part β€” recently a very nice tool came to my attention, that handles grammars, generating parsers etc, and people are generally very pleased with it: Tree-Sitter. It's grammars are supposedly very powerful and said to be working great, to the point that there is an open discussion whether VSCode should use tree-sitter grammars instead of what they use now. Although hand-written parser/lexers/tokenizers can be faster, they also may be a lot harder to maintain, test and fix.
I heard about Tree-Sitter in the context "Cursorless: A spoken language for editing code" by Pokey Rule (Strange Loop 2023). Seems like a cool tool.
I'm writing compilers for my own projects, and I've considered using a parser generator.
Tree-sitter uses Node.js to generate the parser, which is ok, though I recently switched to Deno for my own js programs.
But it outputs a C library or DLL, which I can't include in a website (Well, I'm pretty sure there are C compilers that outputs JavaScript, but I don't feel like setting that up now). But it sounds like it would be a good inclusion for Factorio then.
Last edited by Qon on Sat Dec 23, 2023 9:31 am, edited 2 times in total.
My mods: Capsule Ammo | HandyHands - Automatic handcrafting | ChunkyChunks - Configurable Gridlines
Some other creations: Combinassembly Language GitHub w instructions and link to run it in your browser | 0~drain Laser

Qon
Smart Inserter
Smart Inserter
Posts: 2160
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Qon »

Hares wrote: ↑
Fri Dec 22, 2023 1:46 pm
I have a VERY bold suggestion.
Can we use an AST parser for Arithmetic Combinators?

It would be super-nice to write the whole expressions like this:

Code: Select all

signal_EACH*2 - 3*signal_L
(where signal_EACH & signal_L use respective icons)
I'm currently writing an "assembly" style language, the output of the assembler is Factorio combinators. It's not quite that, but I'm considering also making other kinds of languages that would do things like that next...
My mods: Capsule Ammo | HandyHands - Automatic handcrafting | ChunkyChunks - Configurable Gridlines
Some other creations: Combinassembly Language GitHub w instructions and link to run it in your browser | 0~drain Laser

Svip
Fast Inserter
Fast Inserter
Posts: 101
Joined: Sun Apr 29, 2018 6:19 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Svip »

Qon wrote: ↑
Sat Dec 23, 2023 8:54 am
Tooster wrote: ↑
Fri Dec 22, 2023 12:58 pm
As for the grammar part β€” recently a very nice tool came to my attention, that handles grammars, generating parsers etc, and people are generally very pleased with it: Tree-Sitter. It's grammars are supposedly very powerful and said to be working great, to the point that there is an open discussion whether VSCode should use tree-sitter grammars instead of what they use now. Although hand-written parser/lexers/tokenizers can be faster, they also may be a lot harder to maintain, test and fix.
I heard about Tree-Sitter in the context "Cursorless: A spoken language for editing code" by Pokey Rule (Strange Loop 2023). Seems like a cool tool.
I'm writing compilers for my own projects, and I've considered using a parser generator.
Tree-sitter uses Node.js to generate the parser, which is ok, though I recently switched to Deno for my own js programs.
But it outputs a C library or DLL, which I can't include in a website (Well, I'm pretty sure there are C compilers that outputs JavaScript, but I don't feel like setting that up now). But it sounds like it would be a good inclusion for Factorio then.
Except Tree-Sitter doesn't have C++ language bindings, meaning they would have to write their own or rely on an external library, which would make development difficult. Remember, Factorio is available on macOS, Linux and Windows, so a DLL solution is not a proper solution. And I am not sure how this works on the Switch either.
Last edited by Svip on Sat Dec 23, 2023 9:04 am, edited 1 time in total.

Qon
Smart Inserter
Smart Inserter
Posts: 2160
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Qon »

Turbofant wrote: ↑
Fri Dec 22, 2023 1:56 pm
That looks incredible! :o

Creating a snow covered version for that many entities is quite a lot of work for a Christmas card.

So I guess this screenshot wasn't taken on Nauvis ;)
It wasn't, it was hand painted by Earendel according to a redditor, (a website without proper search functionality).
My mods: Capsule Ammo | HandyHands - Automatic handcrafting | ChunkyChunks - Configurable Gridlines
Some other creations: Combinassembly Language GitHub w instructions and link to run it in your browser | 0~drain Laser

Qon
Smart Inserter
Smart Inserter
Posts: 2160
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Qon »

Svip wrote: ↑
Sat Dec 23, 2023 9:04 am
Qon wrote: ↑
Sat Dec 23, 2023 8:54 am
But it outputs a C library or DLL,
Except Tree-Sitter doesn't have C++ language bindings, meaning they would have to write their own or rely on an external library, which would make development difficult. Remember, Factorio is available on macOS, Linux and Windows, so a DLL solution is not a proper solution. And I am not sure how this works on the Switch either.
I just skimmed through the tree-sitter. I didn't say it outputs a DLL, I said it outputs DLL or C library. Because I didn't know what it made.
It needs a C/C++ compiler
A C Compiler - Tree-sitter creates parsers that are written in C. In order to run and test these parsers with the tree-sitter parse or tree-sitter test commands, you must have a C/C++ compiler installed. Tree-sitter will try to look for these compilers in the standard places for each platform.
So I think they could just include the C output in their source and be done with it potentially? C is kind of a subset of C++, so maybe it would work? Or maybe it can output C++ also then?
My mods: Capsule Ammo | HandyHands - Automatic handcrafting | ChunkyChunks - Configurable Gridlines
Some other creations: Combinassembly Language GitHub w instructions and link to run it in your browser | 0~drain Laser

Justderpingalong
Long Handed Inserter
Long Handed Inserter
Posts: 75
Joined: Wed Mar 08, 2017 3:34 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Justderpingalong »

This was an incredibly technical post and I can understand it's not for everyone: Most people just like to see the new goodies. I myself read it, still don't quite understand it though I got the general idea...

But I did want to point out one specific sentence that just... gave me joy.
My goal was to make the compiler aid modders with the optimisation step.
I just love that. I'm not a modder myself but this is a mindset I believe the entire gaming industry could learn from. This is why factorio is such an amazing game: Not just because the game itself is just amazingly designed; but because when redesigning and optimising the game, you think not just about the community that is playing your game, but also about the community that is modding your game, bringing people even more content to enjoy within your game. So thank you Wube, for thinking about everyone. Merry Christmas and a happy new year everyone!

(Also who else thinks a christmas reskin of the game would be incredibly funny to mess around with?)

User avatar
Mmmmmmorten
Long Handed Inserter
Long Handed Inserter
Posts: 88
Joined: Sat Mar 30, 2019 6:04 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Mmmmmmorten »

Impact on 1.1 ... or is that 1.2?
Reading this wonderful post made me think of an earlier info that 2.0 dlc (much anticipated) will have to update core 1.1 game engine also.
Just wondering how much creating 2.0 will impact 1.1 performance? And this has to be a core change for the game and not just the dlc?
What decides what changes are done to core or just dlc?
M

FuryoftheStars
Smart Inserter
Smart Inserter
Posts: 2768
Joined: Tue Apr 25, 2017 2:01 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by FuryoftheStars »

Mmmmmmorten wrote: ↑
Sat Dec 23, 2023 10:38 am
Impact on 1.1 ... or is that 1.2?
Reading this wonderful post made me think of an earlier info that 2.0 dlc (much anticipated) will have to update core 1.1 game engine also.
Just wondering how much creating 2.0 will impact 1.1 performance? And this has to be a core change for the game and not just the dlc?
What decides what changes are done to core or just dlc?
M
These types of changes are to their engine (anything C++ definitely is), so when 2.0 and the expansion are released, these changes will be a part of the free 2.0 update.

As for 1.1 performance, I'm not sure what you're asking? None of these changes will be for 1.1. We'll all be updated to 2.0 just like for any other update when it drops. Though like for any other major update, they'll probably leave the last 1.1.x on Steam beta for those that don't want to update.
My Mods: Classic Factorio Basic Oil Processing | Sulfur Production from Oils | Wood to Oil Processing | Infinite Resources - Normal Yield | Tree Saplings (Redux) | Alien Biomes Tweaked | Restrictions on Artificial Tiles | New Gear Girl & HR Graphics

Ext3h
Long Handed Inserter
Long Handed Inserter
Posts: 67
Joined: Mon May 19, 2014 12:40 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Ext3h »

I even considered using an external grammar tool which would generate the parser, but I didn't think it would be worth the time to learn using it and the produced code would likely be suboptimal. In the end, I decided to go for an in-house solution.
You really should have given the classic Flex+Bison pair a chance. It's like a 2 day learning curve to get lexer + grammar + C++ bindings down (including the necessary toggle from static variables to context object), and from then on it's easily maintainable.

I guess doing it manually must have cost you about a week or two - and you probably do NOT have user friendly debug output yet in case an expression was provided that doesn't compile due to syntax errors.

Illegal use of tokens is a pain in the ass to deal with manually.

Svip
Fast Inserter
Fast Inserter
Posts: 101
Joined: Sun Apr 29, 2018 6:19 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Svip »

Qon wrote: ↑
Sat Dec 23, 2023 9:09 am
Svip wrote: ↑
Sat Dec 23, 2023 9:04 am
Except Tree-Sitter doesn't have C++ language bindings, meaning they would have to write their own or rely on an external library, which would make development difficult. Remember, Factorio is available on macOS, Linux and Windows, so a DLL solution is not a proper solution. And I am not sure how this works on the Switch either.
I just skimmed through the tree-sitter. I didn't say it outputs a DLL, I said it outputs DLL or C library. Because I didn't know what it made.
It needs a C/C++ compiler

So I think they could just include the C output in their source and be done with it potentially? C is kind of a subset of C++, so maybe it would work? Or maybe it can output C++ also then?
I feel kind of silly now, I failed to recognise Tree-Sitter is written in pure C. No additional compiler is necessary, using a C library in C++ is trivial. C++ is generally speaking a superset of C, although over the decades the two have diverged, but this way they should be able to compile Tree-Sitter straight into Factorio.

coppercoil
Filter Inserter
Filter Inserter
Posts: 494
Joined: Tue Jun 26, 2018 10:14 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by coppercoil »

I want to learn more about this noise:

noise.png
noise.png (38.66 KiB) Viewed 2829 times

FasterJump
Fast Inserter
Fast Inserter
Posts: 231
Joined: Sat Jul 09, 2016 11:43 am
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by FasterJump »

Drury wrote: ↑
Sat Dec 23, 2023 6:53 am
Snow covered belts is quite the paradox - unless they're perfectly stationary for the holidays ;)
A snowstorm has stopped all belts from working and blown away all the items being transported?

Ext3h
Long Handed Inserter
Long Handed Inserter
Posts: 67
Joined: Mon May 19, 2014 12:40 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Ext3h »

coppercoil wrote: ↑
Sat Dec 23, 2023 10:46 pm
I want to learn more about this noise:


noise.png
That had been explained in the post - it's the term "0.1 - abs(perlin() - 0.5) > 0" to form those rivers in the spots where the gradient in the original noise goes through 0. First constant controls the width of the pathways, second constant whether it becomes a connected web, or just a bunch of disconnected lines.

Mathematician
Inserter
Inserter
Posts: 36
Joined: Fri Jan 18, 2019 6:58 pm
Contact:

Re: Friday Facts #390 - Noise expressions 2.0

Post by Mathematician »

:o
Merry Christmas πŸŽ„
Some people say math is useless in life. I say life is useless in math.

Post Reply

Return to β€œNews”