Programmable Speaker Improvements

[Illiander42]

I didn't realise you're the MIDItorio guy

I wonder if we could convince the devs to put a 108-key piano with cyclicsounds in vanilla as a standard/example for everyone else to work from? The "officialness" might help keep everyone else compatible.

[mmmPI]

someone throws a full sound set modpack at the portal so I don't have to

So I have to agree with this position. If there was a de-facto standard instrument pack mod, I would definitely make miditorio support it. At that point implementing note length and sustained notes properly would have a huge effect on the end result. Until then, I don't think the "stop immediately" mode is worth the large rework that would be needed. Even if I define per-instument offsets for how long the note should continue after the midi note off event, I don't think it would sound good.

There is this mod (https://mods.factorio.com/mod/musical-speaker-fixed ), which wasn't ported for 2.0 or space age and doesn't contain the sound sample in the portal version, because it's around 1 GB, it's in archive there : https://github.com/IAmTomaton/musical-s ... tag/v1.0.3

A single portal mod cannot be that big it would take at least 4 of them although the quality of the sounds could be reduced.

The mod was trying to implement some of the things that were implemented in latest version, it contains 108-key set for " all standard MIDI instrument" from a quite permissive license from what i remember, originally made so that the "sustain" could be of arbitrary lengh, or "ready to be made cyclic" i think.

If you guys have some ideas on how to make a set of instruments, which one to select in priority and which would be bloat, i think that would be a starting point, adding instrument is quite easy, but to me hoping to get a standard is difficult, there are at least 5 sets of percussions in the mods, and like half a dozen different piano.

I have plans to add my own custom instruments, but those wouldn't be standard, i planned to add 1 instrument/mod for simplicity sake.

[mgabor]

I've been testing the new speaker behaviors and I found something that wasn't immediately obvious. The volume control is evaluated every tick, not just at the start of the sample playback. This has always been the case, it just didn't matter too much since you had no other option than manually moving the slider.

Now this is a double-edged sword. On the one hand, this now allows shaping the sound with a volume envelope, such as AD or ADSR. On the other hand, since the polyphonic notes of a single speaker share the same volume control, this makes the polyphony mode essentially unusable when paired with a volume control signal.

If I had to choose only one of these features and couldn't use the other, I'd have to pick polyphony. It is unfortunately not feasible to build any sort of generic music player without it. So with my current understanding it seems that the fixed speaker volumes and the note velocity bucketing logic will stay in miditorio for now.

I wonder if we could convince the devs to put a 108-key piano

Yeah, a good piano would definitely go a long way. There are so many (royalty) free sample packs that I'm sure could be used too.

[Illiander42]

If I had to choose only one of these features and couldn't use the other, I'd have to pick polyphony. It is unfortunately not feasible to build any sort of generic music player without it. So with my current understanding it seems that the fixed speaker volumes and the note velocity bucketing logic will stay in miditorio for now.

Doesn't it just increace the number of speakers needed a bit? Or am I thinking in sheet music too much?

[mgabor]

Doesn't it just increace the number of speakers needed a bit? Or am I thinking in sheet music too much?

You could account for each instrument’s note length and allocate notes across a number of speakers so that they never overlap on the same speaker. Assuming that most notes will have different velocities, you could need anywhere from 5-10 speakers per instrument track depending on the speed of the song. If there are chords being played, each note in the chord needs its own speaker as well.

What miditorio currently does is that it groups the notes in a given instrument track by their velocities and adds a speaker for each group with the approximate volume for the group. The number of groups can be set by the user.

So ultimately the first method could work, it just has the possibility of needing much more speakers in certain cases, but the note volumes would be very accurate.

Sidenote: it is unfortunate that the new volume signal takes values 0-100 when MIDI note velocities are 0-127. Not that it matters too much, but there is theoretically a slight loss of fidelity there.

[mgabor]

I just realized that you can technically make the existing sounds play cyclically! You just have to play multiple sounds of the same type started with an exact delay and manipulate their volumes so that they seamlessly transition into one another. This way you could make it so that you only hear the segment you want to repeat for as long as the sound needs to be sustained.

This would be very complicated though, and would need lots of speakers. Also if the volume change is not exact or if it is interpolated between ticks for example it might not be possible.

[Illiander42]

My brain is just getting stuck on "you can't possibly need more than 4 speakers for a violin, 1 for a trombone, 6 for a guitar, 4 for a bass" just from the physical properties of those instrements. I know that makes harps and pianos potentially need a lot though

I'm definitely stuck on thinking like a musician here.

[mgabor]

My brain is just getting stuck on "you can't possibly need more than 4 speakers for a violin, 1 for a trombone, 6 for a guitar, 4 for a bass"

That's a good point, that's the number of polyphony that those instruments are physically capable of. If you pluck a string that's already "playing" a note that will be replaced by the new note.

So what I said only applies to instruments that would actually need to be polyphonic. For the rest, the "Stop playing" toggle works pretty well, unless you want to play the same note that is already playing without leaving a one-tick silence between them.

[mgabor]

Implemented for the next release, 2.0.36.

Donion, can you please take another look at the "Stop playing" behavior? When a sample needs to be stopped, the audio engine must wait for the next zero crossing event of the sample in order to avoid an audible pop.

Here's a programmable speaker playing a square wave for one game tick. I zoomed in, so you can see the waveform. The top one is the original recording, as you can see the square wave is abruptly cut off. That vertical line in the waveform is the audible pop that you can hear.

I edited the bottom waveform to show what it's supposed to look like. The sample is only stopped at the next zero crossing, so the current period can finish playing. In this case there would be no audible pop. The timing difference is negligible.

Screenshot 2025-02-27 121846.png (58.18 KiB) Viewed 372 times

This is most apparent with the squarewave, but the other instruments can also have this "pop" happen depending on how the sample lines up with the game tick boundary. Also it doesn't matter if the sample is played for only one tick or longer, if the stop doesn't line up with the zero crossing there will be a pop artifact.

Most audio engine libraries should have options to respect the zero crossing when stopping sample playback.

You can find more information about zero crossing and how they are handled during audio editing in this article.

The other improvements we are discussing in this thread would be nice to have, but this one I would consider an actual bug.

Ideally the volume changes would also respect the zero crossing, as otherwise pops can occur there too. Granted, they would only audible with larger volume changes, for example when using the volume control to "start" playing a sample midway through.

[mgabor]

I just realized that you can technically make the existing sounds play cyclically! You just have to play multiple sounds of the same type started with an exact delay and manipulate their volumes so that they seamlessly transition into one another.

Here's a simple proof of concept that lengthens a saw wave note by 5 ticks. Original on the right for comparison.

It plays the second note 5 ticks after the first one, and it switches the volumes of the two speakers 5 ticks after that. I have to have the silent speaker on 1 volume instead of 0, because 0 volume samples don't start at all.

You can hear the popping artifacts here too, so the volume changes would definitely need to respect waveform zero crossings for this to be usable. But with those issues fixed, this would become an incredibly powerful sound design tool.

0eNrtWdtuozAQ/ZXKz04FAXJB2v2G9qV9qCrLATexAjZrm3RRxb/v2CTQbNILqJtmpbxEk5nxeObMYWySF7TISlYoLgyKXxBPpNAofnhBmi8FzaxO0JyhGKUs4SlTo0TmCy6okQrVGHGRst8o9utHjJgw3HDWrHdfKiLKfMEUOOB34mBUSA1LpbD7QbjAm2BUoXg8864j2CXliiWNPcQIcjRKZmTBVnTDYT0s2kYlYEtdJG21r79BTk9caUO6ykxV2Iw2XJkSNG2KjceI0WRla4RMC6pcpjH6iWypsjRFaf5G6lPx6scaYhYZrSDdlOlE8aIpDflXhifrq5SBETY+AHHcRiyUXCqa53SRsZEuGF2zozBOr6M9II9Bl3CVlNx00FllT6huGpyENtQSyTsEDbf7WHXODFOuRU0AsqFAQ8I1KbgBlGKjSoaRNrIgFioulkTLUqSw5olmmlnmaXDKASLCUxQDYYQ0zMmeQ3hvGxtkQZM12cistJn7uNPlMnW1lOqJJgyKo1kmn0khs6pYSVG1ezaLyRbFjKVkUbUgNSlvXbZl2Ww+BO8O2XxpxtQBOCtIwxnaFJxKCpLTYrdlszJnWtOlDWvDHXAn6M2d2Ym4c3vhznDu3J+CO2HP4T1vmePXpxjXK1qwUcrpUlrPL5zYd02wooJESyDLk5I54QICtZh2xPU9z2L36eD3HwW3J8WRbkT9uuF7Z9WNH8O7cf8vu3E3sBuTvnPVjy5n8snm6r7P2R3K056Pcni5FR+COBv4ahF8I4hJJpP1sdmIe4daKsbEkYbsFIxsA1CRDu/SzcDxOO/ZnNl5NWc3PKOvO8VuByLpewOHxYXnJ+C53/3AsWPNB0zvrgHRm9cA3bRM78tQz+5nFwwoZs3BtKf9FJYY/QIjVAEGIVXuHP9qlFWU7j4Fhb9R+njgAA7P6xmfeGfLylfZvsNM3D/iWy0NBr/xhfW3NtH3BoB85I2l/wAIew4AP/gfBwDc8TWxRTRYHAMiGnhOXcbBeYwDyOoZmmBzevCxj2c4eMROmm+lAI+xDxJ8gtxIY6sFydpCHIIUgjVyVivBSAmcGILSmiegnDrd1Jo9J85g9dzFsRKcLNZzbsUmOriB3ASAI9e5OP24cwdCOR+nj3b+UBU3LIeSuz884MUQnhzHtWgynofzeRTNwokfhnX9B8fTrws=

Copy blueprint

[mmmPI]

Most audio engine libraries should have options to respect the zero crossing when stopping sample playback.

You can find more information about zero crossing and how they are handled during audio editing in this article.

That's pretty similar to what i tried to explain on the other thread 112836 but i think the article you linked simplifies to get to core concept because there is only 1 wave in their example as a mono sound, whereas in game they are stereo sound, which unfortunately, makes me think it is not always possible to guarantee that the first part of your sentence is achievable. In the other thread it was possible to extend the lengh of a note because there was a moment in the sound when the zero crossing occurs at the same time for both left and right, similar to your picture of square wave. But there was this disclaimer :

( good thing your note is not made of frequency of opposite phase on stereo track , this is a nightmare to sample without clipping ).

This is how it look in audacity :

opposite phase.jpg (148.29 KiB) Viewed 291 times

This is quite tricky to sample,and although it is possible to have a seemless loop to play/extend indefinitely if you manage to cut at a place where it will match exactly in audacity, the game engine seem to add a little time offset and i had not been able to remove entirely the clipping when attempting to cut a note into 3 parts, "begin_sound" (1) ; "middle_sound" (2) ; and "end_sound" (3) and just play them as a single note in game.

This is how it look in Audacity, 2 notes, where it is possible to play 1 2 2 2 2 2 or 123 or 222223 without any clipping, but in the game, only 222222 is seemless, despite the cut at "non-zero-crossing". The transition from 1 to 2 or from 2 to 3 can have tiny variations in pops, i don't know if there is something that can be done for this due to physics, if the frequency of the note is lower than 60 Hz and the player turn things on/off every tick, i feel like it's bound to create sound artifacts, or even "it should" if the game tries to stop the sound as soon as the signal is removed, and it would take "too long" to wait for the next zero crossing, (if there is even one ! ).

02-28-2025, 02-06-26.png (75.26 KiB) Viewed 291 times

I have put the soundfiles into a mod (not in the portal) to test the new feature quickly, it only adds 3 notes of a "Handpan" instrument where the soundwave is of different phase when it crosses the zero line, it is possible to hear the same note extended forever without clipping using this setting :

tinnitus simulator.jpg (42.33 KiB) Viewed 291 times

PS-Handpan_0.0.1.zip

(371.11 KiB) Downloaded 27 times

[Donion]

Thanks for the feedback, here are some responses:

Cyclic notes in vanilla
There are currently no plans to change or add any notes.

Volume controlled by signal from 0 to 100
I was wondering what should the resolution be. SDL audio backend is using 0 to 128 internally, I considered using that for the volume by signal but I though it might be unexpected/unclear. There is also a good chance that with a future SDL update its volume resolution will be much higher. I'm open to suggestions, but 0 to 100 seems the least wrong at the moment.

Stop immediately and zero-crossing
There is no waiting for zero-crossing in the audio backend we're using or in Factorio engine on top of it. We go only as far as fading in/out over some number of ticks (from 4 to 20 ticks usually).
Adding such functionality would be a major addition which I don't want to do.
A tiny fade-out (5-10ms) would probably be doable.

General note on timing and precision
Even with cyclic notes there are no guarantees on timing, they still need to work with the ~60 UPS constraints (or whatever the game speed is). This means that when going from begin_sound to middle_sound or from middle_sound to end_sound, the transition is not sample perfect (looping middle_sound is seamless, this is supported by the backend). The currently playing sound is stopped wherever it is currently if it would finish playing within the next tick, so there is up to ~16.67ms of uncertainty.

Let's not forget this is still a game and not an audio workstation

[Illiander42]

Let's not forget this is still a game and not an audio workstation

Madness! Next you'll be saying this isn't a fully-featured computer!

[mmmPI]

Thanks for the feedback, here are some responses:

Thanks for the improvements already !

Stop immediately and zero-crossing

I have been reading the linked article by mgabor again, and documented a bit more, i think the addition of a tiny (5-10ms) fade-out has no downside.
I think the "waiting for zero crossing" imply the game engine would be allowed to somehow modify the sound to create those if they don't exist "synced in both channel" or stop the sound at a slightly different time for both left and right channel wich i find fascinating. I take your words for it that it's a big functionnality and you don't want to add, but i would be curious to read about what it represent in a game like factorio, what kind of software handles this, if you have some links

General note on timing and precision

That may explain why my other attempts at adding instrument that i thought would be easier didn't work out so far x).

I think i can make sure the transition from begin_sound to middle_sound works in the 60 ups constraint by adding some variable amount of silence at the beginning of the begin_sound ( max 16 ms ).

But the duration of the middle_sound i don't think it can be made to work with the 60 UPS constraint, as the duration needs to include an integer number of period of the soundwave and their duration depends on the note frequency, which is not always going to be perfect. Which i think might cause 1 single pop instead of 2 currently when i try. ( transition between middle_sound and end_sound)

At this point I wonder if the tiny fade-out could be enough to mask the imperfection of this transition between middle and end sound.

Let's not forget this is still a game and not an audio workstation

When i look at the previous blueprint it makes wonder, is the goal of the game to make an audio workstation ? x).

Thanks again for the additions , that's plenty to test