[0.17.79] game.table_to_json uses excessively many digits for numbers

We are aware of them, but they have low priority. We have more important things to do. They go here in order not to take space in the main bug thread list.
Post Reply
Hornwitser
Fast Inserter
Fast Inserter
Posts: 204
Joined: Fri Oct 05, 2018 4:34 pm
Contact:

[0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Hornwitser »

The command

Code: Select all

/c game.print(game.table_to_json{v=1/3})
Prints out {"v":0.333333333333333303727386009995825588703155517578125}, which is excessively many digits for a number with no more than 16 significant digits. The output looks like the exact form of a double precision number, but it's not. It's a rational with 5 as one of the factors of the denominator and is lower than the closest numbers to 1/3 for IEEE 754 double precision numbers by about half a unit of least precision (that being 0.333333333333333314829616256247390992939472198486328125). Whatever number converter you used to convert the number to decimal either wasn't designed to give the exact form, or is broken.

User avatar
Klonan
Factorio Staff
Factorio Staff
Posts: 5148
Joined: Sun Jan 11, 2015 2:09 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Klonan »

Does this cause a problem?

Hornwitser
Fast Inserter
Fast Inserter
Posts: 204
Joined: Fri Oct 05, 2018 4:34 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Hornwitser »

There are two issues caused by this. The first is that the JSON is longer, in Clusterio for example the JSON is passed on to other instances and this being sent over commands causes the operation to take longer and more bandwidth. The second is that there's floating point values for which the encoding to JSON and decoding back to Lua doesn't produce the same value, for example the following prints false:

Code: Select all

/c local v = 0.1 + 2^- 55
game.print(serpent.line(
    game.json_to_table(game.table_to_json{v = v}).v == v
))


posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by posila »

Since 0.13-ish we use (little bit modified) trio for formatting strings:

When saving out json, numbers are converted using

Code: Select all

trio_snprintf(buffer, sizeof(buffer), "%.*g", DBL_MANT_DIG, value);
... in case anyone wants to investigate this before we get to it.
(we compile with strict floating point math - /fp:strict in MSVC)
Attachments
trio.zip
(68.11 KiB) Downloaded 92 times

User avatar
boskid
Factorio Staff
Factorio Staff
Posts: 2227
Joined: Thu Dec 14, 2017 6:56 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by boskid »

Code: Select all

#define DBL_MANT_DIG     53                      // # of bits in mantissa
It is obvious that for each bit of mantissa, we need at least one character on output :)

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by posila »

boskid wrote:
Sat Jan 04, 2020 8:33 pm

Code: Select all

#define DBL_MANT_DIG     53                      // # of bits in mantissa
It is obvious that for each bit of mantissa, we need at least one character on output :)
Well, I think, despite what the thread title says, the main problem here is that double -> str -> double didn't end up in the exactly same number (when it had enough digits available to exactly represent the original number). If we can work that out, we could turn mod settings back to json format. The fact that it uses more digits than it needs is secondary imho.

EDIT: Chaning it to DBL_DECIMAL_DIG results in: {"v":0.3333333333333333}
... hmm

Hornwitser
Fast Inserter
Fast Inserter
Posts: 204
Joined: Fri Oct 05, 2018 4:34 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Hornwitser »

Code: Select all

typedef double trio_long_double_t;
That ain't no long double. This change breaks the formatting of doubles which rely on having more precision than a double to work.

User avatar
TruePikachu
Filter Inserter
Filter Inserter
Posts: 978
Joined: Sat Apr 09, 2016 8:39 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by TruePikachu »

Conversion instability, if it's limited to ~1ULP, can frequently be attributed to `scanf` etc. internally truncating the unrounded output.

Hornwitser
Fast Inserter
Fast Inserter
Posts: 204
Joined: Fri Oct 05, 2018 4:34 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Hornwitser »

On an unrelated note the trio regression suite shows that the modifications you've made causes the following test case to fail

Code: Select all

string.format("%#11.5g", 10e5)
It produces the output 1.00000e+05 (which has 6 significant digits) instead of the expected output of (space)1.0000e+05 (with has 5 significant digits.)

Edit: another interesting note is that in Microsoft Visual C++ the long double data type correspond to double instead of the 80-bit extended precision data type that other compilers use, maybe that's the reason for the long double to double change.

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by posila »

Hornwitser wrote:
Sun Jan 05, 2020 5:58 am
Edit: another interesting note is that in Microsoft Visual C++ the long double data type correspond to double instead of the 80-bit extended precision data type that other compilers use, maybe that's the reason for the long double to double change.
Indeed. I guess it's time to look into using stb_sprintf for float to str formatting or maybe even all string formatting.

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Rseding91 »

Hornwitser wrote:
Sat Jan 04, 2020 5:40 pm
The second is that there's floating point values for which the encoding to JSON and decoding back to Lua doesn't produce the same value
That's the root issue of why it's currently using so many digits: because the conversion just doesn't work correctly when going binary -> string -> binary. I kept upping the number of digits until I gave up on it and just changed the mod-settings.json file to mod-settings.dat at which point the issue(s) stopped happening because we have a competent serialization system for saving data.

JSON is not a competent system for saving data :P

Unless someone else wants to look into this I'm moving this to minor issues and I don't consider it worth the time to look into. If anything I suggest mods encode their own data in binary format, convert to base 64, and send that. You won't ever lose any data that way.
If you want to get ahold of me I'm almost always on Discord.

Hornwitser
Fast Inserter
Fast Inserter
Posts: 204
Joined: Fri Oct 05, 2018 4:34 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Hornwitser »

Rseding91 wrote:
Mon Jan 13, 2020 9:53 pm
That's the root issue of why it's currently using so many digits: because the conversion just doesn't work correctly when going binary -> string -> binary. I kept upping the number of digits until I gave up on it and just changed the mod-settings.json file to mod-settings.dat at which point the issue(s) stopped happening because we have a competent serialization system for saving data.
IEEE 754 guarantees that converting a double precision float into decimal with 17 digits of precision and back into a double precision float will give you the exact same bit representation (with the exception of nan encodings) provided the conversion routine has the necessary precision. The reason you don't see this happen on your system is because you use a library written for Linux like systems that depend on a high precision data type that doesn't exist on MSVC. Adding more digits to the conversion doesn't fix the conversion being broken from lacking the necessary precision to do it correctly.
Rseding91 wrote:
Mon Jan 13, 2020 9:53 pm
Unless someone else wants to look into this I'm moving this to minor issues and I don't consider it worth the time to look into. If anything I suggest mods encode their own data in binary format, convert to base 64, and send that. You won't ever lose any data that way.
Base64 is not a viable option: the encoder in util uses 10ms to encode 1kb of data and from what I've heard of people trying to improve it there isn't much to be done about it, converting data to base64 is unusably slow in Lua.

User avatar
TruePikachu
Filter Inserter
Filter Inserter
Posts: 978
Joined: Sat Apr 09, 2016 8:39 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by TruePikachu »

So, I just went through how Trio does its formatting, and I believe a big part of the issue is that the requested precision is so large compared to the intermediate storage. While the raw math is analytically correct, there's shortcomings in how it is compiled.

Formatting is done by `TrioWriteDouble`, and all intermediate values are `trio_long_double_t` (as others have said: this should ideally be wider than `double`, but it is currently the same). If we let the number be 1/3 and the precision 53:
  1. L3183 sets `fractionNumber` to `number` (exactly 6004799503160661/18014398509481984)
  2. L3210 sets `dblFractionBase` to effectively `pow(10.0,53.0)` (exactly 99999999999999999322094867436162797646170844194406400)
  3. L3319 sets `fractionAdjust` to 0.5 (exactly 1/2)
  4. L3352 divides `fractionAdjust` by `dblFractionBase` (setting it to exactly 8627182933488205/1725436586697640946858688965569256363112777243042596638790631055949824)
  5. L3529 is the print loop for `fractionNumber`. Replication of it in Common Lisp with the exact numbers listed above yield the same output as reported in OP, but with two additional trailing zeroes (automatically omitted from code output).
The loop has massively compounding rounding errors caused by the excessive precision requested; it does multiplication by 10.0 each iteration on `fractionAdjust` (making it more inaccurate than it already was).

If `precision=17`, we have:
  1. L3183 does the same as before
  2. L3210 sets `dblFractionBase` to `pow(10.0,17.0)` (exactly 99999998430674944)
  3. L3319 does the same as before
  4. L3352 sets `fractionAdjust` to exactly 6044629/1208925819614629174706176
  5. L3529 print loop emits 0.33333333333333330
There are still some inaccuracies involved, due to the relatively-narrow working form.

If I test the precision-17 code under GNU CLISP, where I actually have `long-float` support (being a common 80-bit float):
  1. L3183 sets `fractionNumber` to exactly 6004799503160661/18014398509481984
  2. L3210 sets `dblFractionBase` to exactly 100000000000000000
  3. L3319 sets `fractionAdjust` to exactly 1/2
  4. L3352 sets `fractionAdjust` to exactly 13292279957849158729/2658455991569831745807614120560689152
  5. L3529 print loop emits 0.33333333333333331

When comparing all numbers side-by-side, along with the remainder of the digits from `fractionNumber` in the 17-precision cases:

Code: Select all

dbl p53 print: 0.33333333333333330372738600999582558870315551757812500
dbl p17 print: 0.33333333333333330
                (spare digits): 0.37273860099958256
lft p17 print: 0.33333333333333331
                (spare digits): 0.4829616256247390993
  exact input: 0.333333333333333314829616256247390992939472198486328125
I can almost entirely confirm that changing the printed precision will fix the issue of using "excessively many digits", while correcting the intermediate encoding will fix the issue of numbers not being print/read stable.

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Rseding91 »

I tested with the 17 precision digits between the 15 and current number. None of them worked.
If you want to get ahold of me I'm almost always on Discord.

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Rseding91 »

Specifically, it was 15, I changed it to 17, that still failed on different numbers so I changed it to 54 and that still failed so I said fuck it and switched to binary format for mod settings.

See: 58859 where I finally said screw JSON and or string format for numbers. If JSON supported hex floats I would use those since I can store properly a floating point value in 100% precision but nope... spec doesn't support them.
If you want to get ahold of me I'm almost always on Discord.

Hornwitser
Fast Inserter
Fast Inserter
Posts: 204
Joined: Fri Oct 05, 2018 4:34 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Hornwitser »

It doesn't matter how many digits you use when the float to decimal conversion function you use is broken. How many times does this need to be said? The float to decimal conversion function you use does not round trip because it's missing the 80-bit extended precision floats it was designed to work with.

Any double precision float to decimal implementation worth it's salt will round trip, but the one you use doesn't, stop blaming JSON for your bad float to decimal converter.

User avatar
TruePikachu
Filter Inserter
Filter Inserter
Posts: 978
Joined: Sat Apr 09, 2016 8:39 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by TruePikachu »

Okay, I just did some research, and Microsoft might actually be to blame for this one. Visual Studio no longer supports the extended-precision 80-bit floating point number format which is required for Trio to actually give correct answers. Either Trio will need to be compiled under another compiler (and `static_assert(sizeof(long double)>8)`) and linked in, or another library will need to be used.

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by Rseding91 »

I want to use <charconv> which is part of C++17 which can do round-trip-stable float conversion and does it *way* faster than any of the current implementations. https://cppcon2019.sched.com/event/Sft8 ... final-boss
If you want to get ahold of me I'm almost always on Discord.

User avatar
TruePikachu
Filter Inserter
Filter Inserter
Posts: 978
Joined: Sat Apr 09, 2016 8:39 pm
Contact:

Re: [0.17.79] game.table_to_json uses excessively many digits for numbers

Post by TruePikachu »

Just read through my copy of docs, looks good assuming it encodes to the standard format JSON uses (I'd suspect it does, but even if it doesn't there should only need to be minimal string manipulation) and that the implementations used by the various compilers are compatible with each other (the round-trip-stable property is only guaranteed if the implementations are identical).

Post Reply

Return to “Minor issues”