I have a problem while converting a float number to ASCII. I get sometimes a small "jitter" with some special values, like this:

float x
x = 3.1415
DisplayValue x

results in:
3.1414999999999

I think, the reason is the binary storing of the decimal value inside the FPU, so I get some inaccurateness while converting back to dec / ascii. Now I have no idea how to solve this - how to round this "very nearly x" values to "x"? Probably I should round all output to a number of digits?
Posted on 2005-11-22 05:01:08 by beaster
Hello,

What abt working on the resultant string, you first search for the decimal place. Once that is found you count till the number of dp you are seeking for and look at the next character. If it is bigger than or equal to 5, round up. Do be careful with the carry. For example

0.999999999999999999999

would need to be rounded up to

1.0000000000
Posted on 2005-11-22 06:20:33 by roticv
float to ascii algo:
1. tun the FPU to truncation mode (round towards 0)
2. FISTP the value
3. FSUB the FISTPed value
3. FMUL the value by 10000 or something larger if you need (but no more than 1'000'000'000)
4. FISTP the value and represent it after the decimal point
5. repeat 3-4 if you need better precision (but I don't see the point in anything better)

example:

float = 424.3487429

1. turn on the truncation
2. FIST "424"
3. FSUB 424 from the value in st(?).  so we have  0.3487429 on the FPU now
4. a) FMUL by 1'000'000 (we don't need anything better). now we have 3487429.XXXX
    b) FISTP "3487429"
    c) represent it as  "424"  "."  "3487429"

Truncation is needed for FIST'ing. If you try to FIST 3.999 with rounding to nearest integer you'll get "4" (and then add those .999 to this, so you'll see 4.999 which is totally wrong).
Posted on 2005-11-22 13:55:03 by ti_mo_n
4. a) FMUL by 1'000'000 (we don't need anything better). now we have 3487429.XXXX


As beaster found out, that could also be 3487428.9XXX. If you don't turn off truncation and turn on rounding, you would not get the expected result.

beaster:

As explained in Simply FPU, decimal fractions can rarely be represented exactly in binary and the least significant 64th bit gets rounded in a similar manner that you would round some fractions in the decimal system, such as:

1/11 = 0.09090909.....
which would get rounded to 0.090909 with 6 decimal places and be slightly smaller than the exact value. However,
10/11 = 0.909090909...
but would get rounded to 0.909091 with 6 decimal places and be slightly larger than the exact value.

The easiest way for your conversion would be to multiply your float by a power of 10 to give you the number of decimal places you want in the answer and store the result as a rounded integer. Then convert that integer to ASCII and insert the decimal delimiter where it should belong.

Raymond
Posted on 2005-11-22 22:06:13 by Raymond
But this way you can only get a 9-digit number.

corrected algo:

1. turn the FPU to truncation mode (round towards 0)
2. FISTP the value
3. FSUB the FISTPed value
4. turn the FPU to round-to-nearest-integer mode
5. FMUL the value by 10000 or something larger if you need (but no more than 1'000'000'000)
6. FISTP the value and represent it after the decimal point
7. repeat 3-4 if you need better precision (but I don't see the point in anything better)

Even without point 4 you can get only small jitter.

Well both ways are good. (as always) depends on what you want to do ;)
Posted on 2005-12-04 14:19:16 by ti_mo_n