Can anybody help me in this little problem.

I have try to calculate at 2.1 minus 2
as you can see in my attachfile.
Everybody knows at the answer is 0.1,
but FPU says it's 0.09999999999...

So what's the problem !
Posted on 2002-06-04 07:35:49 by japalo
This is interesting, when I tried the following code I got 0.1

TwoPt1 REAL10 2.1
Two dd 2

Tmp dq ?
Buf db 128 dup (?)

fld TwoPt1
fisub Two
fstp Tmp

invoke FloatToStr,Tmp,add Buf
invoke MessageBox,0,addr Buf,0,0

So I did notice the using REAL4 for TwoPt1 resulted in some inprecision so the problem probably lies with whatever method you're using to read the result from the FPU.
Posted on 2002-06-04 11:13:36 by Eóin
Another possiblity is maybe the assembler is generating off values for the REAL10, but we have to see the disassembly to confirm.
Posted on 2002-06-04 11:34:21 by bitRAKE
maby its your controll word.
Posted on 2002-06-04 12:52:30 by Qages
Your RC is set to 11b which is the cause of the problem.
You know, neither 2.1 nor .1 is exactly representable with finite number of binary digits. (If you did not know, try it for fun.) Fall back to 0 (RC part) which is Intel's default value, and you should not see any more like this.

Well, other inexact results are possible under a different situation with different numbers, but the machine is doing its best. What can I say? :)

If you still see the same problem with the same code, then the problem is your printing routine. At least, I don't have the same problem you described even when PC is set to DP. If you created the printing routine, check your code. If you are using somebody else's, tell the coder about the problem and ask to fix it.
Posted on 2002-06-04 14:07:24 by Starless