F # +. Net, calculation error using the System.Math.Floor function

Question

here is the function I wrote to print each digit of a floating-point number in F #:

let rec TestFloor (fnum:float) = let floor = System.Math.Floor(fnum) printfn "fnum:%f floor:%f" fnum floor if floor > 0.0 then TestFloor((fnum - floor) * 10.0) 

In any case, the result looks strange, for example:

 > TestFloor 1.23;; fnum:1.230000 floor:1.000000 fnum:2.300000 floor:2.000000 **fnum:3.000000 floor:2.000000** fnum:10.000000 floor:9.000000 fnum:10.000000 floor:9.000000 fnum:10.000000 floor:9.000000 fnum:10.000000 floor:9.000000 fnum:10.000000 floor:9.000000 fnum:10.000000 floor:9.000000 fnum:10.000000 floor:9.000000 fnum:10.000000 floor:9.000000 fnum:9.999998 floor:9.000000 fnum:9.999982 floor:9.000000 fnum:9.999822 floor:9.000000 ... 

In the fourth line, level 3.0 turned out to be 2.0, which is strange. The following calculations do not work correctly. I wonder what is going on there?

Thanks!

---

Edit 2

@ sepp2k

This is the result of using% .30f:

 > TestFloor 1.23;; fnum:1.230000000000000000000000000000 floor:1.000000000000000000000000000000 fnum:2.300000000000000000000000000000 floor:2.000000000000000000000000000000 **fnum:3.000000000000000000000000000000 floor:2.000000000000000000000000000000** fnum:9.999999999999980000000000000000 floor:9.000000000000000000000000000000 fnum:9.999999999999820000000000000000 floor:9.000000000000000000000000000000 fnum:9.999999999998220000000000000000 floor:9.000000000000000000000000000000 fnum:9.999999999982240000000000000000 floor:9.000000000000000000000000000000 fnum:9.999999999822360000000000000000 floor:9.000000000000000000000000000000 

As you can see on the fourth line, fnum is 3.00... , and the value of floor 2.00..

---

Change 3 - Baxin

Thanks to everyone, now I understand what the problem is.

Answer 1

This is just one of those rounding issues that you get with floating point arithmetic. You will see that his view is 2.99999 ... perhaps endlessly repeating. A call to printfn clearly rounds this to the expected view, but gender still sees it as 2.99999 ...

This is exactly the problem that exists to solve the decimal type, so if we rewrite the use of the decimal number, we get the correct result:

 let rec TestFloor dnum = let fl = floor dnum printfn "fnum:%f floor:%f" dnum fl if fl > 0.0M then TestFloor((dnum - fl) * 10.0M) 

This gives:

 > TestFloor 1.23M;; fnum:1.230000 floor:1.000000 fnum:2.300000 floor:2.000000 fnum:3.000000 floor:3.000000 fnum:0.000000 floor:0.000000 val it : unit = () 

Of course, you can stick with a float, but add a very small tolerance value to make sure that these corner cases are always slightly higher than expected, and not slightly lower, for example:

 let rec TestFloor fnum = let fl = floor (fnum + 0.00000000001) printfn "fnum:%f floor:%f" fnum fl if fl > 0.0 then TestFloor((fnum - fl) * 10.0) 

Which gives the same result as above.

Answer 2

Presumably the fnum value fnum similar to 2.9999999999999999999 and printfn just rounds to 3.0 when displaying it. Try increasing the number of digits displayed with %.30f instead of %f .

Answer 3

Even if you get 3.00000 ... with maximum decimal precision, there may still be rounding. Let them not forget that the value is stored in binary format, and rounding is always necessary for printing purposes (binary and decimal). The answer is that although your value is printed as 3.0000 ..., it is actually less than 3. Adding decimal digits will not reduce it.

Consider this code:

 > let num = 3.0 - 2.**(-51.);; > printfn "%.30f" num;; 3.000000000000000000000000000000 > floor num;; val it : float = 2.0