Floating point arithmetic error

Question

I use the following function to approximate the derivative of a function at a point:

def prime_x(f, x, h): if not f(x+h) == f(x) and not h == 0.0: return (f(x+h) - f(x)) / h else: raise PrecisionError 

As a test, I pass f as fx and x as 3.0. Where fx :

 def fx(x): import math return math.exp(x)*math.sin(x) 

which has exp(x)*(sin(x)+cos(x)) as a derivative. Now, according to Google and my calculator

exp(3)*(sin(3)+cos(3)) = -17.050059 .

So far so good. But when I decided to test a function with small values ​​for h , I got the following:

 print prime_x(fx, 3.0, 10**-5) -17.0502585578 print prime_x(fx, 3.0, 10**-10) -17.0500591423 print prime_x(fx, 3.0, 10**-12) -17.0512493014 print prime_x(fx, 3.0, 10**-13) -17.0352620898 print prime_x(fx, 3.0, 10**-16) __main__.PrecisionError: Mantissa is 16 digits 

Why does the error increase when h decreases (after a certain point)? I expected the opposite until f(x+h) was equal to f(x) .

Answer 1

Floating-point arithmetic (both integer arithmetic and fixed-point arithmetic) has a certain degree of detail: values ​​can only be changed by a certain step size. For the IEEE-754 64-bit binary format, this step size is about 2 ^-52 times the value (about 2.22 • 10 ^-16 ). This is very small for physical measurements.

However, when you make h very small, the difference between f (x) and f (x + h ) is not very large compared to the step size. The difference can only be an integer multiple of the step size.

When the derivative is d, the change in f (x) is about h • d . Even if you calculate f (x) and f (x + h ), and possibly also in a floating point format, the measured value of their difference should be a multiple of the size of step s, therefore it should be round (h • d / s) • s, where round (y) is rounded to the nearest integer. It is clear that since you make h less, h • d / s less, so the effect of rounding it to an integer is relatively large.

Another way to look at this is that for a given f (x), there is some error in calculating the values ​​around f (x). The smaller the value of h, the less f (x + h ) - f (x), but the error remains the same. Therefore, the error increases with respect to h.

Answer 2

When you subtract two numbers that are almost the same, the result has much less accuracy than any of the inputs. This reduces the accuracy of the overall result.

Suppose you have the following two numbers: up to 15 decimal places:

  1.000000000000001 - 1.000000000000000 = 0.000000000000001 

See what happened? The result has only one good figure.