NumPy Accuracy Using a Spot Product

Question

I use Theano / NumPy, using some in-depth knowledge. I found a very nasty problem. I got the weight matrix A (suppose it is 50 * 2048) and the vector function b (2048 dim).

A initialized using

self.alpha = np.random.random((50, 2048)).astype(np.float32) * 2 - 1.0 

b is 2048 dim numpy.ndarray from theano.

Problem

 X = numpy.dot(A, b) Y = [numpy.dot(A[i], b) for i in xrange(50)] 

Some lines X and Y are not strictly equal. I compared them and found that the difference is in 1e-6 to 1e-7.

Currently, I prefer to use the second one to calculate the dot product, as it seems that it can learn more weight. But the first is much faster. So I wonder why there is such a big difference. Is this the reason for the various realizations of the point (matrix, vector) and point (vector, vector)? Thank you very much!

- edit As mentioned above, this is code that you can reproduce.

 import numpy as np test_time = 1000 vector_size = 100 matrix_size = (100, 100) for i in xrange(test_time): a = np.random.random(matrix_size).astype(np.float32) * 2 - 1.0 b = np.random.random(vector_size).astype(np.float32) x = np.dot(a, b) y = [np.dot(a[i], b) for i in xrange(a.shape[0])] for k in xrange(len(y)): epsilon = x[k] - y[k] if abs(epsilon) > 1e-7: print('Diff: {0}\t{1}\t{2}'.format(x[k], y[k], epsilon)) 

Answer 1

Well, as a rule, there is a trade-off between performance and accuracy. You may need to compensate one or the other. Although I personally do not think that the difference of 0.0000001 is a big problem in most applications. If you are looking for better accuracy, you'd better go with float64 , but note that the float64 operations on GPUs are extremely large, especially the NVIDIA 9xx GPUs.

I can notice that the problem mentioned seems to depend on your hardware settings, because I do not encounter such a problem on my machine.

You can also use np.allclose(x, y) to see if the difference is palpable.