Summary
I suppose, but your problem is probably due to the fact that you have an inherent transposition. 2D numpy arrays are indexed as a row, column. The index "x, y" is a column, a row. In this context, numpy.gradient will mainly return dy, dx, not dx, dy.
Try changing the line:
dx, dy = np.gradient(zi)
in
dy, dx = np.gradient(zi)
Also, if your depths are defined as positive, this should be:
dy, dx = np.gradient(-zi)
However, I assume that you have agreements on positive depth, so I will leave this part of the examples below. (Thus, higher values are expected to be lower / lower in the examples below, and water will flow to higher values.)
Reproducing the problem
For example, if we modify the code that you used to use random data, and fill in a few variables that go beyond the scope of your code sample (so this is a separate example):
import numpy as np import matplotlib.pyplot as plt from matplotlib.mlab import griddata
The result will look like this: 
Note that there are many places where the flow lines are not perpendicular to the contours. This is even easier than the wrong direction of the arrows, that something is going wrong. (Although the perpendicular implies an aspect ratio of 1 for the graph, which is not entirely true for these graphs unless you set it.)
Problem fix
If we just change the line
dx, dy = np.gradient(zi)
in
dy, dx = np.gradient(zi)
We get the correct result:

Interpolation Suggestions
On a side note, griddata is a poor choice in this case.
Firstly, this is not a “smooth” interpolation method. It uses delaunay triangulation, which makes “sharp” ridges at the borders of a triangle. This leads to abnormal gradients in these places.
Secondly, it restricts the interpolation to the convex hull of your data points, which may or may not be a good choice.
A radial basis function (or any other smooth interpolation) is a much better choice for interpolation.
As an example, if we modify a piece of code to use RBF:
import numpy as np import matplotlib.pyplot as plt from scipy.interpolate import Rbf

(You will notice that the intersections are not completely perpendicular due to the uneven aspect ratio of the graph. All of them are 90 degrees if we set the aspect ratio of the graph to 1.)
As a parallel comparison of two methods:
