3G coverage map - visualize lat, long, ping data

Question

Suppose I drove a given route with a 3g modem and GPS on my laptop, and my computer records ping delay at home. I have associated ping with GPS lat / long, and now I would like to visualize this data.

I have about 80,000 points per day, and I would like to show a few months. I am particularly interested in displaying the areas in which ping expires sequentially (i.e. Ping = 1000).

Scatter plot

My first attempt was to scatter a chart with a single point for data entry. I made the size 5x larger if it was a timeout, so it was obvious where these areas were. I also dropped the alpha to 0.1, for the rude way of seeing points overlaid.

# Colour c = pings # Size s = [2 if ping < 1000 else 10 for ping in pings] # Scatter plot plt.scatter(longs, lats, s=s, marker='o', c=c, cmap=cm.jet, edgecolors='none', alpha=0.1) 

The obvious problem is that it displays one marker per data point, which is a very poor way to display large amounts of data. If I drive past the same area twice, then the first pass data is displayed only on top of the second pass.

Interpolate on an even grid

Then I tried using numpy and scipy to interpolate on an even grid.

 # Convert python list to np arrays x = np.array(longs, dtype=float) y = np.array(lats, dtype=float) z = np.array(pings, dtype=float) # Make even grid (200 rows/cols) xi = np.linspace(min(longs), max(longs), 200) yi = np.linspace(min(lats), max(lats), 200) # Interpolate data points to grid zi = griddata((x, y), z, (xi[None,:], yi[:,None]), method='linear', fill_value=0) # Plot contour map plt.contour(xi,yi,zi,15,linewidths=0.5,colors='k') plt.contourf(xi,yi,zi,15,cmap=plt.cm.jet) 

From this example

It looks interesting (many colors and shapes), but it extrapolates too far around areas that I have not studied. You do not see the routes that I traveled, only red / blue spots.

If I drove on a large curve, it will interpolate for the area between (see below):

Interpolate over uneven mesh

Then I tried using meshgrid ( xi, yi = np.meshgrid(lats, longs) ) instead of a fixed grid, but they told me that my array is too large.

Is there an easy way to create a grid from my points?

---

My requirements:

• Processing large data sets (80,000 x 60 = ~ 5 m points)
• Display duplicate data for each point either by averaging (I assume that interpolation will do this) or by accepting a minimum value for each point.
• Do not extrapolate too far from data points.

I am pleased with the scatter plot (above), but I need to somehow average the data before I show it.

(Apologies for the dodgy blueprints of mspaint, I cannot download the actual data)

---

Decision:

 # Get sum hsum, long_range, lat_range = np.histogram2d(longs, lats, bins=(res_long,res_lat), range=((a,b),(c,d)), weights=pings) # Get count hcount, ignore1, ignore2 = np.histogram2d(longs, lats, bins=(res_long,res_lat), range=((a,b),(c,d))) # Get average h = hsum/hcount x, y = np.where(h) average = h[x, y] # Make scatter plot scatterplot = ax.scatter(long_range[x], lat_range[y], s=3, c=average, linewidths=0, cmap="jet", vmin=0, vmax=1000) 

Answer 1

To simplify your question, you have two sets of points: one for ping <1000, one for ping> = 1000. Since the number of points is very large, you cannot build them directly using the scatter () method. I created some sample data:

 longs = (np.random.rand(60, 1) + np.linspace(-np.pi, np.pi, 80000)).reshape(-1) lats = np.sin(longs) + np.random.rand(len(longs)) * 0.1 bad_index = (longs>0) & (longs<1) bad_longs = longs[bad_index] bad_lats = lats[bad_index] 

(longs, lats) are points for ping <1000, (bad_longs, bad_lats) are points for ping> 1000

You can use numpy.histogram2d () for scoring:

 ranges = [[np.min(lats), np.max(lats)], [np.min(longs), np.max(longs)]] h, lat_range, long_range = np.histogram2d(lats, longs, bins=(400,400), range=ranges) bad_h, lat_range2, long_range2 = np.histogram2d(bad_lats, bad_longs, bins=(400,400), range=ranges) 

h and bad_h - scoring in each small area of ​​the scraper.

Then you can select many methods for visualization. For example, you can build it using the scatter () method:

 y, x = np.where(h) count = h[y, x] pl.scatter(long_range[x], lat_range[y], s=count/20, c=count, linewidths=0, cmap="Blues") count = bad_h[y, x] pl.scatter(long_range2[x], lat_range2[y], s=count/20, c=count, linewidths=0, cmap="Reds") pl.show() 

Here is the complete code:

 import numpy as np import pylab as pl longs = (np.random.rand(60, 1) + np.linspace(-np.pi, np.pi, 80000)).reshape(-1) lats = np.sin(longs) + np.random.rand(len(longs)) * 0.1 bad_index = (longs>0) & (longs<1) bad_longs = longs[bad_index] bad_lats = lats[bad_index] ranges = [[np.min(lats), np.max(lats)], [np.min(longs), np.max(longs)]] h, lat_range, long_range = np.histogram2d(lats, longs, bins=(300,300), range=ranges) bad_h, lat_range2, long_range2 = np.histogram2d(bad_lats, bad_longs, bins=(300,300), range=ranges) y, x = np.where(h) count = h[y, x] pl.scatter(long_range[x], lat_range[y], s=count/20, c=count, linewidths=0, cmap="Blues") count = bad_h[y, x] pl.scatter(long_range2[x], lat_range2[y], s=count/20, c=count, linewidths=0, cmap="Reds") pl.show() 

Output indicator:

Answer 2

GDAL libraries, including the Python API and related utilities, especially gdal_grid , should work for you. It includes a number of interpolation and averaging methods and parameters for generating data with a grid of scattered points. You should be able to manipulate the mesh cell size to get a nice resolution.

GDAL handles several data formats, but you should be able to pass your coordinates and ping values ​​to CSV and return PNG or JPEG without any problems.

Remember that lat / lon data is not a flat coordinate system. If you intend to include the results with other map data, you will need to figure out which projection map, units, etc. Use.