Matplotlib MemoryError on pcolorfast

Question

Problem:

I am currently loading column data from text files into numpy arrays and then draw them and save the resulting image. Since the values ​​will always be on a uniformly distributed grid, it seemed like the right time to use pcolorfast. Each array is necessarily square, usually between 1024x1024 and 8192x8192. Currently I deal with this only up to 4096x4096 in size. This needs to be done for hundreds of files, and while it successfully completes the fist image, subsequent images are reset to a MemoryError.

Bad decisions:

I guaranteed that here that I have = False in rc.

Limitations:

Images must be saved using all 4096x4096 values ​​and cannot be reduced to 1024x1024 (as suggested here ).

Notes:

After looking at the memory usage during each phase (creating an empty array, loading values, graphing, saving), array A is still in memory after the makeFrame completes. Is there an explicit call to remove it? Should the poison be removed explicitly, or should pylab take care of this? The ideal situation (probably obvious) is to return the memory to ~ the same level as before the makeFrame () call.

All tips are welcome. I tried to solve this for several days, so I hardly missed something obvious. And the obvious solutions would be fun (if the alternative should be that this is a more complex problem).

:

import numpy
import matplotlib
matplotlib.use("AGG")
import matplotlib.pylab as plt

def makeFrame(srcName, dstName, coloring, sideLength,
              dataRanges, delim, dpi):
    v,V,cmap = coloring
    n = sideLength
    xmin,xmax,ymin,ymax = dataRanges
    A = numpy.empty((n,n),float)
    dx = (xmax-xmin) / (n-1)
    dy = (ymax-ymin) / (n-1)

    srcfile = open(srcName,'rb')
    for line in srcfile:
        lineVals = line[:-1].split(delim)
        x = float(lineVals[0])
        y = float(lineVals[1])
        c = float(lineVals[2])

        #Find index from float value, adjust for rounding
        i = (x-xmin) / dx
        if (i - int(i) ) > .05: i += 1

        j = (y-ymin) / dy
        if (j - int(j) ) > .05: j += 1

        A[i,j] = c
    srcfile.close()
    print "loaded vals"

    fig = plt.figure(1)
    fig.clf()
    ax = fig.gca()
    ScalarMap = ax.pcolorfast(A, vmin = v, vmax = V, cmap = cmap)
    fig.colorbar(ScalarMap)
    ax.axis('image')
    fig.savefig(dstName, dpi = dpi)
    plt.close(1)
    print "saved image"

Answer 1

:

• , .
• . matplotlib.cbook.report_memory() , , .

, , , , , : multiprocessing, . , , . , , , .

- :

import matplotlib.cbook as mc
import multiprocessing as mp
import matplotlib.cm as cm

if __name__=='__main__':
    for _ in range(10):
        srcName='test.data'
        dstName='test.png'
        vmin = 0
        vmax = 5
        cmap = cm.jet
        sideLength = 500
        dataRanges = (0.0,1.0,0.0,1.0)
        delim = ','
        dpi = 72
        proc=mp.Process(target=makeFrame,args=(
            srcName,dstName,(vmin,vmax,cmap),sideLength,
            dataRanges,delim,dpi))
        proc.start()
        proc.join()
        usage=mc.report_memory()
        print(usage)