I use Ipython in parallel in an optimization algorithm that loops many times. Parallelism is called in a loop using the map method for the LoadBalancedView dictionary (twice), the DirectView dictionary, and the %px magic call. I am running an algorithm on an Ipython laptop.
I found that the memory consumed by both the algorithm core and one of the controllers grows steadily over time, limiting the number of cycles that I can execute (since the available memory is limited).
Using heapy , I profiled memory usage after starting about 38 thousand cycles:
Partition of a set of 98385344 objects. Total size = 18016840352 bytes. Index Count % Size % Cumulative % Kind (class / dict of class) 0 5059553 5 9269101096 51 9269101096 51 IPython.parallel.client.client.Metadata 1 19795077 20 2915510312 16 12184611408 68 list 2 24030949 24 1641114880 9 13825726288 77 str 3 5062764 5 1424092704 8 15249818992 85 dict (no owner) 4 20238219 21 971434512 5 16221253504 90 datetime.datetime 5 401177 0 426782056 2 16648035560 92 scipy.optimize.optimize.OptimizeResult 6 3 0 402654816 2 17050690376 95 collections.defaultdict 7 4359721 4 323814160 2 17374504536 96 tuple 8 8166865 8 196004760 1 17570509296 98 numpy.float64 9 5488027 6 131712648 1 17702221944 98 int <1582 more rows. Type eg '_.more' to view.>
You can see that approximately half of the memory is used by IPython.parallel.client.client.Metadata instances. A good indicator that results from map calls being cached is an instance of 401177 OptimizeResult , the same number as the number of call optimizations via lbview.map - I do not cache them in my code.
Is there a way to manage this memory usage both on the kernel and on the Ipython parallel controller (memory consumption from the user is comparable to the kernel)?
source share