Effective on-site dictionary filtering

Question

We have a dictionary d1 and a cond condition. We want d1 to contain only values ​​satisfying the cond condition. One way to do this:

 d1 = {k:v for k,v in d1.items() if cond(v)} 

But this creates a new dictionary, which can be very memory inefficient if d1 large.

Another variant:

 for k,v in d1.items(): if not cond(v): d1.pop(k) 

But this changes the dictionary during its repetition and generates an error: "RuntimeError: the dictionary changed size during iteration".

What is the correct way in Python 3 to filter a dictionary in place?

Answer 1

If there are not many keys whose corresponding values ​​satisfy the condition, then you can compile the keys first and then trim the dictionary:

 for k in [k for k,v in d1.items() if cond(v)]: del d1[k] 

If the list [k for k,v in d1.items() if cond(v)] is too large, you can process the dictionary "in turn", i.e. collect keys until their number exceeds a threshold value, cuts off the dictionary and repeats until there are no more keys satisfying the condition:

 from itertools import islice def prune(d, cond, chunk_size = 1000): change = True while change: change = False keys = list(islice((k for k,v in d.items() if cond(v)), chunk_size)) for k in keys: change = True del d[k]