Selective Overwriting DataFrames

Question

Let's say I install memoization with Joblib as follows (using the provided solution here ):

from tempfile import mkdtemp cachedir = mkdtemp() from joblib import Memory memory = Memory(cachedir=cachedir, verbose=0) @memory.cache def run_my_query(my_query) ... return df 

And I will say that I define a pair of queries query_1 and query_2 , it takes a lot of time to complete them.

I understand that with the code as is:

• A second call with any request will use memoized output, ie:

   run_my_query(query_1) run_my_query(query_1) # <- Uses cached output run_my_query(query_2) run_my_query(query_2) # <- Uses cached output 

• I could use memory.clear() to delete the entire cache directory

But what if I want to redo memoization for only one of the queries (for example, query_2 ) without forcing deletion on another query?

Answer 1

The library does not seem to support partial cache deletion.

You can split the functino cache into two pairs:

 from tempfile import mkdtemp from joblib import Memory memory1 = Memory(cachedir=mkdtemp(), verbose=0) memory2 = Memory(cachedir=mkdtemp(), verbose=0) @memory1.cache def run_my_query1() # run query_1 return df @memory2.cache def run_my_query2() # run query_2 return df 

Now you can selectively clear the cache:

 memory2.clear() 

UPDATE after viewing behzad.nouri's comment:

You can use the call method of the decorated function. But, as you can see in the following example, the return value is different from a regular call. You have to take care of this.

 >>> import tempfile >>> import joblib >>> memory = joblib.Memory(cachedir=tempfile.mkdtemp(), verbose=0) >>> @memory.cache ... def run(x): ... print('called with {}'.format(x)) # for debug ... return x ... >>> run(1) called with 1 1 >>> run(2) called with 2 2 >>> run(3) called with 3 3 >>> run(2) # Cached 2 >>> run.call(2) # Force call of the original function called with 2 (2, {'duration': 0.0011069774627685547, 'input_args': {'x': '2'}})