Multiple python threads writing different entries to the same list at the same time - is this normal?

Question

I am trying to fix an error when multiple threads are writing to a list in memory. Right now I have a thread lock and sometimes there are problems associated with the work being done in the threads.

I was hoping to just make a hash of the lists, one for each thread, and remove the thread lock. It seems that each thread can write to its own record without worrying about the others, but perhaps the very fact that they all use the same hash owner will be a problem in itself.

Does anyone know if this will work or not? If not, can I, for example, dynamically add a list to the package for each thread? Is this essentially the same?

I am far from a multithreading specialist, so any advice is welcome.

Thanks,

Answer 1

import threading def job(root_folder,my_list): for current,files,dirs in os.walk(root): my_list.extend(files) time.sleep(1) my_lists = [[],[],[]] my_folders = ["C:\\Windows","C:\\Users","C:\\Temp"] my_threads = [] for folder,a_list in zip(my_folders,my_lists): my_threads.append(threading.Thread(target=job,args=(folder,a_list) for thread in my_threads: thread.start() for thread in my_threads: thread.join() my_full_list = my_lists[0] + my_lists[1] + my_lists[2] 

this way each thread simply modifies its own list and at the end merges all the individual lists

also, as indicated, gives a zero performance boost (actually, probably slower than not slicing it ...), you can get a performance gain by using multiprocessing instead ...

Answer 2

Do not use the list. Use Queue (python2) or queue (python3). There are three types of lines: fifo, lifo and priority. Last for ordered data.

You can put data on one side (with a stream):

 q.put(data) 

And get it on the other hand (perhaps in a loop for, say, a database):

 while not q.empty: print q.get() 

https://docs.python.org/2/library/queue.html