Multiple Python Threads Writing To Different Records In Same List Simultaneously - Is This Ok?

I am trying to fix a bug where multiple threads are writing to a list in memory. Right now I have a thread lock and am occasionally running into problems that are related to the w

Solution 1:

Don't use list. Use Queue (python2) or queue (python3). There is 3 kinds of queue: fifo, lifo and priority. The last one is for ordered data.

You may put data at one side (with thread):

q.put(data)

And get at the other side (maybe in a loop for, say, database):

while not q.empty:
    print q.get()

https://docs.python.org/2/library/queue.html

Solution 2:

import threading

def job(root_folder,my_list):
    for current,files,dirs in os.walk(root):
        my_list.extend(files)
        time.sleep(1)

my_lists = [[],[],[]]
my_folders = ["C:\\Windows","C:\\Users","C:\\Temp"]
my_threads = []
for folder,a_list in zip(my_folders,my_lists):
    my_threads.append(threading.Thread(target=job,args=(folder,a_list)
for thread in my_threads:
   thread.start()
for thread in my_threads:
   thread.join()

my_full_list = my_lists[0] + my_lists[1] + my_lists[2]

this way each thread just modifies its own list and at the end combines all the individual lists

also as pointed out this gives zero performance gain (actually probably slower than not threading it... ) you may get performance gains using multiprocessing instead ...