Pickle Queue objects in python

Question

I have a class that uses a list of Queue objects. I need to sort this class, including the information stored in the queue objects. For instance:

import Queue import pickle class QueueTest(object): def __init__(self): self.queueList = [] def addQueue(self): q = Queue.Queue() q.put('test') self.queueList.append(q) obj = QueueTest() obj.addQueue() with open('pickelTest.dat','w') as outf: pickle.dump(obj,outf) 

returns an error

 raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle lock objects 

Is there a way to get around Queue objects?

Answer 1

I suggest replacing the use of Queue.Queue with collections.deque . The Queue class is designed for synchronous communication between threads, so when used as a regular data structure, it will have unnecessary overhead. collections.deque is a faster alternative. (The name “deque” is pronounced “deck” and means “double line.”)

The deque class has a different API than the Queue type, but it is fairly easy to translate between them. Use deque.append instead of Queue.put and deque.popleft instead of q.get() (or appendleft and pop if you like to go in the other direction). Instead of calling Queue.empty just use the deque instance as a boolean (for example, to check for an empty list).

deque instances are picklable:

 >>> import collections, pickle >>> q = collections.deque(["test"]) >>> pickle.dumps(q) b'\x80\x03ccollections\ndeque\nq\x00]q\x01X\x04\x00\x00\x00testq\x02a\x85q\x03Rq\x04.' 

Answer 2

As you commented on @Blckknght, you don't need the Queue.Queue sync Queue.Queue . So just use collections.deque so that the Queue.Queue class uses itself as the queue data structure. You will need to use .appendleft to emulate FIFO Queue.put and .pop to emulate Queue.get