Python Etching Dictionary EOFError

Question

I have several scripts running on a server that saw and paste various dictionaries. They all use the same base code for etching, as shown below:

SellerDict=open('/home/hostadl/SellerDictkm','rb') SellerDictionarykm=pickle.load(SellerDict) SellerDict.close() SellerDict=open('/home/hostadl/SellerDictkm','wb') pickle.dump(SellerDictionarykm,SellerDict) SellerDict.close() 

All scripts work fine, except for one of them. One that has problems gets to various sites and dumps data and stores it in a dictionary. This code works all day etching and putting up dictionaries and stops at midnight. Then a cronjob starts it again the next morning. This script can run for several weeks without problems, but about once a month the script dies due to EOFError when trying to open a dictionary. The size of dictionaries is usually around 80 MB. I even tried adding SellerDict.flush () before SellerDict.close () when poisoning the data to make sure the evening was red.

Any idea what could be causing this? Python is pretty solid, so I don't think this is due to file size. When the code works fine for a long time before dying, it makes me think that maybe something is stored in the dictionary that causes this problem, but I have no idea.

Also, if you know the best way to save dictionaries other than pickle, I am open to options. As I said, dictionaries are constantly opening and closing. Just to clarify, only one program will use the same dictionary so that the problem is not caused by several programs trying to access the same dictionary.

UPDATE:

Here is the trace that I have from the log file.

 Traceback (most recent call last): File "/home/hostadl/CompileRecentPosts.py", line 782, in <module> main() File "/home/hostadl/CompileRecentPosts.py", line 585, in main SellerDictionarykm=pickle.load(SellerDict) EOFError 

Answer 1

So this actually turned out to be a memory problem. When the computer finishes working with RAM and tries to open or load data, the process will not require this EOFError. I am increasing the RAM on the computer and this has never been a problem again.

Thanks for all the comments and help.

Answer 2

Here's what happens when you don't use lock:

 import pickle # define initial dict orig_dict={'foo':'one'} # write dict to file writedict_file=open('./mydict','wb') pickle.dump(orig_dict,writedict_file) writedict_file.close() # read the dict from file readdict_file=open('./mydict','rb') mydict=pickle.load(readdict_file) readdict_file.close() # now we have new data to save new_dict={'foo':'one','bar':'two'} writedict_file=open('./mydict','wb') #pickle.dump(orig_dict,writedict_file) #writedict_file.close() # but...whoops! before we could save the data # some other reader tried opening the file # now they are having a problem readdict_file=open('./mydict','rb') mydict=pickle.load(readdict_file) # errors out here readdict_file.close() 

Here's the conclusion:

 python pickletest.py Traceback (most recent call last): File "pickletest.py", line 26, in <module> mydict=pickle.load(readdict_file) # errors out here File "/usr/lib/python2.6/pickle.py", line 1370, in load return Unpickler(file).load() File "/usr/lib/python2.6/pickle.py", line 858, in load dispatch[key](self) File "/usr/lib/python2.6/pickle.py", line 880, in load_eof raise EOFError EOFError 

In the end, some reading process will try to read the pickled file, while the writing process is already open for writing. You need to make sure that you have a way to determine if another process has a file that is open for writing before you try to read it.

For a very simple solution, check out this thread that discusses using Filelock .