Memory error due to huge input file size

Question

When I use the following code to read a file:

lines=file("data.txt").read().split("\n") 

I have the following error

 MemoryError 

file size

 ls -l -rw-r--r-- 1 charlie charlie 1258467201 Sep 26 12:57 data.txt 

Answer 1

Obviously, the file is too large to be immediately read into memory.

Why not just use:

 with open("data.txt") as myfile: for line in myfile: do_something(line.rstrip("\n")) 

or if you are not on Python 2.6 or later:

 myfile = open("data.txt") for line in myfile: do_something(line.rstrip("\n")) 

In both cases, you will get an iterator that can be processed in the same way as a list of strings.

EDIT: since your way of reading the entire file into one large line and then splitting the lines into newlines will remove newlines in this process, I added .rstrip("\n") to my examples to better simulate the result.

Answer 2

use this code to read the file line by line:

 for line in open('data.txt'): # work with line