Why do I need a memory error? python

Question

I have a 5gb text file and am trying to read it line by line. My file is in the format: Reviewerid <\ t> pid <\ t> date <\ t> title <\ t> body <\ n> This is my code

o = open('mproducts.txt','w')
with open('reviewsNew.txt','rb') as f1:
    for line in f1:
        line = line.strip()
        line2 = line.split('\t')
        o.write(str(line))
        o.write("\n")

But I get a memory error when I try to start it. I have an 8 gigabyte ram and 1Tb space, then why am I getting this error? I tried to read it in blocks, but then I also get this error.

MemoryError 

Answer 1

Update:

Installing 64-bit Python solves the problem.

The OP used 32-bit Python, therefore, to limit the amount of memory.

---

Reading the whole comment I think this may help you.

• (1024), .
• i.e N .
• yield itertools Python .

. N , .

:

from itertools import islice
#You can change num_of_lines
def get_lines(file_handle,num_of_lines = 10):
    while True:
        next_n_lines = list(islice(file_handle, num_of_lines))
        if not next_n_lines:
            break
        yield next_n_lines


o = open('mproducts.txt','w')

with open('reviewsNew.txt','r') as f1:
    for data_lines in get_lines(f1):
        for line in data_lines:
            line = line.strip()
            line2 = line.split('\t')
            o.write(str(line))
            o.write("\n")
o.close()