Read previous line in python file

Question

I need to get the value of the previous line in a file and compare it with the current line as I iterate through the file. The HUGE file, so I can’t read the whole file or accidentally get the line number using linecache , because the library function still reads the whole file in memory all the time.

EDIT I'm sorry that I forgot the mention that I need to read the file back.

EDIT2

I tried the following:

  f = open("filename", "r") for line in reversed(f.readlines()): # this doesn't work because there are too many lines to read into memory line = linecache.getline("filename", num_line) # this also doesn't work due to the same problem above. 

Answer 1

Just save the previous one when you go to the next

 prevLine = "" for line in file: # do some work here prevLine = line 

This will save the previous line to prevLine during the loop.

edit , obviously, OP should read this file back:

aaand after an hour of research, I tried several times to execute it within the limits of memory

Here you go to Lim, this guy knows what he is doing, here is his best idea:

General Approach # 2: Read the entire file, keep the line position

With this approach, you also read the entire file once, but instead of saving the entire file (all text) in memory, you only save the binary positions inside the file where each line began. You can store these positions in a similar data structure by storing rows in the first approach.

You want to read line X, you need to re-read the line from the file, starting from the position that you saved to start this line.

Pros: It's almost as easy to implement as the first approach Cons: It may take some time to read large files

Answer 2

@Lim, here's how I would write (response to comments)

 def do_stuff_with_two_lines(previous_line, current_line): print "--------------" print previous_line print current_line my_file = open('my_file.txt', 'r') if my_file: current_line = my_file.readline() for line in my_file: previous_line = current_line current_line = line do_stuff_with_two_lines(previous_line, current_line) 

Answer 3

I would write a simple generator for the task:

 def pairwise(fname): with open(fname) as fin: prev = next(fin) for line in fin: yield prev,line prev = line 

Or you can use the pairwise itertools recipe:

 def pairwise(iterable): "s -> (s0,s1), (s1,s2), (s2, s3), ..." a, b = itertools.tee(iterable) next(b, None) return itertools.izip(a, b)