Python - reading 1000 lines from a file at a time

Question

I checked this , this and this .

The third link seemed to have an answer, but it did not do the job.

I cannot have a solution in which the entire file will be loaded into main memory, since the files I will work with will be very large. So I decided to use islice , as shown in the third link. The first 2 links were irrelevant, because they used it for only two lines or read 1000 characters. While I need 1000 lines. for now N is 1000

My file contains lines of 1 million :

Example:

 1 1 1 1 2 1 1 3 1 1 4 1 1 5 1 1 6 1 1 7 1 1 8 1 1 9 1 1 10 1 

So, if I read 1000 lines at a time, I have to go through while 1000 times, but when I type p to check how many times I went through, it does not stop at a 1000 . It reached 19038838 after starting my program in 1400 seconds!

CODE:

 def _parse(pathToFile, N, alg): p = 1 with open(pathToFile) as f: while True: myList = [] next_N_lines = islice(f, N) if not next_N_lines: break for line in next_N_lines: s = line.split() x, y, w = [int(v) for v in s] obj = CoresetPoint(x, y) Wobj = CoresetWeightedPoint(obj, w) myList.append(Wobj) a = CoresetPoints(myList) client.compressPoints(a) // This line is not the problem print(p) p = p+1 c = client.getTotalCoreset() return c 

What am I doing wrong?

Answer 1

As @ Ev.kounis said, your while loop is not working properly.

I would recommend going for the yield function for a piece of data at a time like this:

 def get_line(): with open('your file') as file: for i in file: yield i lines_required = 1000 gen = get_line() chunk = [next(gen) for i in range(lines_required)]