Skip last line of CSV file when repeating in Python

Question

I am working on data analysis using a CSV file that I received from a data warehouse (Cognos). The CSV file has the last line that sums all the lines above, but I do not need this line for my analysis, so I would like to skip the last line.

I was thinking of adding an if statement that checks the column name in my for loop, as shown below.

import CSV with open('COGNOS.csv', "rb") as f, open('New_COGNOS.csv', "wb") as w: #Open 2 CSV files. One to read and the other to save. CSV_raw = csv.reader(f) CSV_new = csv.writer(w) for row in CSV_raw: item_num = row[3].split(" ")[0] row.append(item_num) if row[0] == "All Materials (By Collection)": break CSV_new.writerow(row) 

However, this is like spending a lot of resources. Is there any pythonian way to skip the last line when repeating through a CSV file?

Answer 1

You can write a generator that returns everything except the last entry in the input iterator:

 def skip_last(iterator): prev = next(iterator) for item in iterator: yield prev prev = item 

then wrap the CSV_raw reader CSV_raw with:

 for row in skip_last(CSV_raw): 

The generator basically takes the first record, then starts the loop and at each iteration displays the previous record. When the input iterator is completed, there remains one more line that never returns.

The general version, allowing to skip the last elements of n , will be:

 from collections import deque from itertools import islice def skip_last_n(iterator, n=1): it = iter(iterator) prev = deque(islice(it, n), n) for item in it: yield prev.popleft() prev.append(item) 

Answer 2

Generalized skip-n generator

 from __future__ import print_function from StringIO import StringIO from itertools import tee s = '''\ 1 2 3 4 5 6 7 8 ''' def skip_last_n(iterator, n=1): a, b = tee(iterator) for x in xrange(n): next(a) for line in a: yield next(b) i = StringIO(s) for x in skip_last_n(i, 1): print(x, end='') 1 2 3 4 5 6 7 i = StringIO(s) for x in skip_last_n(i, 3): print(x, end='') 1 2 3 4 5