Insert line in large file

Question

I have a large file (several GB) with text.

For example, it has the following text:

Hello, World! 

I need to insert the word “funny” at position 5 and offset the rest of the text:

 Hello, funny World! 

How can I not read the entire file to compensate for the rest? Or how can I optimize this operation?

Thanks.

Answer 1

You can not. Plain text files cannot be compressed or expanded at the beginning or middle of a file, but only at the end.

Answer 2

Well, you can’t, please see this for more information. How to change a text file in Python?

Answer 3

If your file is several gigabytes, then probably my solution will only apply to 64-bit operating systems:

 from __future__ import with_statement import mmap, os def insert_string(fp, offset, some_bytes): # fp is assumedly open for read and write fp.seek(0, os.SEEK_END) # now append len(some_bytes) dummy bytes fp.write(some_bytes) # some_bytes happens to have the right len :) fp.flush() file_length= fp.tell() mm= mmap.mmap(fp.fileno(), file_length) # how many bytes do we have to shift? bytes_to_shift= file_length - offset - len(some_bytes) # now shift them mm.move(offset + len(some_bytes), offset, bytes_to_shift) # and replace the contents at offset mm[offset:offset+len(some_bytes)]= some_bytes mm.close() if __name__ == "__main__": # create the sample file with open("test.txt", "w") as fp: fp.write("Hello, World!") # now operate on it with open("test.txt", "r+b") as fp: insert_string(fp, 6, " funny") 

NB: this is a Python 2 program on Linux. YMMV.