How to load multiple files into the same arrays in python with loadtxt?

Question

I have several table files and I upload them using

x, y, rho, phi = np.loadtxt(myfile, unpack=True) 

Now consider the case when I have myfile1 and myfile2 with the same format. How to load them into the same arrays (as if there was one file)?

Answer 1

There are several ways to do this (for example, you can load both arrays, then combine them through x = numpy.concatenate((x1, x2)) , etc.), but what I would do is merge the files into fly and transfer the result to loadtxt .

As in the documentation , fname can be:

The file, file name, or generator to read. If the file name extension is -.gz or .bz2, the file is unpacked first. Note that generators must return byte strings for Python 3k.

So you just need a generator that generates myfile1 and then myfile2.

The obvious way to do this with itertools :

 with open(myfile1, 'rb') as f1, open(myfile2, 'rb') as f2: x, y, rho, phi = numpy.loadtxt(itertools.chain(f1, f2)) 

You may notice that I settled on unpack=True . This is because unpack only works if you pass a file name, not a file object or generator. (He looks at the extension, not at the magic of files or something like that.)

If you know that the files will always be gzip or bzip2 files, you can simply replace gzip.GzipFile or bz2.BZ2File with open above.

But if you need to deal with possible compressed files, you need to do the same extension check as numpy , and then create the appropriate object (write the open_compressed function that completes this), after which it stubbornly holds open_compressed wrong decision. So, if this is a problem, I would probably put them separately and then execute numpy.concatenate .

Answer 2

You can use the standard fileinput module:

 import fileinput import glob import numpy as np data = np.loadtxt(fileinput.input(glob.glob("*.dat")))