Fast reading of 10,000 binary files?

Question

I have 10,000 binaries named like this:

file0.bin

file1.bin

............ ............

file10000.bin

Each of the above files contains exactly 391 float values ​​(1564 bytes per file).

My goal is to scan all files in a python array as quickly as possible. If I open and close each file with a script, it takes a lot of time (about 8 minutes!). Are there any other creative ways to read these FAST files?

I am using Ubuntu Linux and would prefer a solution that can work with Python. Thank.

Answer 1

If you want to make ramdisk even faster:

# mkfs -q /dev/ram1 $(( 2 * 10000)) ## roughly the size you need
# mkdir -p /ramcache
# mount /dev/ram1 /ramcache
# df -H | grep ramcache

now concat

# cat file{1..10000}.bin >> /ramcache/concat.bin ## thanks SiegeX

Then let your script in this file

, "#", . , , .

, , Q , , , , 8 .

Answer 2

, , pypy, python JIT-, .

Answer 3

10001 ( 0 10000 ), 8 ?

try: xrange # python 2 - 3 compatibility
except NameError: xrange= range

import array

final= array.array('f')

for file_seq in xrange(10001):
    with open("file%d.bin" % file_seq, "rb") as fp:
        final.fromfile(fp, 391)

? RAM? ?