My program should read fragments from a huge binary file with random access. I have a list of offsets and lengths that can have several thousand entries. The user selects the record, and the program searches for the offset and reads the length bytes.
The program internally uses TMemoryStream to store and process fragments read from a file. Data is read using TFileStream as follows:
FileStream.Position := Offset;
MemoryStream.CopyFrom(FileStream, Size);
This works fine, but unfortunately it is getting slower as the files get bigger. File size starts with a few megabytes, but often reaches several tens of gigabytes. Cheats about 100 kilobytes in size.
The contents of the file are read only by my program. This is the only program that accessed the file at that moment. In addition, files are stored locally, so this is not a network problem.
I am using Delphi 2007 in a windows XP window.
What can I do to speed up access to this file?
edit:
- File access is slow for large files, regardless of which part of the file is being read.
- A program usually does not read a file sequentially. The order of the pieces is user-controlled and cannot be predicted.
- It is always slower to read a piece from a large file than to read an equally large fragment from a small file.
- I'm talking about performance for reading a fragment from a file, and not about the total time it takes to process a whole file. The latter will obviously take longer for large files, but this is not a problem here.
: , , , , . , - , , . . ( ), , , - , .
, .