Multithreaded file access

Question

I am writing a hex editor program and I was thinking about when a user tries to open a very large file (3 GB +). I would not want the user to sit all day to download the entire file when he has already downloaded some data.

So, here is my question: is it possible for several streams to read the file (do not write) at the same time, in different places, and then, as soon as a certain data threshold has been read by 1, this stream displays its data and others continue to read? Will this offer me a performance improvement? Or will the memory bandwidth reduce any speed increase I could get from using multiple threads?

Answer 1

For a hex editor, there is no need to read the entire file into memory. The user can view or modify the data, but not insert or delete.

You can simply use memory mapped files. Data will be automatically read at access, and only the displayed fragment will be displayed. This provides fast scrolling and transition to any place in the file.

Answer 2

You probably don't want to use multiple threads. Even on a multi-core processor, there is still only one path to the disk, so you probably won't get a performance boost (disk access is much slower than memory).

You have a good idea, though with loading and displaying small bits at a time. Just do it in one thread. Read about the first megabyte, show it, and do the following in the background, etc.

And you're right that you might need a separate thread for the GUI. This is one of the reasons BeOS was so incredibly responsive compared to other OSs of that time. He used many different threads for different tasks.

Just don't expect multiple threads to be read from disk.

Alternatively, you can use aio_read() for asynchronous I / O on Linux. If you use Windows, just try googling "windows asynchronous io" (I'm not quite sure how you do this, I don't use Windows).

Answer 3

I’m not sure what kind of impulse you expect. There is one stream of data coming out of the disk, and having several streams read from the disk will just increase the conflict and possibly lead to a slowdown as the disk head bounces back and forth due to competing requests.

You should look into asynchronous I / O instead of processing the data as soon as it appears to make your application look responsive.

Answer 4

forget about reading the whole file. just read the small blocks when the user needs it. this is even easier in a hex editor, as the content does not affect the layout.

screen data is read in milliseconds, the user does not understand that this is done when moving, and does not read all the data in advance

Answer 5

As @bill said, you'll want to use memory mapped files. I think you will find the following tutorials very valuable:

• Memory management for large file editors
• Memory Management for Large File Editors (Part 2)

In the above manuals, you must provide all the information you need.

Answer 6

I think you would be better off using asynchronous or non-blocking I / O. This means that you can send a read request, and then continue processing, and then go to get the results of the request. Thus, a single thread can overlap processing and I / O. A little googling will find API documents for your platform.