Reading and writing binary files in Qt

Question

I am going to work with binary files in a Qt project and, being a bit new to Qt, I'm not sure if I should use QVector<quint8> or QByteArray to store data. Files can be very small (<1MiB) or very large (> 4GiB). Size unknown before execution.

I need to be able to search in random order and be able to handle operations on each byte in the file. Can a memory mapped file be used here?

Thanks for any suggestions.

Answer 1

Loading large files into memory, whether QVector or QByteArray , is probably not a good solution.

Assuming the files have some kind of structure, you should use QFile::seek to place yourself at the beginning of the "record" and use qint64 QIODevice::read ( char * data, qint64 maxSize ) to read one record at a time in the buffer you select .

Answer 2

QIODevice::write has an overload for QByteArray if this affects your decision. QDataStream might be worth a look at big data. In the end, it really is up to you, as various containers will work.

Edit:

I think that the basic input / output of files using any buffer that you prefer is probably all that you need. Use objects like QFile , QDataStream , QByteArray , etc. You can read and process only parts of a file using circular buffers to save memory, especially when it comes to audio, video, or something that lends itself to streams. If there is a known file structure, such as XML, CSV, etc., Which also simplifies partial reading and processing, since you can follow lines or tags by tags.

Memory mapped files use virtual memory to achieve faster I / O, mainly by making a copy of the file on disk in the virtual memory segment, which can then be used by the application as if it were just process memory. The ability to process the file as a process memory allows you to do in-place editing, which is faster than having to look for a position from the beginning of the file and faster than making OS-specific API calls and working with reading / writing to the hard drive. There, as a rule, there is a lot of overhead for files with memory mapping, and there are some possible limitations depending on how paging is implemented on your target platform or what architecture you use. In Qt, you will have to create your own objects for using memory-mapped files, and, I believe, linux systems support this functionality better than windows.