How to do unbuffered file transfer using the ReadFile () WriteFile () API?

Question

The problem is that when I turn off the cache (setting FILE_FLAG_NO_BUFFERING in CreateFile ), I have to pass the number of bytes, which is a multiple of 512 (sector size), to the read and write functions. I use a buffer of 10 MB in size and all that ... until the last buffer operation. The last buffer does not specify several 512 bytes. So how can I read and write this last part of the file?

This is what I wrote so far ...

 procedure MarusCopyFileNoCache(SrcName,DestName:string); const BufSize = 10485760; {10MB} var Src,Dest:THandle; Buffer:Pointer; Bufs,x:integer; AllSize:int64; N,junk:DWord; begin Src:=CreateFileW(PWideChar('\\?\'+SrcName), GENERIC_READ, FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_FLAG_NO_BUFFERING, 0); Dest:=CreateFileW(PWideChar('\\?\'+DestName), GENERIC_WRITE, 0, nil, CREATE_ALWAYS, FILE_FLAG_NO_BUFFERING, 0); try AllSize:=MFileSize(Src); {this is my function to get the int64 file size} Bufs:=Ceil(AllSize/BufSize); GetMem(Buffer,BufSize); try for x:=1 to Bufs do begin ReadFile(Src, Buffer^, BufSize, N, nil); WriteFile(Dest, Buffer^, N, junk, nil); end; finally FreeMem(Buffer,BufSize); end; finally CloseHandle(Src); CloseHandle(Dest); end; end; 

Answer 1

When using FILE_FLAG_NO_BUFFERING you need to read and write full sectors, but the last buffer will be a partial sector ( ReadFile() will report fewer bytes than the full sector size). When you write this last buffer, you still have to write it as a full sector, otherwise WriteFile() will fail. Thus, the output file size may be too large. However, since you know the file size you need, you can use SetFileInformationByHandle() to set the final size of the output file after you finish writing to it and before you close its handle.

For instance:

 type FILE_INFO_BY_HANDLE_CLASS = ( FileBasicInfo = 0, FileStandardInfo = 1, FileNameInfo = 2, FileRenameInfo = 3, FileDispositionInfo = 4, FileAllocationInfo = 5, FileEndOfFileInfo = 6, FileStreamInfo = 7, FileCompressionInfo = 8, FileAttributeTagInfo = 9, FileIdBothDirectoryInfo = 10, // 0xA FileIdBothDirectoryRestartInfo = 11, // 0xB FileIoPriorityHintInfo = 12, // 0xC FileRemoteProtocolInfo = 13, // 0xD FileFullDirectoryInfo = 14, // 0xE FileFullDirectoryRestartInfo = 15, // 0xF FileStorageInfo = 16, // 0x10 FileAlignmentInfo = 17, // 0x11 FileIdInfo = 18, // 0x12 FileIdExtdDirectoryInfo = 19, // 0x13 FileIdExtdDirectoryRestartInfo = 20, // 0x14 MaximumFileInfoByHandlesClass); FILE_END_OF_FILE_INFO = record EndOfFile: LARGE_INTEGER; end; function SetFileInformationByHandle( hFile: THandle; FileInformationClass: FILE_INFO_BY_HANDLE_CLASS; lpFileInformation: Pointer; dwBufferSize: DWORD ): BOOL; stdcall; external 'kernel32.dll' delayed; 

 procedure MarcusCopyFileNoCache(SrcName, DestName: string); const BufSize = 10485760; {10MB} var Src, Dest: THandle; Buffer: PByte; FinalSize: Int64; SectorSize, N, ignored: DWORD; eof: FILE_END_OF_FILE_INFO; begin Src := CreateFile(PChar('\\?\'+SrcName), GENERIC_READ, FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_FLAG_NO_BUFFERING, 0); if Src = INVALID_HANDLE_VALUE then RaiseLastOSError; try Dest := CreateFile(PChar('\\?\'+DestName), GENERIC_WRITE, 0, nil, CREATE_ALWAYS, FILE_FLAG_NO_BUFFERING, 0); if Dest = INVALID_HANDLE_VALUE then RaiseLastOSError; try try FinalSize := 0; SectorSize := 512; // <-- TODO: determine this dynamically at runtime GetMem(Buffer, BufSize); try repeat if not ReadFile(Src, Buffer^, BufSize, N, nil) then RaiseLastOSError; if N = 0 then Break; // EOF reached Inc(FinalSize, N); // round up the number of bytes read to the next sector boundary for writing N := (N + (SectorSize-1)) and (not (SectorSize-1)); if not WriteFile(Dest, Buffer^, N, ignored, nil) then RaiseLastOSError; until False; finally FreeMem(Buffer, BufSize); end; // now set the final file size eof.EndOfFile.QuadPart := FinalSize; if not SetFileInformationByHandle(Dest, FileEndOfFileInfo, @eof, SizeOf(eof)) then RaiseLastOSError; finally CloseHandle(Dest); end; except DeleteFile(PChar(DestName)); raise; end; finally CloseHandle(Src); end; end; 

Answer 2

I think you know the answer. You need to read or write several times in sector size. This is a given.

When reading, I think this means that for the final reading, reading will be successful, but you will not read the entire buffer. This will be seen from lpNumberOfBytesRead . Good. There is no problem asking to read the full buffer, but there are only as many bytes as are left in the file.

Therefore, you do not need to check the file size, you can just continue reading until reading returns more bytes. And while we are on this subject, do not use floating point arithmetic here. Use the div and mod , integer operators.

To write, you need to write the final data block, rounded to a multiple of the sector size. It will write too much. You cannot fix this by calling SetEndOfFile with an unbuffered handle. Instead, call SetFileInformationByHandle . Skip FileEndOfFileInfo and specify the true length of the file.