As you already discovered in your research, there is no pretty elegant solution for authenticated encryption of large files.
Traditionally, there are two ways to solve this problem:
Divide the file into pieces, encrypt each fragment separately and let each piece have its own authentication tag. AES-GCM would be the best way to do this. This method causes the file size to swell in proportion to the file size. You will also need a unique nonce for each fragment. You also need a way to indicate where the pieces start / end.
Encrypt using AES-CTR using a buffer, call Hash.Write on the HMAC for each encrypted data buffer. The advantage of this is that encryption can be done in one go. The disadvantage is that decryption requires one pass to verify the HMAC, and then another pass to actually decrypt. It is significant that the file size remains the same, plus about 48 or so bytes for the result of IV and HMAC.
None of them are perfect, but for very large files (~ 2 GB or more) the second option is preferable.
I included the encryption example in Go using the second method below. In this case, the last 48 bytes are IV (16 bytes) and the result is HMAC (32 bytes). Also pay attention to HMACing IV.
const BUFFER_SIZE int = 4096 const IV_SIZE int = 16 func encrypt(filePathIn, filePathOut string, keyAes, keyHmac []byte) error { inFile, err := os.Open(filePathIn) if err != nil { return err } defer inFile.Close() outFile, err := os.Create(filePathOut) if err != nil { return err } defer outFile.Close() iv := make([]byte, IV_SIZE) _, err = rand.Read(iv) if err != nil { return err } aes, err := aes.NewCipher(keyAes) if err != nil { return err } ctr := cipher.NewCTR(aes, iv) hmac := hmac.New(sha256.New, keyHmac) buf := make([]byte, BUFFER_SIZE) for { n, err := inFile.Read(buf) if err != nil && err != io.EOF { return err } outBuf := make([]byte, n) ctr.XORKeyStream(outBuf, buf[:n]) hmac.Write(outBuf) outFile.Write(outBuf) if err == io.EOF { break } } outFile.Write(iv) hmac.Write(iv) outFile.Write(hmac.Sum(nil)) return nil }
source share