How do you encrypt large files / byte streams in Go?

I have some large files that I would like to encrypt before sending them to a wire or saving to disk. Although it seems to encrypt streams , it seems like a warning against doing this , and instead, people recommend breaking files into pieces and using GCM or crypto / nacl / secretbox.

Processing data streams is more difficult due to the requirement of authenticity. We cannot encrypt-then-MAC: by nature we usually do not know the size of the stream. We cannot send the MAC after the stream ends, as this usually indicates that the stream is closed. We cannot decrypt the stream on the fly, because we need to see all the ciphertext to check the MAC. Trying to provide a stream adds enormous complexity to the problem, without good answers. The solution is to break the stream into discrete pieces and treat them like messages.

Files are segmented into 4KiB blocks. Each block receives a new random 128 bit IV bit each time it changes. A 128-bit authentication tag (GHASH) protects each block from changes.

If a large amount of data is decrypted, it is not always possible to buffer all decrypted data until the authentication tag has been verified. Dividing the data into small fragments eliminates the problem of deferred authentication, but introduces a new one. The pieces can be reordered ...... because each piece is encrypted separately. Therefore, the order of the pieces must be encoded into pieces in some way in order to be able to detect the permutation of any number of pieces.

Can anyone with real cryptographic experience point me in the right direction?

Update

I realized by asking this question that there is a difference between the simple lack of the ability to put the entire byte stream into memory (10 GB file encryption), and the byte stream is also an unknown length that can continue long before the stream starts to decode (24-hour live video stream).

What interests me most is the big drops, where the end of the stream can be reached before the start needs to be decoded. In other words, encryption that does not require all plaintext / encrypted text to be loaded into memory at the same time.

+5
source share
2 answers

As you already discovered in your research, there is no pretty elegant solution for authenticated encryption of large files.

Traditionally, there are two ways to solve this problem:

  • Divide the file into pieces, encrypt each fragment separately and let each piece have its own authentication tag. AES-GCM would be the best way to do this. This method causes the file size to swell in proportion to the file size. You will also need a unique nonce for each fragment. You also need a way to indicate where the pieces start / end.

  • Encrypt using AES-CTR using a buffer, call Hash.Write on the HMAC for each encrypted data buffer. The advantage of this is that encryption can be done in one go. The disadvantage is that decryption requires one pass to verify the HMAC, and then another pass to actually decrypt. It is significant that the file size remains the same, plus about 48 or so bytes for the result of IV and HMAC.

None of them are perfect, but for very large files (~ 2 GB or more) the second option is preferable.

I included the encryption example in Go using the second method below. In this case, the last 48 bytes are IV (16 bytes) and the result is HMAC (32 bytes). Also pay attention to HMACing IV.

 const BUFFER_SIZE int = 4096 const IV_SIZE int = 16 func encrypt(filePathIn, filePathOut string, keyAes, keyHmac []byte) error { inFile, err := os.Open(filePathIn) if err != nil { return err } defer inFile.Close() outFile, err := os.Create(filePathOut) if err != nil { return err } defer outFile.Close() iv := make([]byte, IV_SIZE) _, err = rand.Read(iv) if err != nil { return err } aes, err := aes.NewCipher(keyAes) if err != nil { return err } ctr := cipher.NewCTR(aes, iv) hmac := hmac.New(sha256.New, keyHmac) buf := make([]byte, BUFFER_SIZE) for { n, err := inFile.Read(buf) if err != nil && err != io.EOF { return err } outBuf := make([]byte, n) ctr.XORKeyStream(outBuf, buf[:n]) hmac.Write(outBuf) outFile.Write(outBuf) if err == io.EOF { break } } outFile.Write(iv) hmac.Write(iv) outFile.Write(hmac.Sum(nil)) return nil } 
+3
source

Using HMAC after encryption is a valid method. However, HMAC can be quite slow, especially if SHA-2 is used. You could do the same with GMAC, the base GCM MAC. It may be difficult to find an implementation, but GMAC is above the ciphertext, so you can just execute it separately if you want. There are other methods, such as Poly1305 with AES, which are used for TLS 1.2 and 1.3.

For GCM (or CCM or EAX or any other authenticated cipher) you need to authenticate the order of the pieces. You can do this by creating a separate file encryption key, and then using the nonce input (12-byte IV) to indicate the fragment number. This will solve the IV storage problem and ensure that the pieces are in order. You can generate a file encryption key using KDF (if you have a unique way to specify a file) or by transferring a random key using the master key.

+3
source

Source: https://habr.com/ru/post/1276146/


All Articles