Creating GCM / CBC Ciphers in Golang Streams

Question

The GCM and CBC AES ciphers in Go cannot be used with StreamWriter or StreamReader, which forces me to allocate the entire file to memory. Obviously, this is not ideal for large files.

I thought about making them streaming by allocating some fixed block size into memory and feeding them to GCM or CBC, but I guess this is probably a bad idea, as there must be a reason why they were designed like this way.

Can someone explain why these modes of operation cannot be used without allocating entire files to memory?

Answer 1

The simple answer is how they developed the API.

CBC and GCM are very different modes. GCM is AEAD (Authenticated Encryption with Associated Data) mode. Do you really need authentication? If not, CBC is suitable for large files. You can also use CTR or OFB, but they are streaming modes and very strict in choosing IV.

Before introducing anything, I suggest you read about these modes. You should at least understand what parameters they need, for what purpose and how they should be generated.

Cbc

BlockMode interface is suitable for encrypting large files. You just need to encrypt block by block. CBC requires filling, but Go has no implementation. At least I don't see him. You will have to handle it somehow.

Gcm

GCM uses the AEAD interface, which allows you to encrypt and decrypt the whole message. There is absolutely no reason why this should be implemented like this. GCM is a streaming mode, it is really suitable for streaming encryption. The only problem is authentication. GCM creates a tag at the end that acts like a MAC. To use this tag, you cannot simply encrypt an endless stream of data. You must break it into pieces and authenticate separately. Or do something else, but at some point you should read this tag and check it, otherwise it makes no sense to use GCM.

What some libraries do, including Go, they add this tag at the end implicitly during encryption and read and verify it when decrypting. Personally, I think this is a very bad design. The tag should be accessible as a separate object, you cannot simply assume that it will always be at the end. And this is not the only problem in implementing Go. Sorry for this pomp. I have come across a particularly poor implementation lately.

As a solution, I suggest you split the stream into pieces and encrypt them separately with a unique nonce (which is very important). Each snippet will have a tag at the end that you should check. This way you can use GCM authentication. Yes, this is disgusting, but Go does not give you access to internal methods, so you can create your own encryption API.

Alternatively, you may find another implementation. Maybe even a C library. I can offer mbedtls. For me, this is the best implementation I've come across in terms of APIs.

Answer 2

Here is a stream implementation for reading from BlockMode. Your mileage may vary.

 type BlockReader struct { buf []byte block cipher.BlockMode in io.Reader } func NewBlockReader(blockMode cipher.BlockMode,reader io.Reader) *BlockReader { return &BlockReader{ block: blockMode, in: reader, } } func (b *BlockReader) Read(p []byte) (n int, err error) { toRead := len(p) mul := toRead/b.block.BlockSize() size := mul*b.block.BlockSize() if cap(b.buf) != size{ b.buf = make([]byte,toRead,toRead) } read, err := b.in.Read(b.buf) if err != nil { return 0,err } if read < b.block.BlockSize(){ return 0,io.ErrUnexpectedEOF } b.block.CryptBlocks(b.buf,b.buf) return copy(p,b.buf),nil }