Io.Copy cause out of memory in golang

Question

I use io.Copy () to copy a file of about 700 MB, but this causes a lack of memory

bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)

//key step
fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName)
if err != nil {
    return nil, err
}

file, err := os.Open(fileName) //the file size is about 700Mb
if err != nil {
    return nil, err
}
defer file.Close()

//iocopy
copyLen, err := io.Copy(fileWriter, file) // this cause out of memory
if err != nil {
    fmt.Println("io.copy(): ", err)

    return nil, err
}

The following error message:

runtime: memory allocated by OS (0x752cf000) not in usable range [0x18700000,0x98700000)
runtime: out of memory: cannot allocate 1080229888-byte block (1081212928 in use)
fatal error: out of memory

I allocate enough memory for buf, this causes a lack of memory in bodyWriter.CreateFormFile ()

buf := make([]byte, 766509056)
bodyBuf := bytes.NewBuffer(buf)
bodyWriter := multipart.NewWriter(bodyBuf)

fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) // out of memory
if err != nil {
    return nil, err
}

Answer 1

This is because you “copy” to bodyBuf, which is a buffer in memory, forcing Go to try to allocate a memory block the size of the entire file.

Based on usage multipart, it looks like you are trying to transfer a file via http? In this case, do not pass bytes.Bufferin multipart.NewWriter, directly pass your http connection.