How can you upload files as a stream?

Question

There are a number of tutorials on file allocation using http.Request in go, but almost always they start as follows:

 file, err := os.Open(path) if err != nil { return nil, err } fileContents, err := ioutil.ReadAll(file) 

Let's say you read the entire file in memory, and then convert it to Buffer and pass it into a request, something like this:

 func send(client *http.Client, file *os.File, endpoint string) { body := &bytes.Buffer{} io.Copy(body, file) req, _ := http.NewRequest("POST", endpoint, body) resp, _ := client.Do(req) } 

If you want to publish a massive file and not read it in memory, instead store the file in pieces ... how would you do it?

Answer 1

If you need to set Content-Length , this can be done manually. The following snippet is an example of downloading a file and additional parameters in the form of a stream (code based on the Buffer without a multi-page POST in the Golang )

 //NOTE: for simplicity, error check is omitted func uploadLargeFile(uri, filePath string, chunkSize int, params map[string]string) { //open file and retrieve info file, _ := os.Open(filePath) fi, _ := file.Stat() defer file.Close() //buffer for storing multipart data byteBuf := &bytes.Buffer{} //part: parameters mpWriter := multipart.NewWriter(byteBuf) for key, value := range params { _ = mpWriter.WriteField(key, value) } //part: file mpWriter.CreateFormFile("file", fi.Name()) contentType := mpWriter.FormDataContentType() nmulti := byteBuf.Len() multi := make([]byte, nmulti) _, _ = byteBuf.Read(multi) //part: latest boundary //when multipart closed, latest boundary is added mpWriter.Close() nboundary := byteBuf.Len() lastBoundary := make([]byte, nboundary) _, _ = byteBuf.Read(lastBoundary) //calculate content length totalSize := int64(nmulti) + fi.Size() + int64(nboundary) log.Printf("Content length = %v byte(s)\n", totalSize) //use pipe to pass request rd, wr := io.Pipe() defer rd.Close() go func() { defer wr.Close() //write multipart _, _ = wr.Write(multi) //write file buf := make([]byte, chunkSize) for { n, err := file.Read(buf) if err != nil { break } _, _ = wr.Write(buf[:n]) } //write boundary _, _ = wr.Write(lastBoundary) }() //construct request with rd req, _ := http.NewRequest("POST", uri, rd) req.Header.Set("Content-Type", contentType) req.ContentLength = totalSize //process request client := &http.Client{} resp, err := client.Do(req) if err != nil { log.Fatal(err) } else { log.Println(resp.StatusCode) log.Println(resp.Header) body := &bytes.Buffer{} _, _ = body.ReadFrom(resp.Body) resp.Body.Close() log.Println(body) } } 

Answer 2

It turns out that you can actually pass the *File object (or any stream) directly to NewRequest .

Note that NewRequest (as shown here: https://golang.org/src/net/http/request.go?s=21674:21746#L695 ) will not actually set ContentLength if the stream is explicitly one of:

• * bytes.Buffer
• * bytes.Reader
• * strings.Reader

Since *File not one of them, the request will be sent without the length of the content if you did not install it manually, which may cause some servers to drop the body of the incoming request, leaving the body on the server when it seems was sent right off the bat.