How to save data streams in S3? Does aws-sdk-go example not work?

Question

I am trying to save this data stream to S3-compatible storage. The size is not known until the end of the stream and can vary from 5 MB to ~ 500 GB.

I tried various possibilities, but did not find a better solution than to implement the outline. My best guess is to make a fixed size buffer fill it with my stream and write it to S3. Is there a better solution? Maybe a way where it is transparent to me without writing the entire stream to memory?

There is an example program in aws-sdk-go readme that takes data from stdin and writes it to S3: https://github.com/aws/aws-sdk-go#using-the-go-sdk

When I try to transfer data using pipe |, I get the following error: failed to upload object, SerializationError: failed to compute request body size caused by: seek /dev/stdin: illegal seek Am I doing something wrong, or does the example not work, as I expect?

Although I tried minio-go, PutObject () or client.PutObjectStreaming () . It is functional, but consumes as much memory as it stores data.

• Is there a better solution?
• Is there a small sample program that can transfer arbitrary data to S3?

Answer 1

sdk Uploader , os.Stdin "" io.Reader. , Uploader, io.Reader, , Seeker, , Seek . os.Stdin - *os.File, Seeker, , PutObjectWithContext.

Uploader , , , .

, , .

package main

import (
    // ...
    "io"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
)

type reader struct {
    r io.Reader
}

func (r *reader) Read(p []byte) (int, error) {
    return r.r.Read(p)
}

func main() {
    // ... parse flags

    sess := session.Must(session.NewSession())
    uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
        u.PartSize = 20 << 20 // 20MB
        // ... more configuration
    })

    // ... context stuff

    _, err := uploader.UploadWithContext(ctx, &s3manager.UploadInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
        Body:   &reader{os.Stdin},
    })

    // ... handle error
}

, minio-go, , .