Download CSV stream from Ruby to S3

Question

I am dealing with potentially huge CSV files that I want to export from my Rails application, and since it runs on Heroku, my idea was to transfer these CSV files directly to S3 when they were created.

Now I have a problem that a Aws::S3file is waiting to be able to perform a download, while in my Rails application I would like to do something like:

S3.bucket('my-bucket').object('my-csv') << %w(this is one line)

How can i achieve this?

Answer 1

s3 = Aws::S3::Resource.new(region:'us-west-2')
obj = s3.bucket.object("#{FOLDER_NAME}/#{file_name}.csv")
file_csv = CSV.generate do |csv|
    csv << ActionLog.column_names
    ActionLog.all.each do |action_log|
      csv << action_log.attributes.values
    end
  end
  obj.put body: file_csv

file_csv = CSV.generateis to create a CSV data string in Ruby. After creating this CSV line we put in S3 using bucket with outline

#{FOLDER_NAME}/#{file_name}.csv

ActionLog.

Answer 2

s3, , . https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

, aws-sdk-ruby V3 upload_stream, , , . . https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#upload_stream-instance_method

client = Aws::S3::Client.new(
  region: 'ap-northeast-1',
  credentials: your_credential
)

obj = Aws::S3::Object.new('your-bucket-here', 'path-to-output', client: client)

require "csv"
obj.upload_stream do |write_stream|
  [
    %w(this is first line),
    %w(this is second line),
    %w(this is third line),
  ].each do |line|
    write_stream << line.to_csv
  end
end

this,is,first,line
this,is,second,line
this,is,third,line

Answer 3

http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html#write-instance_method, , .

EDIT http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjSingleOpRuby.html may be more relevant since the first link points to ruby ​​aws-sdk v1

require 'aws-sdk'

s3 = Aws::S3::Resource.new(region:'us-west-2')
obj = s3.bucket('bucket-name').object('key')

# string data
obj.put(body: 'Hello World!')

# IO object
File.open('source', 'rb') do |file|
  obj.put(body: file)
end