AWS S3 Event Notification Using Lambda Function in Java

Question

I am trying to use the Lambda function to notify about an S3 Put event. My Lambda function should be called when I add / add a new JSON file to my S3 bucket. The task I have is that there are not enough documents for this to implement such a Lambda function in Java. Most of the docs I found for Node.js

I want my Lambda function to be called, and then inside this Lambda function, I want to use this added json, and then send this JSON to AWS ES.

But what all classes should I use for this? Does anyone know about this? S3 abd ES configured and running. Automatically generated code for lambda `

@Override public Object handleRequest(S3Event input, Context context) { context.getLogger().log("Input: " + input); // TODO: implement your handler return null; } 

What's next?

Answer 1

Finally, here are the steps for integrating S3 -> Lambda -> ES using Java.

• Build your S3, Lamba, and ES on AWS. The steps are here .

• Use the Java code below in your lambda function to retrieve a newly added object in S3 and send it to the ES service.

   public Object handleRequest(S3Event input, Context context) { AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain()); for (S3EventNotificationRecord record : input.getRecords()) { String s3Key = record.getS3().getObject().getKey(); String s3Bucket = record.getS3().getBucket().getName(); context.getLogger().log("found id: " + s3Bucket+" "+s3Key); // retrieve s3 object S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key)); InputStream objectData = object.getObjectContent(); //Start putting your objects in AWS ES Service String esInput = "Build your JSON string here using S3 objectData"; HttpClient httpClient = new DefaultHttpClient(); HttpPut putRequest = new HttpPut(AWS_ES_ENDPOINT + "/{Index_name}/{product_name}/{unique_id}" ); StringEntity input = new StringEntity(esInput); input.setContentType("application/json"); putRequest.setEntity(input); httpClient.execute(putRequest); httpClient.getConnectionManager().shutdown(); } return "success";} 

• Use Postman or Sense to create the actual index and display accordingly in ES.

• After completing the download and running proxy.js on your computer. Make sure you configure ES Security steps suggested in this post.

• Test setup and Kibana by running http: // localhost: 9200 / _plugin / kibana / url from your device.

• Everything is installed. Go ahead and install your dashboard in Kiban. Check it out by adding new objects to your S3 bucket

Answer 2

S3 event handling in Lambda can be done, but you should keep in mind that the S3Event object carries a reference to the object, not the object itself. To get to the actual facility, you must use the AWS SDK yourself. A request to an S3 object in a lambda function will look like this:

 public Object handleRequest(S3Event input, Context context) { AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain()); for (S3EventNotificationRecord record : input.getRecords()) { String s3Key = record.getS3().getObject().getKey(); String s3Bucket = record.getS3().getBucket().getName(); context.getLogger().log("found id: " + s3Bucket+" "+s3Key); // retrieve s3 object S3Object object = s3Client.getObject(new GetObjectRequest(s3Bucket, s3Key)); InputStream objectData = object.getObjectContent(); //insert object into elasticsearch } return null; } 

Now the pretty tricky part is to insert this object into ElasticSearch. Unfortunately, the AWS SDK does not provide any features for this. The default approach would be to call REST against the AWS ES endpoint. There are various examples of them on how to continue calling an ElasticSearch instance.

Some people seem to come with the following project:

Jest - Elasticearch Java Rest Client