Avoiding Old Results in AWS Cloud Routing API

Question

We use the AWS cloud route to retrieve data (cloud route events). We used the gem 'aws-sdk-cloudtrail' (version 1.0). According to Cloud Trail, we can get a maximum of 50 results (last time). To get the previous (older) results, we use the "next character" obtained in the previous answer. We do this until we get an empty "next character". When we get an empty token, it means that all the cloud footprint data has been received.

For example: Assume that 100 events are registered in Cloud Trail: In the first api call, we got the last 50 results along with the token to get the next 50 (older than 50). In the second api call, we get the remaining 50 results (older results) along with the next token as nil. This means that there will be no more results.

In our case, we save all the results obtained from the trail in our local database. We repeat this periodically. When you do this a second time (repeating the process explained above), we again get some new and some older results. We repeat the API call again until we get the “next token” as zero. This results in redundant data that has already been stored in the database during the first cycle. Is there any way to get only a second entry into the event log of the second half.

Answer 1

Like @Vorsprung, you can use the maximum event time from your local database.

Here is a detailed solution for your use case / problem:

1. Query to your local database to check that cloudtrail data is present in the local database.

    IF yes 
        // It means you have stored some data from cloudtrail before.
        // And now you are going to do request to cloudtrail for new trail events.
        // Note - At a time of the first request you don't have a token (i.e. next-token)

        GOTO Step 3

    ELSE
        // It means you have not stored any data from cloudtrail before.
        // And now you are going to do the first request to cloudtrail. 
        // Note - At a time of the first request you don't have a token (i.e. next-token)

        GOTO Step 2

2.  LOOP true

        token = nil

        IF token
            // Send request to cloudtrail to get next bactch of latest cloudtrail events, now pass token(i.e. next-token) as parameter.
            // Which will return the maximum latest 50 trail events.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END

        ELSE
            // Send request to cloudtrail to get the latest cloudtrail events.
            // Which will return the maximum latest 50 trail events.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END
        END
    END

3.  LOOP true

        token = nil
        start_date_time = max_trail_event_date_time_form_local_db

        IF token
            // Send request to cloudtrail to get next bactch of latest cloudtrail events, now pass token and start_date_time(i.e. next-token and max_event_date_time_form_local_db) as parameters.
            // Which will return the maximum latest 50 events which are logged after start_date_time.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END

        ELSE
            // Send request to cloudtrail to get the latest cloudtrail events, now pass start_date_time(i.e. max_trail_event_date_time_form_local_db) as parameter.
            // Which will return the maximum latest 50 events which are logged after start_date_time.
            // It will also return next-token if more cloudtrail events are remaining.

            IF next-token
              token = next-token
            ELSE
                BREAK LOOP;
            END
        END
    END

, .

Answer 2

, cloudtrail.

Answer 3

You save the "NextToken" in your local database and pass it the next time you call the API. Here is an example.

import boto3

cloudtrail = boto3.client('cloudtrail')
paginator = cloudtrail.get_paginator('lookup_events')

StartingToken = None

page_iterator = paginator.paginate(
    LookupAttributes=[{'AttributeKey':'EventName','AttributeValue': 'RunInstances'}],
    PaginationConfig={'PageSize':10, 'StartingToken':StartingToken })
for page in page_iterator:
    for event in page["Events"]:
        print(event["EventName"],event["EventTime"])
    try:
        token_file = open("token","w") 
        token_file.write(page["NextToken"]) 
        StartingToken = page["NextToken"]
    except KeyError:
        exit()