CloudWatch login from EC2 instances

Question

Currently, my EC2 servers host a website that logs each registered user action in a separate log file on a local EC2 instance, for example username.log. I am trying to find a way to make log events for them in CloudWatch using the PHP SDK without slowing down the application, and while maintaining a separate log file for each registered member of my website.

I cannot understand this for life:

OPTION 1: How can I log into CloudWatch asynchronously using the CloudWatch SDK? My PHP application is very slow, since each line of the log takes about 100 ms to go directly to CloudWatch. Sample code below.

OPTION 2: As an alternative, how can I configure the installed CloudWatch agent on EC2 to just MANDATORY store all my log files, which would basically download them asynchronously to CloudWatch for me in a separate process? The CloudWatch EC2 logging agent requires a static “configuration file” ( AWS documentation ) on your server , which, as far as I know, should contain a list of all your log files (“log streams”) that I cannot predict when the server starts. Is there any way around this (i.e. just watch ALL the log files in the directory)? The following is an example configuration file.

All ideas are welcome here, but I don’t want my solution to simply "drop all your logs into a single file so that your log names are always predictable."

!

---

1: SDK ( ~ 100 /logEvent ):

// Configuration to use for the CloudWatch client
$sharedConfig = [
    'region'  => 'us-east-1',
    'version' => 'latest',
    'http'    => [
        'verify' => false
    ]
];

// Create a CloudWatch client
$cwClient = new Aws\CloudWatchLogs\CloudWatchLogsClient($sharedConfig);

// DESCRIBE ANY EXISTING LOG STREAMS / FILES
$create_new_stream = true;
$next_sequence_id = "0";
$result = $cwClient->describeLogStreams([
        'Descending' => true,
        'logGroupName' => 'user_logs',
        'LogStreamNamePrefix' => $stream,
]);
// Iterate through the results, looking for a stream that already exists with the intended name
// This is so that we can get the next sequence id ('uploadSequenceToken'), so we can add a line to an existing log file
foreach ($result->get("logStreams") as $stream_temp) {
    if ($stream_temp['logStreamName'] == $stream) {
        $create_new_stream = false;
        if (array_key_exists('uploadSequenceToken', $stream_temp)) {
            $next_sequence_id = $stream_temp['uploadSequenceToken'];
        }
        break;
    }
}   

// CREATE A NEW LOG STREAM / FILE IF NECESSARY
if ($create_new_stream) { 
    $result = $cwClient->createLogStream([
        'logGroupName' => 'user_logs',
        'logStreamName' => $stream,
    ]);
}

// PUSH A LINE TO THE LOG *** This step ALONE takes 70-100ms!!! ***

$result = $cwClient->putLogEvents([
    'logGroupName' => 'user_logs',
    'logStreamName' => $stream,
    'logEvents' => [
        [
            'timestamp' => round(microtime(true) * 1000),
            'message' => $msg,
        ],
    ],
    'sequenceToken' => $next_sequence_id
]);

2: CloudWatch ( , , , ):

[general]
state_file = /var/awslogs/state/agent-state  

[applog]
file = /var/www/html/logs/applog.log
log_group_name = PP
log_stream_name = applog.log
datetime_format = %Y-%m-%d %H:%M:%S