Blue Team Lab — AWS Security Incident Investigation with Splunk

6 min readJan 19, 2024

Hello,

Your organization uses AWS for hosting critical data and applications. An incident has been reported involving unauthorized data access and potential exfiltration. The organization’s security team has detected unusual activities and needs to investigate the incident to understand the scope, identify the attacker, and prevent further data breaches.

Logs came from AWS to Splunk. and you have to do your research on Splunk. When you enter the lab at CyberDefenders, the following section welcomes you.

I get to know the environment I will examine in detail.I take notes somewhere. When I entered AWSRaid on the desktop, it sent me directly to the relevant index. Therefore, find the relevant index again and
I won’t bother finding the relevant sourcetype.

If you ask what are indexes and sourcetypes:

Indexes in the Splunk environment, that is, data indexing, after meaningful data arrives, I need to store it in a way that I can quickly search on it, in short, indexing provides this.

In Siem environments such as Splunk, raw data, that is, raw log, is received, that data is processed, then indexed and presented to the end user. In other words, the aim is to ensure that data can be queried and analyzed more quickly and effectively.

After scanning the index and sourcetype sections and general fields, I start a search for a few AWS fields that catch my attention.

For example, Eventname, userIdentity.userName field, requestParameters.bucketName fields attract my attention.
And also, for the fields that do not appear in the area I will be examining, I call them more fields and add them to the fields, as there were no ticks in the fields related to the users.

And then, for example, I say bring me all eventNames with the | stats count by [fieldname] command.

From here, I especially want to check the eventNames whose count is 1 (this is my personal preference).

The eventNames here represent IAM (Identity and Access Management) capabilities associated with AWS. These are used to manage access to AWS IAM users and resources. Some of them are:

In the splunk in this lab, these data come as eventName. I can also go by looking at the EventNames that I suspect or that I definitely need to look at. Otherwise, I can proceed directly with questions.

You can find a more comprehensive list above here:

https://docs.aws.amazon.com/

Let’s continue the questions.

Questions:

Knowing which user account was compromised is essential for understanding the attacker’s initial entry point into the environment. What is the username of the compromised user?

First, I started by finding the user-related fields because it asked me for a user. Then, I looked for any users I suspected and looked at the events they created.

Afterwards, there are some event types and commands that are important for security that I need to look at, and I am giving you a list of them. (The list I personally use)

Now I go and look at what logs came from these eventNames one by one.

(I am not adding the printouts of all of them here one by one)
While examining it, I noticed that each eventName produces an event only once. And these confirmed my suspicions. When I went to the relevant Event:

(I check the data both in the raw log and in the format provided by splunk.)

In the CreateLoginProfile logs, I saw that the user “helpdesk.luke” created a username called “marketing.mark”. And I also saw that attempts were made to change the authorization in the AttachUserPolicy events. These findings further confirmed my suspicions.
Again, in the AddUserToGroup events, here is the log of the suspicious user being added to a group called Admins.

Or if we look at it based on network (there are no questions asking for the suspicious attacker IP, but we will find it anyway)

PutObject, GetObject, GetBucketLocation events such as GetBucketPublicAccessBlock may contain public IP information,
For example, let’s look at the IPs for GetBucketPublicAccessBlock:
It is an AWS Identity and Access Management (IAM) API operation used to control public access in your AWS account.

I can also make reputation checks or whois queries for suspected sourceIPs from sites such as abuseipdb.

We have actually solved many questions until this part, but let’s see the questions anyway.

2. We must investigate the events following the initial compromise to understand the attacker’s motives. What’s the UTC timestamp of the attacker’s first access to an S3 object?

For this question, it was enough to look at the first log of GetObject eventName. There were 8 different events, but since it asked me for the first access date, I focused on this.

3- Among the S3 buckets accessed by the attacker, one contains a DWG file. What is the name of this bucket?

Since I had no idea what a DWG file was, all I had to do was leave the phrase DWG in search. I saw the bucket name in the relevant log here. But if I couldn’t find it directly, I would then check the ListObjects, PutObjects, DeleteObject and CopyObject logs (it already came from getobject) because:

PutObject: The event name PutObject specifies adding a file to a bucket. You can check this event name in case a new file is added.

DeleteObject: The event name DeleteObject specifies the deletion of a file from a bucket. If a DWG file is deleted, you can check this event name.

CopyObject: The event name CopyObject specifies copying a file from one bucket to another. File copying operations can also be examined.

4- We’ve identified changes to a bucket’s configuration that allowed public access, a significant security concern. What is the name of this particular S3 bucket?

5-Creating a new user account is a common tactic attackers use to establish persistence in a compromised environment. What is the username of the account created by the attacker?

Actually, we know the answer to this question, but I still took a look at the eventNames I related.