PercyPay

Scenario

In this Sherlock challenge, participants are tasked with investigating a potential data breach at PercyPay, an innovative payment provider. Participants must analyze the provided artifacts to determine the extent of the breach, identify the threat actor’s actions, and recommend appropriate response measures. The challenge is designed to test skills in threat intelligence, log analysis, and incident response within a cloud environment.

Executive Summary

A developer uploaded a file to the public website bucket that carried the long-term access key of a web application IAM user in an HTML comment. The threat actor picked the key up, confirmed it with an identity check and read the user’s policies. They used a policy-version switch on a Lambda-related policy to grant themselves rights, then created a Lambda function with a more privileged role and used it to attach the managed administrator policy to the stolen user. With admin rights they deleted the GuardDuty detector, looked at the CloudTrail log group, listed the S3 buckets and downloaded one customer’s payment data in bulk. They tried to set up cross-bucket replication to an outside bucket, which failed, then created a second admin IAM user with its own access key from a different IP and stopped CloudTrail logging.

Artifacts

• CloudTrail: gzip JSON log files from the first evidence zip → flattened into one JSON list and queried with Python
• Website host export: HTML files from the percypay-website-host bucket in the second evidence zip → read with grep

Timestamp note: CloudTrail eventTime values are UTC. Answers that ask for a time use YYYY-MM-DD hh:mm:ss, so copy the field and drop the T and Z.

The zips are encrypted with the standard HTB archive password.

unzip -P hacktheblue -d work/ct <cloudtrail zip>

unzip -P hacktheblue -d work/web <website zip>

Flatten every log file into one list so each question becomes a filter. All queries below run over that list.

python3 -I -c "import gzip,glob,json;r=[];[r.extend(json.load(gzip.open(f))['Records']) for f in glob.glob('work/ct/**/*.json.gz',recursive=True)];json.dump(sorted(r,key=lambda e:e['eventTime']),open('work/all.json','w'))"

Credential Access

Start before the compromise window. The website bucket is the only place a long-term key could leak, so list write events against percypay-website-host ahead of the first event from the stolen key. A developer account uploads a file there minutes earlier, and that upload is the odd one out: nothing else in the bucket’s history looks like a key file being dropped in.

python3 -I -c "import json;[print(e['eventTime'],e['userIdentity'].get('userName'),e['eventName'],e.get('requestParameters')) for e in json.load(open('work/all.json')) if e['eventSource']=='s3.amazonaws.com' and 'percypay-website-host' in str(e.get('requestParameters')) and e['eventName']!='GetObject']"

Gotcha: The format mask is nine characters. Anonymous website reads and a plain GetObject look anomalous but do not fit it, and neither is the event that preceded the compromise. The upload does.

The resources array of that PutObject event carries the object ARN. The uploader is developer-dan, and the event lands at 20:37:21, five minutes before the stolen key is first used.

python3 -I -c "import json;[print(e['eventTime'],e['userIdentity'].get('userName'),e['resources']) for e in json.load(open('work/all.json')) if e['eventName']=='PutObject' and 'percypay-website-host' in str(e.get('requestParameters'))]"

From Q36 the uploaded file name already says what it holds. Open it in the website export and the key ID sits in an HTML comment. Confirm it is the live one by checking accessKeyId on the threat actor’s CloudTrail events: the same ID authenticates every call in the intrusion.

grep -rn 'AKIA' work/web/

The secret is in the same HTML comment as the key ID, which is how the threat actor got a working credential pair without any exploitation.

grep -n -B2 -A2 'AKIA' work/web/backend-access-key.html

Filter CloudTrail on the leaked key from Q3 and read userIdentity.arn. Every threat actor event resolves to the same IAM user, the web application’s service user.

python3 -I -c "import json;print({e['userIdentity']['arn'] for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY'})"

Tip: The mask is a full ARN, not a username. Take userIdentity.arn as is.

Discovery

Sort the events for the Q3 key by time and take the first. The threat actor’s first move is an identity check, so this is also the start of the intrusion.

python3 -I -c "import json;ev=[e for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY'];print(ev[0]['eventTime'],ev[0]['eventName'])"

The event at the top of the Q8 list. An STS call that returns the caller’s identity needs no permissions, which is why attackers run it first with a fresh key.

python3 -I -c "import json;ev=[e for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY'];print(ev[0]['eventSource'],ev[0]['eventName'])"

Count sourceIPAddress across the Q3 key’s events. One address accounts for 196 of 216 events, including all 164 GetObject calls, so it is the main operator IP.

python3 -I -c "import json,collections;print(collections.Counter(e['sourceIPAddress'] for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY'))"

The same counter shows the full set: the main IP from Q4, a second address next to it, and 149.40.62.16. The count of distinct values is the answer.

python3 -I -c "import json;print(len({e['sourceIPAddress'] for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY'}))"

After the identity check the actor reads the user’s permissions. The first ListPolicyVersions and GetPolicyVersion calls name a customer-managed policy tied to Lambda exports.

python3 -I -c "import json;[print(e['eventTime'],e['eventName'],e.get('requestParameters')) for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY' and e['eventName'] in ('ListPolicyVersions','GetPolicyVersion')][:4]"

Privilege Escalation

A managed policy can hold several versions with only one active. The actor did not create a policy, they flipped the default version on a second policy at 20:45:42, switching the user to a more permissive version that already existed.

python3 -I -c "import json;[print(e['eventTime'],e['eventName'],e['requestParameters']) for e in json.load(open('work/all.json')) if e['eventName'] in ('SetDefaultPolicyVersion','CreatePolicyVersion')]"

The requestParameters of the Q12 event name the policy whose default version changed.

python3 -I -c "import json;[print(e['eventTime'],e['requestParameters']) for e in json.load(open('work/all.json')) if e['eventName']=='SetDefaultPolicyVersion']"

With the Lambda rights active, the next create call is a CreateFunction. The ARN is in the responseElements.

python3 -I -c "import json;[print(e['eventTime'],e['responseElements'].get('functionArn')) for e in json.load(open('work/all.json')) if e['eventName']=='CreateFunction20150331']"

The userIdentity of the CreateFunction event is the compromised web application user from Q2, now acting with the new Lambda policy.

python3 -I -c "import json;[print(e['userIdentity']['userName']) for e in json.load(open('work/all.json')) if e['eventName']=='CreateFunction20150331']"

The function runs under a role called lambda-commander, not the creating user’s identity. Handing a role to a service at create time requires the iam:PassRole permission. This one was reasoned from the role field in the CreateFunction request and the format mask rather than read from a dedicated log entry, because CloudTrail does not record a separate PassRole event.

python3 -I -c "import json;[print(e['requestParameters'].get('role')) for e in json.load(open('work/all.json')) if e['eventName']=='CreateFunction20150331']"

Events with the lambda-commander role in userIdentity are the function acting. At 20:48:05 it calls IAM and attaches a managed policy to a user, which is how the stolen user received more rights than any policy edit had given it.

python3 -I -c "import json;[print(e['eventTime'],e['eventName'],e['userIdentity'].get('arn')) for e in json.load(open('work/all.json')) if 'lambda-commander' in str(e['userIdentity'])]"

The AttachUserPolicy request carries both the policyArn and userName. The AWS-managed administrator policy goes onto the web application user.

python3 -I -c "import json;[print(e['requestParameters']) for e in json.load(open('work/all.json')) if e['eventName']=='AttachUserPolicy']"

Tip: The mask asks for the policy ARN only, not the user name.

Defense Evasion

After the admin attach, the next event source in the stolen key’s timeline is the threat detection service. The actor calls ListDetectors to find what is watching the account.

python3 -I -c "import json;[print(e['eventTime'],e['eventSource'],e['eventName']) for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY' and e['eventTime']>'2024-12-15T20:48']"

Listing the detectors is followed by DeleteDetector. The ID is in its requestParameters. Deleting the detector removes GuardDuty findings for everything that follows.

python3 -I -c "import json;[print(e['eventTime'],e['requestParameters']) for e in json.load(open('work/all.json')) if e['eventName']=='DeleteDetector']"

The next event source after GuardDuty is CloudWatch Logs, where the actor runs DescribeLogGroups to find where CloudTrail is being written.

python3 -I -c "import json;[print(e['eventTime'],e['eventSource'],e['eventName']) for e in json.load(open('work/all.json')) if e['eventName']=='DescribeLogGroups']"

The DescribeLogGroups response lists the log group that receives CloudTrail. Its ARN is in the event’s resources.

python3 -I -c "import json;[print(e['resources']) for e in json.load(open('work/all.json')) if e['eventName']=='DescribeLogGroups']"

Collection

The first S3 event for the stolen key is a bucket listing, the account-wide view of what can be taken.

python3 -I -c "import json;[print(e['eventTime'],e['eventName']) for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY' and e['eventSource']=='s3.amazonaws.com'][:3]"

The bulk GetObject run starts at 20:51:40. The last call before it is the key listing of the target bucket. This was read from the sequence of events just before the burst and the mask length, not from a labelled marker in the log.

python3 -I -c "import json;ev=[e for e in json.load(open('work/all.json')) if e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY'];[print(e['eventTime'],e['eventName']) for e in ev if '20:50'<e['eventTime'][11:16]<'20:52'][:12]"

The bucketName in the ListObjects and GetObject requests from Q23 and Q24 is the payments bucket.

python3 -I -c "import json,collections;print(collections.Counter(e['requestParameters']['bucketName'] for e in json.load(open('work/all.json')) if e['eventName'] in ('GetObject','ListObjects') and e['userIdentity'].get('accessKeyId')=='AKIA4PLMARDZH6ZV23ZY'))"

Count distinct object keys in the GetObject events against the payments bucket, not the event count, so repeat reads of one file do not inflate the number.

python3 -I -c "import json;print(len({e['requestParameters']['key'] for e in json.load(open('work/all.json')) if e['eventName']=='GetObject' and e['requestParameters'].get('bucketName')=='percypay-payment-data'}))"

Every key read is under one customer prefix, in the form <Customer>/<date>/<file>.

python3 -I -c "import json;print({e['requestParameters']['key'].rsplit('/',1)[0] for e in json.load(open('work/all.json')) if e['eventName']=='GetObject' and e['requestParameters'].get('bucketName')=='percypay-payment-data'})"

The date segment of the same key prefix from Q6 gives the day the stolen payment files cover. It is a single value across all 164 keys.

Exfiltration

Look for CreateRole after the GetObject burst. The response returns the new role’s ARN.

python3 -I -c "import json;[print(e['eventTime'],e['responseElements']['role']['arn']) for e in json.load(open('work/all.json')) if e['eventName']=='CreateRole']"

An inline policy is attached with PutRolePolicy, so the name is in its request.

python3 -I -c "import json;[print(e['requestParameters']['policyName']) for e in json.load(open('work/all.json')) if e['eventName']=='PutRolePolicy']"

The policy document in the same PutRolePolicy event lists the resource for s3:ReplicateObject: every object in a bucket the account does not own. The name reads like a transactions bucket the actor controls.

python3 -I -c "import json;[print(e['requestParameters']['policyDocument']) for e in json.load(open('work/all.json')) if e['eventName']=='PutRolePolicy']"

The replication setup to the outside bucket fails. The errorMessage field on the PutBucketReplication event holds the text, so the second exfiltration path did not work.

python3 -I -c "import json;[print(e['eventTime'],e.get('errorCode'),e.get('errorMessage')) for e in json.load(open('work/all.json')) if e['eventName']=='PutBucketReplication']"

Persistence

A CreateUser event follows the failed replication.

python3 -I -c "import json;[print(e['eventTime'],e['requestParameters']['userName']) for e in json.load(open('work/all.json')) if e['eventName']=='CreateUser']"

Filter AttachUserPolicy to the new user from Q29 and take its eventTime.

python3 -I -c "import json;[print(e['eventTime'],e['requestParameters']) for e in json.load(open('work/all.json')) if e['eventName']=='AttachUserPolicy' and e['requestParameters']['userName']=='tfe-user']"

CreateAccessKey for the new user returns the ID in its response.

python3 -I -c "import json;[print(e['responseElements']['accessKey']['accessKeyId']) for e in json.load(open('work/all.json')) if e['eventName']=='CreateAccessKey']"

Pivot on userIdentity.userName for the new user and sort. The first event is the earliest use of the backdoor key, about a minute after the admin policy went on.

python3 -I -c "import json;ev=[e for e in json.load(open('work/all.json')) if e['userIdentity'].get('userName')=='tfe-user'];print(ev[0]['eventTime'],ev[0]['eventName'])"

The backdoor user’s events come from a different address than the three used with the stolen key, a clean separation to hand to the response team.

python3 -I -c "import json;print({e['sourceIPAddress'] for e in json.load(open('work/all.json')) if e['userIdentity'].get('userName')=='tfe-user'})"

Defense Evasion

Search for StopLogging. The event source names the service whose trail the actor turned off after the backdoor was in place.

python3 -I -c "import json;[print(e['eventTime'],e['eventSource'],e['eventName']) for e in json.load(open('work/all.json')) if e['eventName']=='StopLogging']"

Attack Summary

Phase	Action
Credential Access	developer-dan uploads a page to the public website bucket with the web app user’s access key and secret in an HTML comment
Discovery	Stolen key used for GetCallerIdentity, then IAM policy reads
Privilege Escalation	SetDefaultPolicyVersion on lambda-export-invoke grants Lambda rights
Privilege Escalation	web-processor-function created with the lambda-commander role via PassRole
Privilege Escalation	The function attaches AdministratorAccess to the web app user
Defense Evasion	GuardDuty detector deleted; CloudWatch log groups enumerated
Collection	ListBuckets, ListObjects, then 164 GetObject reads of one customer’s payment files
Exfiltration	replicator-role and replicator-policy created; PutBucketReplication to an outside bucket fails
Persistence	tfe-user created, given AdministratorAccess and an access key, used from a new IP
Defense Evasion	CloudTrail logging stopped

Forensic Timeline

Timestamp (UTC)	Event
20:37:21 (date not recorded in the notes)	developer-dan uploads the key file to percypay-website-host
2024-12-15 20:42:16	First use of the stolen key: GetCallerIdentity
2024-12-15 20:45:42	SetDefaultPolicyVersion on lambda-export-invoke
2024-12-15 20:48:05	lambda-commander role attaches AdministratorAccess
2024-12-15 20:51:40	Bulk GetObject of CustomerB payment data begins
2024-12-15 20:53:26	AdministratorAccess attached to tfe-user
2024-12-15 20:54:42	tfe-user first used

IOCs

Type	Value
IP (stolen key, main)	149.102.244[.]70
IP (stolen key, secondary)	149.40.62[.]16
IP (backdoor user)	89.187.177[.]75
Access key (compromised)	AKIA4PLMARDZH6ZV23ZY (percypay-webapp-user)
Access key (backdoor)	AKIA4PLMARDZOYCZJ3NS (tfe-user)
IAM user	tfe-user
IAM role	arn:aws:iam::857609373938:role/replicator-role
Lambda function	arn:aws:lambda:eu-west-1:857609373938:function:web-processor-function
S3 bucket (attacker)	apt-we-love-transactions
File	backend-access-key.html in percypay-website-host

MITRE ATT&CK

Technique	ID	Description
Unsecured Credentials: Credentials In Files	T1552.001	Access key and secret left in an HTML comment on the public website bucket
Valid Accounts: Cloud Accounts	T1078.004	Leaked key used to authenticate as the web app user
Account Discovery: Cloud Account	T1087.004	GetCallerIdentity and IAM reads
Permission Groups Discovery: Cloud Groups	T1069.003	Policy version listing and review
Account Manipulation	T1098	Default policy version switched; AdministratorAccess attached
Cloud Infrastructure Discovery	T1580	ListDetectors, DescribeLogGroups, ListBuckets
Impair Defenses: Disable or Modify Cloud Logs	T1562.008	CloudTrail StopLogging
Impair Defenses: Disable or Modify Tools	T1562.001	GuardDuty detector deleted
Cloud Storage Object Discovery	T1619	ListObjects on the payment bucket
Data from Cloud Storage	T1530	164 GetObject reads of CustomerB files
Transfer Data to Cloud Account	T1537	Replication role and PutBucketReplication toward an outside bucket
Create Account: Cloud Account	T1136.003	tfe-user created
Account Manipulation: Additional Cloud Credentials	T1098.001	Access key created for tfe-user

Defender Takeaways

Secrets in web content are a breach waiting for a reader. A static site bucket should never carry a credential, even in a comment. Scan uploads for key patterns such as AKIA before publishing, and alert on any PutObject to the website bucket from a developer identity that carries a secret-shaped string.

A policy version switch is a privilege escalation, not routine IAM churn. SetDefaultPolicyVersion on a policy attached to a service user at 20:45 was followed within minutes by a function creation and an admin attach. Alert on that call outside change windows and restrict iam:SetDefaultPolicyVersion to a small admin role.

iam:PassRole plus lambda:CreateFunction is an admin path. A web application user that can create functions and hand them a privileged role can run anything as that role. Scope PassRole to named, low-privilege roles with a condition on the target service, and alert on AttachUserPolicy calls made by a role session.

Security tooling must be protected from the account it watches. The actor deleted the GuardDuty detector, looked up the CloudTrail log group and later stopped logging, all with the same stolen credential. Deny guardduty:DeleteDetector and cloudtrail:StopLogging with a service control policy, and ship logs to a separate account so a stopped trail still leaves a record.

Bulk reads and unknown IAM users are detectable. 164 GetObject calls on one prefix in a short window, a new IAM user with AdministratorAccess and a first-use IP that differs from the owner’s pattern are all CloudTrail signals. Revoke both keys from the IOC table, delete tfe-user, remove the leaked HTML file and rotate every secret it ever held.

Conclusion

One access key in an HTML comment gave the actor a full chain: policy abuse, a PassRole Lambda to reach administrator rights, CustomerB’s payment files and a second admin user that survives revoking the first key. Each step was visible in CloudTrail until the actor stopped it, which is why logging has to be protected as hard as the data.