In this scenario, players investigate a Kubernetes breach by analyzing artifacts to track the attacker’s actions, focusing on tasks such as investigating compromised namespaces and detecting malicious behavior.
An attacker found a Flask application exposed outside the cluster through a NodePort service and fuzzed its /system/ path until one endpoint answered. That endpoint ran commands for them, and they used it to install curl inside the application pods and pipe a script from their own web server into a shell. One replica became the foothold. From there they created a separate pod of their own and left a root cron job on the node pointing at a backdoor script. The whole chain is readable from the application log, the per-pod process dumps and the node journal.
deployments.log, services.log. Output of kubectl describe, read with grep for cluster configurationgrepgrep -agrepTimestamp note: Everything is UTC and the fuzzing starts on 2024-11-08. The fuzzing answer wants the very first probe, not the burst that follows.
The deployment description records the desired replica count. Search it for Replicas and read the desired figure, which also tells you how many flask-app-* pod logs to expect.
grep Replicas describes/deployments.logThe service description for flask-app-service is of type NodePort. The port on the right of the mapping is the one reachable from outside the cluster, and it is the one the attacker hit.
grep NodePort describes/services.logGrep every replica’s flask-app.log for /system/ and sort by time. The earliest hit is a single GET /system/ls that returns 404, followed minutes later by a burst of same-second 404s as the wordlist ran.
grep "/system/" default/flask-app-*/flask-app.logGotcha: The burst of same-second 404s looks like the start of the fuzzing, but it is the tool running at speed. The first manual-looking probe a few minutes earlier is the answer.
Same log, filtered past the 404s. After the fuzzing run, one path under /system/ starts returning 200 to POST requests, which is the discovered endpoint being used rather than probed.
grep "/system/" default/flask-app-*/flask-app.logThe pod logs show apt installing a package through the execute endpoint, and the process dumps show what it was for: a sh -c that fetches from the attacker’s web server and pipes the result into bash. The program is the HTTP client.
grep -a "sh -c" default/processes/*The same sh -c process line from Q5 carries the target of the curl: the attacker’s host on port 8080, fed straight into bash.
grep -a "sh -c" default/processes/*Three replicas are running (Q1) and the process dumps are split per pod. Only one of them holds the curl | bash process from Q5 and Q6, so it is the pod the attacker actually landed in.
grep -a "sh -c" default/processes/*Outside the flask-app replicas there is an alpine pod whose container is named evil, and the node journal shows a useradd evil on the same name. The format hint (***l) is the container and account name, which matches the pod the attacker spun up from the foothold.
Search the node journals for backdoor. Cron logs a root CMD entry on schedule for a script under /opt, which is the persistence the attacker left on the node.
grep -rha backdoor system_logs/*.log| Phase | Action |
|---|---|
| Discovery | Attacker reaches the Flask app on the NodePort and fuzzes /system/ |
| Initial Access | Fuzzing finds /system/execute, which accepts POST and returns 200 |
| Execution | Commands through the endpoint install curl in the pods |
| Command and Control | curl from the attacker’s web server piped into bash in one replica |
| Persistence | Malicious alpine pod created with a container named evil |
| Persistence | Root cron job runs /opt/backdoor.sh |
| Type | Value |
|---|---|
| IP | 10.129.231[.]112 |
| Endpoint | /system/execute |
| Pod (foothold) | flask-app-77fbdcfcff-2tqgw |
| Pod (malicious) | evil |
| File | /opt/backdoor.sh |
| Technique | ID | Description |
|---|---|---|
| Active Scanning: Wordlist Scanning | T1595.003 | Fuzzing of /system/ paths |
| Exploit Public-Facing Application | T1190 | Command-execution endpoint on the NodePort service |
| Command and Scripting Interpreter: Unix Shell | T1059.004 | sh -c commands through the endpoint |
| Ingress Tool Transfer | T1105 | curl pull of a script from the attacker’s server |
| Deploy Container | T1610 | Rogue alpine pod with container evil |
| Scheduled Task/Job: Cron | T1053.003 | Root cron entry for /opt/backdoor.sh |
A debug-style execute endpoint should never ship. The app exposed a path that ran commands on request, reachable from outside the cluster through a NodePort. Remove it, or put authentication in front of anything that can run code.
Fuzzing is loud and the log shows it. A run of 404s on one prefix, ending in a first 200 on a new path, is a clean detection. Alert on bursts of 404s per source against a single service, and keep per-pod access logs.
Package installs inside running pods are a signal. apt pulling curl into an application container, followed by curl | bash, is not normal pod behaviour. Use read-only root filesystems and egress policies so a pod cannot fetch from an arbitrary internal host.
Watch for pods that don’t belong to a deployment. A bare alpine pod alongside the application replicas, plus cron entries under /opt on the node, should be caught by admission control and file integrity checks.
One unauthenticated command-execution endpoint turned a public NodePort into a root cron backdoor on the node. The detection value sat in the first few probes of the fuzzing run, well before the exploit landed.