Fail2Shell

Scenario

Following initial network compromise, the attacker performed lateral movement and obtained credentials for a Linux server. After authenticating, they exfiltrated sensitive data before establishing a persistence mechanism. The IR team remotely collected triage data for analysis. The task is to reconstruct the attacker’s actions and identify the impact from a Linux memory image.

Methodology

Environment Setup — Volatility via WSL

The Windows VM didn’t have Volatility installed natively, but had WSL with an Ubuntu instance available. Volatility 3 was present at ~/tools/volatility3/ within the WSL environment. The memory image at C:\Users\BTLOTest\Desktop\Artefacts\Fail2Shell\mem.mem is accessible from WSL at /mnt/c/Users/BTLOTest/Desktop/Artefacts/Fail2Shell/mem.mem.

export MEM="/mnt/c/Users/BTLOTest/Desktop/Artefacts/Fail2Shell/mem.mem"
cd ~/tools/volatility3

Reconnaissance — Process Analysis

Running linux.pstree and linux.pslist immediately surfaced a suspicious process:

python3 vol.py -f $MEM linux.pstree
python3 vol.py -f $MEM linux.pslist

Two processes named shared — PID 1492 (parent) and PID 1493 (child) — running directly under PID 1 (init). Legitimate processes rarely spawn from init with generic single-word names. The binary path retrieved via linux.psaux confirmed:

1492  1     shared  /home/ubuntu/.local/shared
1493  1492  shared  /home/ubuntu/.local/shared

This is the attacker’s persistence and C2 agent. The creation timestamp from pslist placed it at 2025-07-01 09:15:57 UTC.

Initial Access — Attacker Source IP

PSReadLine and bash history via the linux.bash plugin returned only the IR team’s own capture commands (downloading and running avml). Auth events were retrieved via strings instead:

strings $MEM | grep "Accepted password\|Accepted publickey"

Three successful authentication events from 192.168.1.100 — for users ubuntu, ahmed, and allam. This is the attacker’s lateral movement source IP.

Network — C2 Communication

linux.sockstat generated substantial output and was redirected to file for analysis:

python3 vol.py -f $MEM linux.sockstat > socket.txt
grep -E "TCP|UDP" socket.txt | grep "ESTABLISHED" | grep -v "127.0.0.1\|AF_UNIX"

The shared process (PID 1493) maintained an established TCP connection to 162.159.133.234:43. Port 43 is conventionally WHOIS — an uncommon but effective port for blending C2 traffic into allowlisted egress. The destination resolves to a Cloudflare IP, indicating domain fronting.

Persistence — PAM Backdoor

A broad strings search confirmed the persistence mechanism:

strings $MEM | grep -E "authorized_keys|ssh-rsa|pam_exec|good\.zip" | sort -u

The output revealed pam_exec.so quiet /home/ubuntu/.local/shared — a one-line PAM configuration that executes the shared binary silently on every authentication event. This is a classic PAM backdoor technique: by adding pam_exec to a PAM stack file like /etc/pam.d/common-auth, the attacker ensures their C2 agent fires every time any user authenticates via SSH, regardless of which account is used.

The inode of the modified PAM config was retrieved using the linux.pagecache.Files plugin:

python3 vol.py -f $MEM linux.pagecache.Files > pagecache.txt
grep -iE "pam\.d" pagecache.txt

/etc/pam.d/common-auth showed a modification timestamp of 2025-07-01 05:38:14 UTC — the only PAM config file with an attack-day mtime — confirming it as the injection target. Its inode number is 6293320.

C2 Application — Discord Bot

A strings search targeting common C2 frameworks and messaging platforms returned unexpected results:

strings $MEM | grep -iE "discord|telegram|bot" | head -30

The output revealed /tmp/_MEIcIYfzp/discord/ — a PyInstaller extraction directory containing a full Discord bot bundle including discord.py, urllib3, requests, and aiohttp. PyInstaller packages Python applications as self-extracting binaries; on first execution, they expand their payload into a /tmp/_MEI<random>/ directory.

The shared binary is itself the PyInstaller bundle — a single executable acting as both C2 implant and Discord command-delivery bot. Environment variables extracted via linux.envars confirmed this:

_PYI_ARCHIVE_FILE             /home/ubuntu/.local/shared
_PYI_APPLICATION_HOME_DIR     /tmp/_MEIcIYfzp

Discord Bot Identification — Snowflake Decoding

Confirming the C2 platform was Discord raised a follow-up question: when was the bot account itself created? Discord accounts use snowflake IDs — 64-bit integers where the upper 42 bits encode the creation timestamp in milliseconds since the Discord epoch (2015-01-01 00:00:00 UTC). Decoding any Discord ID yields its account creation moment.

A targeted strings search isolated the Discord URL infrastructure first to confirm the API was actually being used in memory:

strings $MEM | grep -E "discord\.com|discordapp\.com|gateway\.discord"

gateway.discord.gg, discord.com/api/v10, and discord.com/channels/ all confirmed live API traffic. Next, the actual JSON message payloads were carved out by grepping for the bot’s identifier flag:

strings $MEM | grep -E '"bot":true' | head -10

Embedded in the gateway message JSON, the bot’s user object appeared verbatim:

"user":{
  "username":"Notion",
  "id":"1280260947289706610",
  "discriminator":"2698",
  "bot":true
}

The bot was named “Notion” — a deliberate masquerade as the popular productivity app to look innocuous in a Discord member list. The snowflake ID 1280260947289706610 was decoded:

import datetime
DISCORD_EPOCH = 1420070400000
sid = 1280260947289706610
ts_ms = (sid >> 22) + DISCORD_EPOCH
print(datetime.datetime.fromtimestamp(ts_ms/1000, tz=datetime.timezone.utc))
# 2024-09-02 20:19:55.932000+00:00

The bot account was created on 2024-09-02 20:19:55 UTC — almost 10 months before deployment, sitting dormant until used in this attack on July 1 2025.

The attacker’s own Discord user m4shl3 (ID 844625487879471104) decoded to 2021-05-19 17:20:08 UTC — a long-standing personal account, useful pivot data for attribution.

Command Recovery — Discord Message Carving

The full bot command history was recoverable by carving the !cmd prefix from message JSON in memory:

strings $MEM | grep -oE '"content":"![a-z]+ [^"]+"' | sort -u

Three commands surfaced:

"content":"!cmd ls -lha /home/allam/"
"content":"!cmd rm -r /var/log"
"content":"!cmd rm /root/.bash_history"

The most recent command was determined by sorting the associated message snowflake IDs (newer ID = newer message). The final command issued was ls -lha /home/allam/ — the attacker’s last action was likely confirming the cleanup before disconnecting, not the cleanup itself.

Attack Timeline — Pagecache Analysis

The linux.pagecache.Files plugin returns filesystem-level timestamps (atime/mtime/ctime) for all cached inodes. This provided the authoritative UTC timeline for the attack:

python3 vol.py -f $MEM linux.pagecache.Files > pagecache.txt

The shared binary’s inode entry:

-rwxrwxr-x  atime: 2025-07-01 05:40:00 UTC
            mtime: 2025-07-01 05:38:10 UTC
            ctime: 2025-07-01 05:39:21 UTC
            /home/ubuntu/.local/shared

Three timestamps, three events:

• mtime 05:38:10 — binary written to disk via download
• ctime 05:39:21 — chmod +x applied, making it executable
• atime 05:40:00 — first execution, pam_exec fired shared for the first time

The atime of 05:40:00 UTC represents the first time the persistence mechanism was validated — the moment pam_exec successfully invoked the shared binary following a system authentication event.

Anti-Forensics — Log Deletion

The !cmd rm -r /var/log message was issued at 2025-07-01T09:17:08 UTC per its Discord message timestamp. The deletion of /var/log was confirmed by lsof.txt output showing auth.log, syslog, and kern.log all marked (deleted) with last-seen timestamps around 09:17 UTC. This is a standard anti-forensic move to eliminate SSH auth logs and syslog entries.

Attack Summary

IOCs

MITRE ATT&CK

Defender Takeaways

PAM configuration integrity monitoring — The attacker modified /etc/pam.d/common-auth to add a pam_exec entry executing an arbitrary binary on every authentication. File integrity monitoring on /etc/pam.d/ with alerting on any modification would catch this within seconds of the change. Tools like AIDE or Wazuh can enforce this at minimal cost.

Outbound egress filtering — The shared implant connected to 162.159.133.234:43 — an unusual port for legitimate traffic. Strict egress policies that restrict outbound connections to approved ports and destinations would have severed the C2 channel. Port 43 (WHOIS) has no legitimate use case for a Linux server and should be blocked by default. Discord’s gateway endpoints (gateway.discord.gg, discord.com/api/v*) should also be on a corporate egress denylist for production servers — there is no operational reason for a Linux server to talk to Discord.

Process anomaly detection — A process named shared running directly under PID 1 with no legitimate parent chain is a strong indicator of compromise. EDR or auditd rules alerting on processes spawning from init with non-standard binary paths would surface this immediately. PyInstaller extraction directories (/tmp/_MEI*) are another high-signal indicator — legitimate use is rare on production servers.

Discord snowflake decoding for attribution — Discord IDs are timestamps. Any account, message, channel, or guild ID recovered from memory or network captures decodes directly to a UTC timestamp via (snowflake >> 22) + 1420070400000. This is invaluable for incident timeline reconstruction and identifying long-dormant attacker infrastructure (in this case, a bot provisioned 10 months before deployment). Tools like snowsta.mp or a one-line Python helper make this trivial for analysts to apply during triage.

Volatile data preservation — Critical evidence including the PAM configuration timestamps, process creation times, and Discord bot memory contents was only recoverable via memory forensics. A volatile data collection workflow (memory image + lsof + netstat snapshot) should be standard procedure when a Linux host is suspected of compromise — before any remediation steps that would destroy this evidence.

Bash history hardening — The attacker deleted bash histories as a final cleanup step. Configuring HISTFILE=/dev/null detection or shipping bash history to a remote SIEM in real time prevents this anti-forensic technique from being effective. Centralised log shipping to an append-only store would also have preserved the /var/log contents deleted by rm -r /var/log.