The Koi Incident: A Case Study
A post-mortem on how a cybersecurity firm's AI fabricated an entire cyber-espionage conspiracy — and why nobody caught it until it was too late.
The Context
In December 2025, Koi AI — in the final stages of a lucrative acquisition by Palo Alto Networks — was under intense financial pressure to demonstrate the "explosive" discovery capabilities of their AI platform. This incentivized the team to prioritize a sensationalized criminal narrative over basic human validation.
Anatomy of a Catastrophe
Walk through the four phases that turned an AI hallucination into a $72 million crisis.
A Pressured Investigation
- 1
December 2025: Cybersecurity firm Koi AI was finalizing a multi-million-dollar acquisition by Palo Alto Networks.
- 2
Koi tasked its AI-driven 'Wings' platform with finding links between global malware campaigns.
- 3
The team sought to connect the 'ShadyPanda' operation to a newly discovered 'Zoom Stealer' campaign.
- 4
Goal: attribute both campaigns to a single Chinese threat actor called 'DarkSpectre.'
- 5
Intense financial pressure incentivized sensationalized findings over basic validation.
The Fabrication vs. Reality
Compare the AI's fabricated "intelligence report" with what was actually verified.
"One extension bridged the gap: Twitter X Video Downloader. This extension communicated with infinitynewtab.com — core ShadyPanda infrastructure. But when we analyzed its behavior, we found something unexpected: it wasn't just running data exfiltration and user surveillance. It was harvesting meeting intelligence from 28+ video conferencing platforms."
No extension named "Twitter X Video Downloader" existed in the researchers' dataset of extension IDs.
The closest match — "Twiter X Video Downloader" (one 't') — had absolutely no connection to infinitynewtab.com, contained no network calls to it, and possessed no mechanism to communicate with it.
MeetingTV's actual software did not steal meeting data — it simply read URLs to offer legitimate, explicitly disclosed webinar recommendations.
The AI's output was dangerously convincing because it perfectly mirrored the format, tone, and complexity of a real cybersecurity threat intelligence report.
- •The AI wove hallucinations into a 32-page document filled with real technical artifacts
- •It correctly identified real software functions but misinterpreted them as malicious
- •Because the output aligned with the team's desired narrative, automation bias took over
- •The researchers trusted the confident output without any independent verification
5-Step AI Diligence Checklist
For professionals in high-stakes industries: never act on an AI suggestion before completing these mandatory verification steps.
Verify the Raw Data Manually
Before trusting an AI's correlation, physically verify that the underlying data exists. A basic manual check would have revealed the pivot extension didn't exist.
Search for 'no-results' — if a citation only exists in your AI chat, it's likely a hallucination.
Review Context & Public Disclosures
AI systems lack real-world common sense and may misinterpret legal activities as malicious. Reading public descriptions would have shown transparent disclosure.
Run a 'counter-prompt': Ask the AI for the strongest argument AGAINST its own findings.
Rule Out Alternative Explanations
Actively brainstorm non-malicious reasons for the AI's findings. Consider domain spoofing, infrastructure hijacking, or legitimate references by malware authors.
Paste the same prompt into a different AI model. If facts differ between models, the data is unverified.
Contact the Subject Before Action
A simple human conversation can prevent catastrophic errors. Despite accusing a company of espionage, Koi never contacted them despite public contact information.
Apply the 'Life & Livelihood' rule — if it impacts someone's health, legal standing, or finances, a human must review 100%.
Assess the Downstream Blast Radius
Understand what automated systems are connected to your outputs. In cybersecurity, IOC designations cascade through global networks instantly.
Log your AI usage — keep records of prompts and outputs for accountability and learning.
Diligence Is the Only Safety Net
Don't let a chatbot make your life's biggest decisions. Learn how to spot the hallucination before it's too late.
Get a weekly newsletter on the biggest AI hallucination issues affecting society
Real, documented incidents — legal cases, medical errors, financial fallout, and more — delivered to your inbox every week. No spam, unsubscribe anytime.
