The Self-Expanding Stolen Inference Supply Chain Leveraging AI Agents

In the rapidly evolving landscape of cybersecurity threats, a new tactic has emerged that underscores the sophistication with which attackers are leveraging technology to their advantage. This technique involves attackers using semi-autonomous coding agents to exploit poorly secured LLM resale gateways. Through common web vulnerabilities and account farming techniques, these attackers acquire unauthorized API access. The aggregated AI inference capacity is then resold, presenting a sophisticated supply chain threat.

Campaign or TTP Overview

This tactic has started to surface in late 2023, targeting AI service providers and their resale gateways. These platforms often offer API access to large language models (LLMs) for third-party developers and companies. However, poorly secured instances have become the target of attackers who see an opportunity to exploit these vulnerabilities for profit.

According to recent threat intelligence reports, these semi-autonomous coding agents have been observed orchestrating complex chains of exploits, focusing on extracting as much AI inference capacity as possible. The harvested resources are then bundled and sold on underground markets, providing a scalable income stream for cybercriminals. This threat has not been attributed to a specific group, but the scheme’s complexity suggests involvement from actors with significant technical skill.

How It Was Built

The attack chain begins with the identification of vulnerable AI inference gateways, typically using openly available scanning tools to locate exposed endpoints. Once a target is identified, the attackers employ a mix of SQL injection and cross-site scripting attacks to breach the system’s defenses.


POST /api/v1/access HTTP/1.1

Host: vuln-llm-reseller.com

Content-Type: application/json

Content-Length: 150



{

    "username": "injected' OR '1'='1",

    "password": "password"

}

After breaching the initial defenses, the attackers use automated agents to perform account farming. This involves creating a multitude of fake developer accounts to siphon off API calls, effectively expanding their control over the target’s inference capabilities without triggering immediate suspicions of abnormal access patterns.

The final phase of the operation involves collating the stolen inference capacity through these farmed accounts and listing them on dark web marketplaces. Here they are sold to buyers looking for cheaper alternatives to legitimate API access, highlighting the economic incentive behind these attacks.

Why It Worked

Several factors contributed to the effectiveness of this campaign. First, the attack capitalized on common vulnerabilities in API security. Many AI service providers might prioritize development speed over secure coding practices, leaving them susceptible to injection attacks. Furthermore, the use of semi-autonomous agents allows attackers to rapidly replicate their efforts across multiple targets, scaling up the operation with minimal manual intervention.

The attack also cleverly exploited the economic model of LLM resale platforms, where API access is seen as a valuable commodity. By farming accounts and simulating legitimate usage patterns, attackers evade immediate detection. This approach allows them to aggregate a significant portion of a target’s inference capacity without raising red flags until substantial damage has already occurred.

Operator Takeaways

For red team operators, this campaign provides several insights into how semi-autonomous mechanisms can be employed effectively. You can consider implementing similar automated strategies to test your organization’s ability to detect unauthorized access attempts and resilience to injection attacks. Moreover, the campaign exemplifies the power of leveraging resource aggregation techniques to simulate impact assessment more realistically in penetration tests.

Do’s and Don’ts

Do: Utilize automation to scale your engagement efforts, simulating both the access and usage patterns similar to those seen in this attack.
Don’t: Overlook the significance of seemingly small access vulnerabilities. Such gaps can be exploited massively over time.
Do: Experiment with resource aggregation tactics to understand their role in potential attack scenarios and client impact foundations.
Don’t: Ignore the economic motivations behind LLM exploitation — understanding these can help tailor your engagements more realistically.

References

SANS Internet Storm Center