Automated adversaries: Preparing for the age of offensive AI

Offensive Cyber Agents

When this technology is fully applied to offensive cybersecurity we’re looking at autonomous agents that will perform compromises thousands of times faster than humans. Remember when you had that Ivanti or Fortinet zero day that took several hours to patch? That’s too slow.

Research falls broadly into two buckets: foundational research that seeks to advance fundamental understanding without immediate practical application, and applied research that builds on those foundations to solve real-world problems. Foundational research and technology often stop-starts in unpredictable peaks and valleys, but applied research is an incremental march of consistent refinement. Why I think fully automated compromise may be close is because I believe the foundational technologies for full-stack offensive cyber tooling already exist, and just need applied research to get us there.

A large part of human-operated hacking takes place using command line tools. This depends slightly on the user’s preferences around tooling and the type of system being targeted, but for most non-web attacks command line tools are the go to for offensive cyber. LLMs have shown their ability to excel at written language and this is amongst the most impressive achievements of current neural networks. Crucially, this specific type of interaction (issuing structured, text-based commands and interpreting system output) is similar to programming, a domain where LLMs have shown surprising fluency.

However, many have tried it and it has become clear there are challenges. LLMs struggle with long-term memory, reliable state tracking, and understanding nuanced system context, all of which are critical for executing multi-step operations over time. They may generate plausible commands, but lack the robustness to adapt dynamically when output deviates from expectation or when tools return ambiguous or system-specific errors.

LLMs also have fundamental problems that may limit them.

They are currently consuming an ever increasing amount of compute for what seem to be marginal gains (although new models like DeepSeek suggest there may still be efficiencies to come). While they have mastered language, they struggle with simple reasoning and fail at tasks like counting R’s in “strawberry”. Researchers are trying to overcome these problems, but there is a lot of debate whether LLMs are “the future” or whether they’re just a cool trick before we develop in another direction.

Outside of LLMs, other neural networks are progressing, but they tend to be hyper-focused on a single task. We have seen they can perform well in complex incomplete information environments. In the long term, giving neural networks (or whatever comes after) direct access to the network stack (“here’s an IP header, here’s a TCP handshake, have fun”) is a realistic yet terrifying prospect.

This is where we are with AI in general, with an increasing number of tools, libraries, and APIs making it easier than ever to apply AI to different problems. As a response, we’ve been inundated with many new AI products over the past couple of years, with the primary public focus being tools that feed existing data into LLMs and provide a presentation layer on top of it.

Reason 1 – It’s hard

Try to play chess against an LLM and you will quickly see a problem: it will begin with expert level openings but soon descend into making impossible moves as the system has no knowledge of what state the board is actually in.

Command line hacking commands have to be entered perfectly or they will not carry out their intended actions. Large-scale or enterprise-wide compromises are rarely a single step and at a minimum likely require reconnaissance, exploitation, lateral traversal, and privilege escalation. That’s just to get a foothold before you begin to exfiltrate or encrypt data, embed persistence, or do whatever else you want to do once you are in. Even those individual stages are not simple: “learn privilege escalation” is not a single problem, since privesc on a Windows system looks nothing like privesc on Linux, and similar differences exist for almost every step.

From an AI perspective, each of these is a discrete task that is understood in isolation. However, these steps need to be carried out sequentially, often switching back and forth as new parts of the environment are discovered. Current non-LLM neural networks are normally built to solve a single challenge, while LLM’s show mastery of language but their context and state fall apart as complexity and ingested data increase.

This is a hard problem regardless of whether you are trying to solve it with a generalist system like an LLM or by chaining together a set of task-specific models. Most approaches fall apart on multi-stage reasoning, and combining that with a brittle understanding of system state makes full-chain automation a nightmare.

Even without full automation, incremental improvements in just one or two stages of the chain can significantly reduce the time it takes to breach a target. Faster recon, better exploit selection, or smarter lateral movement can all shorten the window defenders have to respond, which makes even partial advances in offensive AI dangerous.

Reason 2 – The market

When LLMs first opened their APIs there was a rush of developers trying to apply them to offensive security, with limited success due to the reasons outlined above. Additionally, out-of-the-box LLM models have built-in protections designed to prevent malicious use, creating another small speedbump for quick deployment of would-be offensive developers. It quickly became clear that attempts to develop offensive agents would require serious dedication of time, effort and cost that few could afford. Technologies were quick to develop that superficially looked impressive, but when carrying out a complete compromise chain, small errors escalate into unsolvable problems. Despite this, there are several players in the field who are all motivated to develop successful models.

Red teams and cybersecurity vendors might invest in deep, expensive research, but most focus on quick-to-market applications that deliver immediate business value. Few maintain large, experienced research teams dedicated to genuine innovation. While some promising AI-driven web application scanners exist, they remain limited to that narrow context. Certainly many are claiming AI capabilities and it makes good marketing material, but scratch the surface and most of it is wrapper code around LLM APIs or improved report generation. A few are pushing toward chaining real attacks, but most of what we’ve seen is focused on automating recon, evidence collection, or simulating known threats. Ultimately these platforms are designed to help defenders understand risk and provide assurance rather than burn down networks, so any tooling they have produced tends to identify vulnerabilities but stop short of exploitation.

Cybercrime, for its part, is usually about scale and speed, not novelty. Most groups run a handful of tried-and-tested chains, often starting with phishing, then escalating to ransomware or extortion. They’re creative in delivery, but the underlying techniques are well-established, and their innovation usually comes from adapting public tools rather than inventing new ones. That said, they’re fast followers, and if a vendor builds something they can use, expect to see cracked or reengineered versions in the wild within weeks.

There’s also academia. It’s not a for-profit market, but it remains a valuable source of innovation. Academic research tends to focus on narrow, well-scoped problems, and while it’s not driving end-to-end offensive AI, it has laid the foundations for many of the techniques others are now building on. These papers don’t arrive as prebuilt tools but often become the scaffolding for more applied work, particularly when published openly.

The real research, though, is almost certainly happening behind closed doors. Nation-states, particularly those with advanced signals intelligence agencies, have a long history of running years ahead of public understanding. Cryptography is the classic example: whether it’s public key crypto, elliptic curves, or quantum resistance, our intelligence agencies have time and time again proved that they have been a step or two ahead. Even outside of crypto, let’s not forget that EternalBlue – the exploit behind WannaCry and NotPetya – was an NSA exploit that gave point and click access to the majority of Windows hosts worldwide.

If a state actor were first to deploy autonomous cyber agents, we may not even know: not because it hasn’t happened, but because the best operators are stealthy by design. Unlike ransomware gangs who benefit from noise, nation-states often pursue persistence and discretion.

What happens next?

All of these tools show promise at one part of the attack chain; successfully chained together the speed of compromise will accelerate to levels our current defensive practices cannot deal with. With expensive 24/7 SOCs or MSSPs you’ll typically still have at best, SLAs of 15 minutes, far slower than automated attacks may need. Zero-day vulnerabilities will continue to be a challenge, as patching itself often takes hours, and that’s assuming a weekday with on call network engineers and emergency change boards. Over the past few years we’ve seen many on network edge devices such as Fortinet and Ivanti and I personally saw several cases where vulnerabilities were exploited but the network intrusion didn’t go any deeper because attackers were overwhelmed with options. With heavily automated attack chains, total environmental compromise could be near instantaneous.

Two things currently limit the speed of offensive cyber: 1) the knowledge of the attackers, which takes time and dedication to learn, and 2) how quickly they can work, having to track information over hundreds of different systems and services to target those that are most likely to yield results. Both are overcome as we automate more of the attack chain, and there are few inherent time sinks present in compromises (examples like UDP port scanning or password cracking stand out as exceptions, but they are typically optional steps). I see the dirty insides of a lot of networks, and it’s rare that there aren’t critical exploitation pathways inside, they just take time to find. If that time is reduced to near zero (in human terms), many of our current practices will not suffice.

What makes this eventual outcome likely is that we already have the foundational technology to get there, we simply need to apply it to the specific domain of offensive cyber. Don’t get me wrong, “simply” is doing some heavy-lifting and this will require large well-funded research projects, but given the stakes at play sooner or later someone is going to get there.

So as practitioners of defensive cyber, what should we be doing to prepare?

---