Blog · 28 September 2024 · Esben Kran · 10 min read

Let's make AI agents safe

AI agents are ‘robots in cyberspace.’

They are computing systems with a brain designed to orchestrate actions from perception.

An agent could be a Discord bot that receives every message on a server (Perception), decides whether the message is spam (Brain), and then deletes the message (Action).

However, an agent could also be a cyber operative bot which receives orders to find vulnerabilities on government websites and executes a Perception -> Brain -> Action sequence on its own plans.

Diagram of an AI agent's architecture, from Xi et al. (2023). The agent takes in inputs from its environment, such as text, images, audio and locations (Perception). Its brain stores memory and knowledge and does planning and reasoning to make decisions (Brain). It then acts through text, tool calls such as APIs, and a robot body (Action). Example: a person asks if it will rain tomorrow and to be given an umbrella; the agent reasons from the current weather and online weather reports that it will likely rain, and hands over the umbrella.

From Xi et al. (2023)

Of course, this is far from an exhaustive list of things AI agents can do - for good or bad.

The question is how bad it might get. If agents become smarter than humans and begin to act of their own volition, they could ‘attempt to seize control of the world, resulting in irreversible consequences for humanity.’^[See also Bengio 2023]

As a result, the safety of AI agents seems critical for our shared future. Now, the real question becomes how we can go from a potential existential risk to a world with agents that pose no danger to humanity and other life?

Why agent safety is a difficult research field

Agent safety research is difficult because it involves many different types of entities and wide range of vulnerabilities and failure modes. As a result, it’s hard to develop research that generalizes to all agents. However, we need to give it a shot!

Compared to traditional security research, the field of agent safety ensures that the world is safe from high-risk agent deployments. A key risk to avoid is the possibility that AI agents “go rogue” (Bengio 2023, He et al. 2024) - that they take autonomous actions outside the oversight of humans and cause catastrophic damage or otherwise disenfranchise society.

There’s a few ways to approach this question. Either, we implement algorithms that make them verifiably safe or we evaluate when an agent is worrying us and build automatic capabilities to shut it down. Let’s take a short look at both:

Making agents safer

Research is not useful without application. In our work, we need to directly make agents safer while ensuring that our solutions work for any agent. This includes making agent communication reliable and controlled, improving oversight of agents’ actions, improving separation of concerns to avoid leaking data or permissions, reducing the risk of rogue AI, having tight controls on agent permissions, and having verifiably safe agent tools.

“Security of AI Agents” is a great overview of current security vulnerabilities in agent architectures. The authors also describe solutions, such as maintaining sessions for user-agent interaction similar to website sessions to avoid data leaking between users and improve control. Most of the paper (which I highly recommend giving a read) details cybersecurity risks, though it also points out that agents may “overuse” permissions and system control if given the ability to; risks similar to what METR found during their GPT-4 safety testing.

In this category, we’re also interested in understanding how we can create technology to support responsible regulatory oversight. As Xi et al. (2023) mention, “Stringent regulatory policies need to be established to ensure the responsible use of LLM-agents”. However, if we don’t want a surveillance state in the face of AI risk, we’ll need to come up with new ways to improve oversight.

With “building safer agents” and “improving the deployment infrastructure to support control and safety” in mind, let’s come up with a few ideas we can pursue:

A note on multi-agent research

I suggest that you avoid multi-agent research when exploring agent safety. Often, a multi-agent research project becomes too complex to study in any useful capacity and if we want to build multi-agent coordination mechanisms, we will need single-agent control systems anyways, meaning that you are always bound by the technology available at the single-agent level when researching multiple agents.

That is not to say it isn’t important and I suggest you take a look at the NeurIPS Concordia Contest (…and our previous hackathon) if you are interested in this topic.

Foundations of agency, alignment, and LLMs

The field of agent foundations has been a rabbit hole of research into how to design the main drives of agents so they become safe and reliable (Soares & Fallenstein, 2017). Despite its somewhat lackluster research output during the last 20-or-so years, you may argue that it paved the way for modern AI.

The field explores topics such as what it means for a learning system to learn an algorithm, what the fundamental properties of learning architectures are, and how we can give the correct specifications to agents.

In the context of building safety in agents, this field does become relevant, with theoretical insights such as the loss of human agency in the face of expert systems and designing the reward function to follow human specifications.

Some research ideas for what we might explore in this direction (remember if we don’t have compute, we can primitively use prompting to change model behavior):

This is an elusive field and if you are just starting out, I would recommend sticking to the empirical research mentioned in other parts of this post. It’s immensely important but it’s incredibly easy to get stuck.

When does an agent become scary?

Testing agents is a complex task and hundreds of researchers have developed tests for various abilities across domains. For the conversation here, we’ll constrain our analysis to testing and monitoring agents for catastrophic risk.

The scientific question of when an agent becomes worrying to us is currently under discussion. From my perspective, most risk arises in the application areas the agent is optimized for, such as cyber offense operations, autonomous AI research, systematic manipulation, and complex multi-agent coordination.

A general heuristic might be “if I would be worried that a malicious human would be able to do this, then I would be worried if any agent is able to do this.”

Detecting capabilities

Task-based testing has become a promising avenue (see e.g. Lin et al. 2023). Organizations such as the UK AI Safety Institute, METR and Palisade Research are engaging in “Demonstrations as Evaluations”, evaluating agents by their ability to complete tasks that simultaneously serve as believable demonstrations for real-world capability.

The main idea for the above organizations is to identify which tasks we would be very worried if autonomous agents are capable of, construct challenges for said task, and test autonomous agents against these challenges. To avoid going down the rabbit hole of Inspect, the METR Task Standard and SWE-bench, it’s fair to say that this is one of the most promising testing approaches we have for realistic capability testing.

Some research projects you can develop

Where to go next?

I hope this short overview of how we can make progress on agent safety was insightful and inspired. If you are interested in exploring the topic further, check out the following resources:

Optional resources:

Of course, if you aren’t already, consider joining our Agent Security Hackathon where we spend a weekend solving exactly these problems.

Let’s make agents safe.

Originally published on blog.kran.ai on 28 September 2024.