Securely Enabling Agents to Run Tests That Rely on Containers

testcontainers and similar setups let automated tests use ephemeral data stores (Redis, PostgreSQL, etc.), along with basically any other containerized application. They keep test scenarios more in line with real production environments while also saving lots of time that otherwise might’ve been spent mocking said applications. A true win-win.

The downside is that these setups need to spin up containers. Giving an AI agent control of a container runtime presents a clear security risk because the agent can use it to break containment. That’s why many agent sandboxes block access to container runtimes unless you explicitly poke holes in their isolation. But if AI agents aren’t allowed to run test cases, they become a whole lot less useful.

I’ve spent some time looking for an answer to this quandary and haven’t found a satisfactory one. Most practical solutions involve some kind of virtualization, which almost always requires root access to set up. However, I just want a simple script I can run to make this work until more standardized, secure tooling is widely adopted.

My idea is to sidestep the problem by not having the agent create the containers itself. Instead, I’ll create a Docker Compose file that lets me spin up everything with one command. Then I’ll have the agent make slight adjustments to the Testcontainers fixtures so they use those containers when it runs the tests.

The one security compromise I still have to make, however, is letting the agent talk to localhost. While this is more secure than giving it access to the Docker daemon, it should still be done cautiously. With Codex, I did this by adding the following config.toml file under .codex/ in my project. As of this writing, the newer permission profiles are still in beta, so I’m using the older sandbox_workspace_write syntax:

TOML

[sandbox_workspace_write]
network_access = true

[features.network_proxy]
enabled = true
domains = { "127.0.0.1" = "allow" }
allow_local_binding = false

First things first, let’s take a look at my soon to be billion dollar SaaS project (please don’t copy):

PYTHON

from uuid import uuid4

from redis import Redis


redis_client = Redis(host="localhost", port=6379, decode_responses=True)


def get_unique_uuid() -> str:
    while True:
        candidate = str(uuid4())
        if not redis_client.exists(candidate):
            redis_client.set(candidate, "issued")
            return candidate

Now let’s take a look at the test suite:

PYTHON

from unittest.mock import call, patch

import pytest
from testcontainers.community.redis import RedisContainer

import uuid_service


EXAMPLE_UUID = "00000004-0008-0015-0016-000000023042"


@pytest.fixture(scope="module")
def redis_client():
    with RedisContainer() as container:
        yield container.get_client()


@pytest.fixture(autouse=True)
def use_test_redis(redis_client, monkeypatch):
    redis_client.flushdb()
    monkeypatch.setattr(uuid_service, "redis_client", redis_client)


def test_returns_and_stores_new_uuid(redis_client):
    with patch.object(uuid_service, "uuid4", return_value=EXAMPLE_UUID):
        with patch.object(
            redis_client, "exists", wraps=redis_client.exists
        ) as exists:
            result = uuid_service.get_unique_uuid()

    assert result == EXAMPLE_UUID
    exists.assert_called_once_with(EXAMPLE_UUID)
    assert redis_client.exists(result) == 1


def test_retries_when_uuid_already_exists(redis_client):
    redis_client.set(EXAMPLE_UUID, "issued")

    with patch.object(
        uuid_service,
        "uuid4",
        side_effect=[EXAMPLE_UUID, EXAMPLE_UUID[::-1]],
    ):
        with patch.object(
            redis_client, "exists", wraps=redis_client.exists
        ) as exists:
            result = uuid_service.get_unique_uuid()

    assert result == EXAMPLE_UUID[::-1]
    assert exists.call_args_list == [
        call(EXAMPLE_UUID),
        call(EXAMPLE_UUID[::-1]),
    ]
    assert redis_client.exists(result) == 1

Let’s run the tests ourselves and verify that everything works:

CONSOLE

$ pytest
..                                                                       [100%]
2 passed in 0.53s

Great, now let’s have Codex run the tests:

TEXT

May I let the test suite access your local Docker daemon so Testcontainers can start its Redis container?

As expected, Codex can’t start the Testcontainers-managed Redis container without access to my local Docker daemon. Let’s use Docker Compose to keep Redis on an internal network and expose it through a fixed HAProxy TCP proxy that Codex can reach without controlling Docker itself.

Let’s save this setup as compose.yaml:

YAML

services:
  redis:
    image: redis:8-alpine
    networks:
      - redis-network

  redis-proxy:
    image: haproxy:3.2-alpine
    ports:
      - "127.0.0.1:6379:6379"
    networks:
      - redis-network
      - internet
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro

networks:
  redis-network:
    internal: true

  internet:

Then let’s create haproxy.cfg next to compose.yaml with a fixed Redis backend:

TEXT

defaults
    mode tcp
    timeout connect 5s
    timeout client 30s
    timeout server 30s

frontend redis
    bind :6379
    default_backend redis

backend redis
    server redis redis:6379

Now let’s update the test fixture so it falls back to the Redis instance on localhost when Docker isn’t available. Since Redis uses raw TCP, the fallback uses Codex’s SOCKS proxy when ALL_PROXY is present:

PYTHON

import os

from docker.errors import DockerException
from python_socks.sync import Proxy
from redis import Redis
from redis.connection import Connection, ConnectionPool


class SocksConnection(Connection):
    def __init__(self, *args, proxy_url, **kwargs):
        super().__init__(*args, **kwargs)
        self.proxy_url = proxy_url.replace("socks5h://", "socks5://", 1)

    def _connect(self):
        sock = Proxy.from_url(self.proxy_url).connect(
            dest_host=self.host,
            dest_port=self.port,
            timeout=self.socket_connect_timeout,
        )
        sock.settimeout(self.socket_timeout)
        return sock


def local_redis_client():
    proxy_url = os.environ.get("ALL_PROXY")
    if proxy_url:
        return Redis(
            connection_pool=ConnectionPool(
                connection_class=SocksConnection,
                proxy_url=proxy_url,
                host="127.0.0.1",
                port=6379,
            )
        )

    return Redis(host="127.0.0.1", port=6379)


@pytest.fixture(scope="module")
def redis_client():
    try:
        container = RedisContainer()
        container.start()
    except DockerException:
        print(
            "Unable to start Testcontainers Redis; attempting to connect to "
            "an existing Redis instance on port 6379"
        )
        yield local_redis_client()
        return

    try:
        yield container.get_client()
    finally:
        container.stop()

Now lets have codex try again!

CONSOLE

$ pytest
..                                                                       [100%]
2 passed in 0.16s

General Hammond from Stargate shouting “YeeHaw!”

Well, that was fun, but it was quite a lot of work just to let pytest access a Redis container for integration tests. That’s why I’m starting to think the safest and most effective place for agents to live long term won’t be the same environments developers use day to day. Instead, they’ll need environments built from the ground up with agent tooling and security in mind. I’m not sure what the standard solution will end up being: lightweight VMs, browser sandboxes, or something else entirely.

In the meantime, since I intend to get more and more value out of agents, I’m building iris-sandbox as part of my Adventures in Declarative Agent Security series . My goal is to reach the point where you can define the tools and agents you want in a simple YAML file, run docker compose up, exec into the environment, and start giving them work. There’s still a long way to go, though: supporting more agents and tools, building the YAML configuration layer, and figuring out the best way to get repositories and files into the environment and commits back out.