docker compose run keeps the one-off container running after you kill the compose client

Symptom

A test script starts a one-off copy of a service with docker compose run in the background, waits for a log line, then kills the background job. The script ends, and the next test in the same stack fails in ways that look like races:

• state changes between two of its steps that nothing in the test caused (in the reference case, a background worker re-ran a layout pass and moved items between the test's steps);
• memory pressure (docker compose exec exiting with 137);
• two consumers on the same queue or schedule.

docker ps shows a container named like <project>-<service>-run-<hex> still Up, although the docker compose run process is gone.

When it happens

docker compose run --rm --no-deps -T -e NODE_ENV=production worker \
  sh -c 'timeout 45 npx tsx src/worker/index.ts' > "$LOG" 2>&1 &
RUNPID=$!
# ... wait for a line in $LOG, assert things ...
kill "$RUNPID"; wait "$RUNPID"      # the client exits; the container does not

Reproduced with Docker Compose v5.4.0:

$ docker compose -p crun run --rm --no-deps -T worker sh -c 'sleep 40' &
$ docker ps --format '{{.Names}} {{.Status}}'
crun-worker-run-7b1059c70c64 Up 4 seconds
$ kill $!; wait $!; echo "client exit=$?"
client exit=143
$ docker ps --format '{{.Names}} {{.Status}}'      # 3 s later
crun-worker-run-7b1059c70c64 Up 7 seconds

It hurts most when suites run back to back on one stack (sharded or serial CI). The leftover container keeps working for the rest of its timeout, beside the next suite's own copy of the service.

Cause

docker compose run is a client. It creates a container in the Docker engine, attaches to it and waits. Killing the client (SIGTERM from kill, or the CI step ending) stops the client. The container belongs to the engine and runs on until its command exits. --rm removes the container only after it stops, so it doesn't help here. The container's name is generated (…-run-<random>), so the script doesn't know what to remove.

Fix

Name the container, and remove it explicitly, both right after the step and in an EXIT trap:

RUNNAME="probe-worker-$(date +%s)"
trap 'docker rm -f "${RUNNAME:-none}" > /dev/null 2>&1' EXIT

docker compose run --rm --no-deps -T --name "$RUNNAME" -e NODE_ENV=production worker \
  sh -c 'timeout 120 node_modules/.bin/tsx src/worker/index.ts' > "$LOG" 2>&1 &
RUNPID=$!
# ... wait, assert ...
kill "$RUNPID" 2>/dev/null || true; wait "$RUNPID" 2>/dev/null || true
docker rm -f "$RUNNAME" > /dev/null 2>&1 || true

• docker rm -f stops and removes the container in one step, whatever state it's in.
• The trap covers a failing assertion or set -e exit before the cleanup line.
• If the script already has an EXIT trap, add the rm -f to it. A second trap … EXIT replaces the first.
• Use a name that's unique per run, so parallel runs on one engine don't remove each other's containers.

Verify

docker compose run --rm --no-deps -T --name crun-probe worker sh -c 'sleep 40' &
sleep 5; kill $!; wait $!
docker rm -f crun-probe
docker ps -a --filter name=crun-probe --format '{{.Names}}'     # nothing

In a test suite, assert at the end that no container from the run is left, for example docker ps -q --filter label=com.docker.compose.oneoff=True --filter label=com.docker.compose.project=<project> returns nothing.

Notes

• A related failure from the same probe. Started with NODE_ENV=production, the worker died at import, before the line the test waited for:

MAILER=console prints verification tokens to logs — set MAILER=smtp (with SMTP_* vars) in production

A module the worker had just started importing (pulled in through an unrelated auth helper) refuses a console mailer in production. This was not only a test problem: the same import crashed the production worker after the next deploy, because its compose service had no mail settings. The probe got -e MAILER=smtp -e SMTP_HOST=127.0.0.1 (the SMTP transport connects only when it sends), and the production worker got the same mail settings as the api. Lesson: when a production-mode probe dies at import, check whether production would die the same way, and boot every service with production settings in CI.

• npx tsx … inside the container may contact the npm registry before it runs. Calling node_modules/.bin/tsx directly made the probe start faster on a slow shared CI host.