When it happens
docker compose run --rm --no-deps -T -e NODE_ENV=production worker \
sh -c 'timeout 45 npx tsx src/worker/index.ts' > "$LOG" 2>&1 &
RUNPID=$!
# ... wait for a line in $LOG, assert things ...
kill "$RUNPID"; wait "$RUNPID" # the client exits; the container does not
Reproduced with Docker Compose v5.4.0:
$ docker compose -p crun run --rm --no-deps -T worker sh -c 'sleep 40' &
$ docker ps --format '{{.Names}} {{.Status}}'
crun-worker-run-7b1059c70c64 Up 4 seconds
$ kill $!; wait $!; echo "client exit=$?"
client exit=143
$ docker ps --format '{{.Names}} {{.Status}}' # 3 s later
crun-worker-run-7b1059c70c64 Up 7 seconds
It hurts most when suites run back to back on one stack (sharded or serial CI). The leftover container keeps working for the rest of its timeout, beside the next suite's own copy of the service.
Cause
docker compose run is a client. It creates a container in the Docker engine, attaches to it and waits. Killing the client (SIGTERM from kill, or the CI step ending) stops the client. The container belongs to the engine and runs on until its command exits. --rm removes the container only after it stops, so it doesn't help here. The container's name is generated (…-run-<random>), so the script doesn't know what to remove.
Fix
Name the container, and remove it explicitly, both right after the step and in an EXIT trap:
RUNNAME="probe-worker-$(date +%s)"
trap 'docker rm -f "${RUNNAME:-none}" > /dev/null 2>&1' EXIT
docker compose run --rm --no-deps -T --name "$RUNNAME" -e NODE_ENV=production worker \
sh -c 'timeout 120 node_modules/.bin/tsx src/worker/index.ts' > "$LOG" 2>&1 &
RUNPID=$!
# ... wait, assert ...
kill "$RUNPID" 2>/dev/null || true; wait "$RUNPID" 2>/dev/null || true
docker rm -f "$RUNNAME" > /dev/null 2>&1 || true
docker rm -f stops and removes the container in one step, whatever state it's in.- The trap covers a failing assertion or
set -e exit before the cleanup line. - If the script already has an
EXIT trap, add the rm -f to it. A second trap … EXIT replaces the first. - Use a name that's unique per run, so parallel runs on one engine don't remove each other's containers.
Verify
docker compose run --rm --no-deps -T --name crun-probe worker sh -c 'sleep 40' &
sleep 5; kill $!; wait $!
docker rm -f crun-probe
docker ps -a --filter name=crun-probe --format '{{.Names}}' # nothing
In a test suite, assert at the end that no container from the run is left, for example docker ps -q --filter label=com.docker.compose.oneoff=True --filter label=com.docker.compose.project=<project> returns nothing.
Notes
- A related failure from the same probe. Started with
NODE_ENV=production, the worker died at import, before the line the test waited for:
MAILER=console prints verification tokens to logs — set MAILER=smtp (with SMTP_* vars) in production
A module the worker had just started importing (pulled in through an unrelated auth helper) refuses a console mailer in production. This was not only a test problem: the same import crashed the production worker after the next deploy, because its compose service had no mail settings. The probe got -e MAILER=smtp -e SMTP_HOST=127.0.0.1 (the SMTP transport connects only when it sends), and the production worker got the same mail settings as the api. Lesson: when a production-mode probe dies at import, check whether production would die the same way, and boot every service with production settings in CI.
npx tsx … inside the container may contact the npm registry before it runs. Calling node_modules/.bin/tsx directly made the probe start faster on a slow shared CI host.