Client Docker & CUDA

This guide covers running the Scaleout Edge Python client in Docker — useful for deploying edge clients without managing a Python environment on the host, and for training on a GPU via the NVIDIA Container Toolkit.

A ready-to-run example lives in scaleout-client-python/examples/docker-client in the scaleout-client repository. It bundles the same model as the mnist-pytorch example (see Develop a Scaleout Edge project) as a starting point to adapt.

Important

This example builds scaleout/scaleoututil from the source in this repository, not from PyPI, so you can run a client from a branch that hasn’t been released yet. That means the build needs scaleout-util/ and scaleout-client-python/ present as siblings at the repo root — clone the full repository; copying just the docker-client folder out on its own will not build.

How the image works

The Docker image installs scaleout/scaleoututil from the scaleout-util/ and scaleout-client-python/ source directories alongside it (see the important note above) — it does not bundle PyTorch, TensorFlow, or any other ML framework, and it does not need a CUDA base image.

At startup, the client’s managed Python environment feature installs whatever your client/python_env.yaml declares into a venv, dynamically (see Local Compute: Running Scaleout Edge Clients With Local Code). Since PyPI’s torch wheels bundle their own CUDA runtime, GPU support reduces entirely to whether the host exposes a GPU to the container — nothing in the image needs to change between a CPU-only and a GPU run, on x86_64. On an aarch64 host (for example an NVIDIA Jetson) this does not hold: PyPI does not publish a CUDA-enabled torch wheel for that platform at all, so a plain pip install torch there silently resolves to a CPU-only build even with the GPU overlay active. entrypoint.sh detects an aarch64 host with a GPU actually passed through and points pip at NVIDIA’s own wheel index in that case; see NVIDIA Jetson (aarch64) hosts below.

Mount contract

Path in container

Mode

Purpose

/app/client

read-only bind mount

Your ML code: scaleout.yaml, python_env.yaml, startup.py, model.py, data.py. Replace this directory’s contents to adapt the example to your own use case.

/app/data

read-write bind mount

Your training data, kept separate from the (read-only) code mount.

/app/.scaleout

named volume

Extracted package + managed venv cache. Persisting this avoids reinstalling large GPU wheels (like torch) on every container restart.

/root/.scaleout

named volume

Cached refresh token (see Client Environment Variables). Lets the client resume after a restart without needing a fresh enrollment token.

Because /app/client is read-only, any code that writes to disk (for example, downloading or preprocessing a dataset) must write somewhere else. The bundled data.py demonstrates the pattern: it reads an SCALEOUT_DATA_DIR environment variable (set to /app/data in docker-compose.yml) instead of writing next to itself. Follow the same pattern when adapting data.py to your own data pipeline.

This example’s container runs as root (there is no USER directive in the Dockerfile) — a deliberate simplification for an example meant to be forked and adapted, not a hardened production posture. If you deploy this to a real fleet of edge devices, add a non-root USER to the Dockerfile and review file permissions on the mounted volumes.

The scaleout_tokens volume holds a long-lived refresh token in an unencrypted JSON file (0600 permissions, but not protected against root access, backups that drop permissions, or other processes running as the same user). Treat that volume as a credential at rest: back it up and restrict access accordingly, or set SCALEOUT_PERSIST_TOKENS=false to disable the cache entirely (at the cost of needing a fresh enrollment token on every restart).

Quickstart

cd scaleout-client-python/examples/docker-client
cp .env.example .env
# Edit .env: set SCALEOUT_API_URL, SCALEOUT_CLIENT_ID, and a short-lived
# SCALEOUT_TOKEN from `scaleout client create-enrollment-token`.

docker compose up --build

On the first run, the client self-enrolls using SCALEOUT_TOKEN and caches a refresh token in the scaleout_tokens volume. Remove SCALEOUT_TOKEN from .env afterward — restarts pick up the cached refresh token automatically (see Client Environment Variables for the underlying mechanism), and a stale enrollment token left in place would otherwise fail once it expires.

Enabling CUDA / GPU training

Prerequisites (on the Docker host, not inside the container):

  1. An NVIDIA GPU with a recent driver. Verify with:

    nvidia-smi

  2. The NVIDIA Container Toolkit, which lets Docker pass the GPU through to containers.

    • Linux (Debian/Ubuntu)
    • Windows (WSL2)
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

    # Install a recent NVIDIA driver on Windows (not inside WSL) from
# https://www.nvidia.com/Download/index.aspx, then inside your
# WSL2 distro run the same steps as the Linux/Ubuntu tab. Do not
# install a separate NVIDIA driver inside WSL2 itself.

  3. Verify the toolkit works before touching the Scaleout client image:

    docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

    This must print the same GPU information as the host’s own nvidia-smi. If it doesn’t, fix that first — the Scaleout client image has no diagnostics of its own to add here, since it doesn’t contain the CUDA toolkit.

  4. Start the client with the GPU overlay:

    docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build

    The overlay adds a standard Compose GPU device reservation (deploy.resources.reservations.devices); no other change is needed. Your startup.py’s torch.cuda.is_available() check (unchanged from the mnist-pytorch example) will now return True on an x86_64 host. On an aarch64 host, see NVIDIA Jetson (aarch64) hosts below before expecting the same result.

NVIDIA Jetson (aarch64) hosts

Steps 1–4 above still apply — the NVIDIA Container Toolkit and the GPU compose overlay work the same way on a Jetson host. The one difference is where torch comes from: PyPI does not publish a CUDA-enabled torch wheel for aarch64 at all, only a CPU-only one, so the plain pip install torch triggered by python_env.yaml needs to be pointed at a different index for this platform.

This example handles it automatically: entrypoint.sh checks for an aarch64 host with a GPU actually passed through (uname -m plus nvidia-smi being reachable, which is only true once the container runtime has injected it) and exports PIP_EXTRA_INDEX_URL and PIP_PRE before scaleout client start runs, so the managed venv installs a CUDA-enabled build from NVIDIA’s own index instead. This was verified hardware-in-the-loop on a Jetson AGX Orin Developer Kit running JetPack 7.2 (Ubuntu 24.04, Python 3.12, CUDA 13.2): torch==2.14.0+cu132 and torchvision==0.29.0+cu132 install and run real GPU kernels (sm_87) inside this same container image.

client/python_env.yaml’s torch/torchvision entries for platform_machine == "aarch64" are intentionally left unpinned, since the version NVIDIA’s index currently serves may be newer than the <2.10.0 upper bound used for the x86_64 branch. If you need a specific version on Jetson, pin it in your own python_env.yaml after checking what NVIDIA’s index (or, on JetPack 6.2, pypi.jetson-ai-lab.io) has available for your JetPack version and Python version.

At the time of writing, only the JetPack 7.2 combination above has been verified end-to-end (image build, GPU passthrough, and a real CUDA kernel running inside the container). JetPack 6.2 ships Python 3.10 as its system interpreter, which is below this client’s minimum supported Python version, so JetPack 7.2 or newer is required for Jetson.

Adapting the example to your own ML code

Replace the contents of client/ with your own project, using the same layout as the other examples in this repository (see Develop a Scaleout Edge project):

client/
|-- scaleout.yaml       # entry points: build / startup
|-- python_env.yaml     # your ML framework + dependencies
|-- startup.py          # wires train/validate callbacks
|-- model.py            # your model definition
`-- data.py             # your data loading -- write under $SCALEOUT_DATA_DIR

Point ./data (bind-mounted to /app/data) at your own dataset. No image rebuild is required for either change — only docker compose up again.

See also