.. _client-docker: Client Docker & CUDA ===================== This guide covers running the Scaleout Edge Python client in Docker — useful for deploying edge clients without managing a Python environment on the host, and for training on a GPU via the NVIDIA Container Toolkit. A ready-to-run example lives in ``scaleout-client-python/examples/docker-client`` in the `scaleout-client repository `__. It bundles the same model as the ``mnist-pytorch`` example (see :doc:`projects`) as a starting point to adapt. .. important:: This example builds ``scaleout``/``scaleoututil`` from the source in this repository, not from PyPI, so you can run a client from a branch that hasn't been released yet. That means the build needs ``scaleout-util/`` and ``scaleout-client-python/`` present as siblings at the repo root — **clone the full repository**; copying just the ``docker-client`` folder out on its own will not build. .. contents:: On this page :local: :depth: 1 ---- How the image works -------------------- The Docker image installs ``scaleout``/``scaleoututil`` from the ``scaleout-util/`` and ``scaleout-client-python/`` source directories alongside it (see the important note above) — it does **not** bundle PyTorch, TensorFlow, or any other ML framework, and it does **not** need a CUDA base image. At startup, the client's managed Python environment feature installs whatever your ``client/python_env.yaml`` declares into a venv, dynamically (see :ref:`local_compute-label`). Since PyPI's ``torch`` wheels bundle their own CUDA runtime, GPU support reduces entirely to whether the *host* exposes a GPU to the *container* — nothing in the image needs to change between a CPU-only and a GPU run, **on x86_64**. On an aarch64 host (for example an NVIDIA Jetson) this does not hold: PyPI does not publish a CUDA-enabled ``torch`` wheel for that platform at all, so a plain ``pip install torch`` there silently resolves to a CPU-only build even with the GPU overlay active. ``entrypoint.sh`` detects an aarch64 host with a GPU actually passed through and points pip at NVIDIA's own wheel index in that case; see :ref:`jetson-gpu` below. Mount contract --------------- .. list-table:: :header-rows: 1 :widths: 25 25 50 * - Path in container - Mode - Purpose * - ``/app/client`` - read-only bind mount - Your ML code: ``scaleout.yaml``, ``python_env.yaml``, ``startup.py``, ``model.py``, ``data.py``. Replace this directory's contents to adapt the example to your own use case. * - ``/app/data`` - read-write bind mount - Your training data, kept separate from the (read-only) code mount. * - ``/app/.scaleout`` - named volume - Extracted package + managed venv cache. Persisting this avoids reinstalling large GPU wheels (like ``torch``) on every container restart. * - ``/root/.scaleout`` - named volume - Cached refresh token (see :ref:`client-envs`). Lets the client resume after a restart without needing a fresh enrollment token. Because ``/app/client`` is read-only, any code that writes to disk (for example, downloading or preprocessing a dataset) must write somewhere else. The bundled ``data.py`` demonstrates the pattern: it reads an ``SCALEOUT_DATA_DIR`` environment variable (set to ``/app/data`` in ``docker-compose.yml``) instead of writing next to itself. Follow the same pattern when adapting ``data.py`` to your own data pipeline. This example's container runs as ``root`` (there is no ``USER`` directive in the Dockerfile) — a deliberate simplification for an example meant to be forked and adapted, not a hardened production posture. If you deploy this to a real fleet of edge devices, add a non-root ``USER`` to the Dockerfile and review file permissions on the mounted volumes. The ``scaleout_tokens`` volume holds a long-lived refresh token in an **unencrypted** JSON file (0600 permissions, but not protected against root access, backups that drop permissions, or other processes running as the same user). Treat that volume as a credential at rest: back it up and restrict access accordingly, or set ``SCALEOUT_PERSIST_TOKENS=false`` to disable the cache entirely (at the cost of needing a fresh enrollment token on every restart). Quickstart ---------- .. code-block:: bash cd scaleout-client-python/examples/docker-client cp .env.example .env # Edit .env: set SCALEOUT_API_URL, SCALEOUT_CLIENT_ID, and a short-lived # SCALEOUT_TOKEN from `scaleout client create-enrollment-token`. docker compose up --build On the first run, the client self-enrolls using ``SCALEOUT_TOKEN`` and caches a refresh token in the ``scaleout_tokens`` volume. Remove ``SCALEOUT_TOKEN`` from ``.env`` afterward — restarts pick up the cached refresh token automatically (see :ref:`client-envs` for the underlying mechanism), and a stale enrollment token left in place would otherwise fail once it expires. ---- Enabling CUDA / GPU training ------------------------------ Prerequisites (on the Docker host, not inside the container): 1. **An NVIDIA GPU with a recent driver.** Verify with: .. code-block:: bash nvidia-smi 2. **The NVIDIA Container Toolkit**, which lets Docker pass the GPU through to containers. .. only:: html .. tabs:: .. code-tab:: bash :caption: Linux (Debian/Ubuntu) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker .. code-tab:: bash :caption: Windows (WSL2) # Install a recent NVIDIA driver on Windows (not inside WSL) from # https://www.nvidia.com/Download/index.aspx, then inside your # WSL2 distro run the same steps as the Linux/Ubuntu tab. Do not # install a separate NVIDIA driver inside WSL2 itself. .. only:: latex or rinoh *Linux (Debian/Ubuntu):* .. code-block:: bash curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker *Windows (WSL2):* install a recent NVIDIA driver on Windows itself (not inside WSL2), then run the Linux/Ubuntu steps above inside your WSL2 distro. 3. **Verify the toolkit** works before touching the Scaleout client image: .. code-block:: bash docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi This must print the same GPU information as the host's own ``nvidia-smi``. If it doesn't, fix that first — the Scaleout client image has no diagnostics of its own to add here, since it doesn't contain the CUDA toolkit. 4. **Start the client with the GPU overlay:** .. code-block:: bash docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build The overlay adds a standard Compose GPU device reservation (``deploy.resources.reservations.devices``); no other change is needed. Your ``startup.py``'s ``torch.cuda.is_available()`` check (unchanged from the ``mnist-pytorch`` example) will now return ``True`` **on an x86_64 host**. On an aarch64 host, see :ref:`jetson-gpu` below before expecting the same result. .. _jetson-gpu: NVIDIA Jetson (aarch64) hosts ------------------------------ Steps 1–4 above still apply — the NVIDIA Container Toolkit and the GPU compose overlay work the same way on a Jetson host. The one difference is where ``torch`` comes from: PyPI does not publish a CUDA-enabled ``torch`` wheel for aarch64 at all, only a CPU-only one, so the plain ``pip install torch`` triggered by ``python_env.yaml`` needs to be pointed at a different index for this platform. This example handles it automatically: ``entrypoint.sh`` checks for an aarch64 host with a GPU actually passed through (``uname -m`` plus ``nvidia-smi`` being reachable, which is only true once the container runtime has injected it) and exports ``PIP_EXTRA_INDEX_URL`` and ``PIP_PRE`` before ``scaleout client start`` runs, so the managed venv installs a CUDA-enabled build from NVIDIA's own index instead. This was verified hardware-in-the-loop on a Jetson AGX Orin Developer Kit running JetPack 7.2 (Ubuntu 24.04, Python 3.12, CUDA 13.2): ``torch==2.14.0+cu132`` and ``torchvision==0.29.0+cu132`` install and run real GPU kernels (``sm_87``) inside this same container image. ``client/python_env.yaml``'s ``torch``/``torchvision`` entries for ``platform_machine == "aarch64"`` are intentionally left unpinned, since the version NVIDIA's index currently serves may be newer than the ``<2.10.0`` upper bound used for the x86_64 branch. If you need a specific version on Jetson, pin it in your own ``python_env.yaml`` after checking what NVIDIA's index (or, on JetPack 6.2, ``pypi.jetson-ai-lab.io``) has available for your JetPack version and Python version. At the time of writing, only the JetPack 7.2 combination above has been verified end-to-end (image build, GPU passthrough, and a real CUDA kernel running inside the container). JetPack 6.2 ships Python 3.10 as its system interpreter, which is below this client's minimum supported Python version, so JetPack 7.2 or newer is required for Jetson. ---- Adapting the example to your own ML code ------------------------------------------- Replace the contents of ``client/`` with your own project, using the same layout as the other examples in this repository (see :doc:`projects`): .. code-block:: text client/ |-- scaleout.yaml # entry points: build / startup |-- python_env.yaml # your ML framework + dependencies |-- startup.py # wires train/validate callbacks |-- model.py # your model definition `-- data.py # your data loading -- write under $SCALEOUT_DATA_DIR Point ``./data`` (bind-mounted to ``/app/data``) at your own dataset. No image rebuild is required for either change — only ``docker compose up`` again. See also -------- - :doc:`projects` — the ``client/`` project structure this example follows. - :doc:`localcompute` — how local vs. remote compute packages work. - :doc:`client-envs` — the client's environment variables. Note: the token-cache and managed-venv/archive-directory variables this example's volumes back (``SCALEOUT_TOKEN_CACHE_DIR``, ``SCALEOUT_HOME_DIR``, ``SCALEOUT_PERSIST_TOKENS``, ``SCALEOUT_VENV_DIR``, ``SCALEOUT_ARCHIVE_DIR``) are not yet documented on that page. - :doc:`deployment` — the Compose vs. Kubernetes deployment split for the *server side* (Controller/Combiner/UI), independent of this client-side guide.