.. _scaleout-cli:

Command-Line Interface (CLI)
============================

The Scaleout Edge Command-Line Interface (CLI) is designed to streamline management of the Scaleout Edge platform, making it easier for users to deploy, monitor and interact with their Edge network.

With the Scaleout Edge CLI, users can:

- Start and manage Scaleout Edge services such as the **combiner**, **controller**, **API server**, and **Server Function hooks** (requires Enterprise license or specific research related agreement).
- Start and manage **clients**.
- Interact with the **controller** to:

  - Manage sessions, including starting, stopping, and monitoring their progress.
  - Retrieve data and results related to sessions, such as aggregated models and validation metrics.
  - Query the state of the network, including the status of connected combiners and clients.

For detailed usage and examples, refer to the sections below.

Scaleout CLI (open source)
--------------------------

The ``scaleout`` commands allow users to log in to Scaleout Edge and interact with the platform.

Authentication and contexts
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The standard way to use the CLI is to log in once with ``scaleout login``. This creates a saved **context** that stores your instance URL and credentials. All subsequent commands automatically use the active context — no need to pass ``--host`` or ``--token`` every time.

.. code-block:: bash

    # 1. Log in — opens a browser window for authentication
    scaleout login <url-to-instance>

    # 2. All commands now work without extra flags
    scaleout session list
    scaleout model list

The ``--host`` (``-H``) and ``--token`` flags are only needed when you want to **override** the active context, for example to target a different instance in a script or CI environment.

Commands:

**login**

- **scaleout login** - Log in to the Scaleout CLI by providing the URL to your Scaleout Edge instance. A browser window will open for authentication. Example:

.. code-block:: bash

     scaleout login <url-to-instance>

**context**

The ``scaleout context`` commands allow users to list and switch between saved CLI contexts (i.e. authenticated instances).

- **scaleout context** - List all saved contexts, or switch to a specific one by name or index. Example:

.. code-block:: bash

    scaleout context
    scaleout context production
    scaleout context 2

**remove**

- **scaleout remove** - Remove a saved CLI context by name or index. Example:

.. code-block:: bash

    scaleout remove production
    scaleout remove 2 --yes

**run**

The ``scaleout run`` commands allow users to run entry points and prepare the environment for Scaleout Edge projects.

- **scaleout run build** - Run the build entry point for a Scaleout Edge project, defined in ``scaleout.yaml``. Example:

.. code-block:: bash

    scaleout run build --path /path/to/project/client

- **scaleout run install** - Install the package environment from a ``scaleout.yaml`` into the active Python environment. Example:

.. code-block:: bash

    scaleout run install --path /path/to/project/client

**package**

The ``scaleout package`` commands allow users to create and manage compute packages.

- **scaleout package create** - Create a new package tarball for a Scaleout Edge project. Example:

.. code-block:: bash

    scaleout package create --path /path/to/project/client

- **scaleout package set-active** - Upload a package and set it as the active compute package. Example:

.. code-block:: bash

    scaleout package set-active --file package.tgz --name my-package

- **scaleout package list** - List all packages in your Scaleout Edge project. Example:

.. code-block:: bash

    scaleout package list

- **scaleout package get** - Retrieve a specific package by ID. Example:

.. code-block:: bash

    scaleout package get --id <package-id>

**model**

The ``scaleout model`` commands allow users to manage models in your Scaleout Edge project.

- **scaleout model set-active** - Upload a model file and set it as the active (seed) model. Example:

.. code-block:: bash

    scaleout model set-active --file seed.npz

- **scaleout model list** - List all models in your Scaleout Edge project. Example:

.. code-block:: bash

    scaleout model list

- **scaleout model get** - Retrieve a specific model by ID. Example:

.. code-block:: bash

    scaleout model get --id <model-id>

**client**

The ``scaleout client`` commands allow users to enroll, start and inspect Scaleout Edge clients.

Client authentication is based on **enrollment tokens** — short-lived tokens that an admin generates and hands to the edge operator. When a client uses an enrollment token it is registered on the server and receives a long-lived refresh token that is used for all subsequent connections. This means enrollment tokens can be safely distributed without giving permanent access; revoking a client later is independent of the token.

Enrollment tokens can be created in two ways:

- **From the UI** — navigate to the **Settings** tab (or the **Clients** tab) in the Scaleout Edge Dashboard. The page also shows the exact CLI command to paste on the edge machine.
- **From the CLI** — use ``scaleout client create-enrollment-token`` (described below). Like other CLI commands, it resolves host and credential from ``-H``/``-t`` if given, otherwise from an active CLI context (``scaleout login``).

.. rubric:: Recommended workflow: enroll, then start

Enroll the client once to register it with the server and receive credentials, then start it separately using the resulting ``client_id`` (or a saved config file). Enrollment and starting are decoupled, so the client gets a persistent identity that can be restarted at any time without needing the enrollment token again.

.. code-block:: bash

    # Admin: create a token
    TOKEN=$(scaleout client create-enrollment-token --name fleet-batch-1 --expires-in 7d)

    # Edge operator: enroll once and save a config file
    scaleout client enroll \
        --enrollment-token $TOKEN \
        --api-url <url-to-instance> \
        --name pi-unit-42 \
        --client-id rpi-serial-abc123 \
        --config-dir ./configs
    # → writes ./configs/rpi-serial-abc123.yaml

    # Edge operator: start anytime using the config
    scaleout client start --init ./configs/rpi-serial-abc123.yaml

.. rubric:: Alternative: inline enrollment

For ephemeral or CI nodes that don't need a persistent identity, an enrollment token can be passed straight to ``start`` instead of running ``enroll`` separately — the client self-enrolls on first contact and begins training immediately:

.. code-block:: bash

    scaleout client start --api-url <url-to-instance> --token $TOKEN

.. warning::

   This does not create a persistent identity: without ``--client-id``, every invocation self-enrolls as a new client, so there's nothing to resume on restart, and restarting requires a valid token again. Prefer enroll-then-start above for anything that needs to survive restarts.


.. rubric:: Command reference

- **scaleout client create-enrollment-token** — Create a new enrollment token. Resolves host and credential the same way as every other command: from ``-H``/``-t`` if given, otherwise from an active CLI context (``scaleout login``). Prints the token to stdout only, making it safe to capture with ``$(...)``. Status messages are written to stderr.

  Options:

  - ``-n`` / ``--name`` (required) — Human-readable label for the token (also used as a label for enrolled clients).
  - ``--expires-in`` — Lifetime of the token. Accepts ``24h``, ``7d``, ``90m``, etc. Default: ``24h``.
  - ``-p`` / ``--protocol``, ``-H`` / ``--host``, ``-P`` / ``--port``, ``-t`` / ``--token`` — Explicit connection details, overriding the active CLI context.
  - ``--no-verify-tls`` — Do not verify the server TLS certificate (connection is still encrypted).

  Example:

  .. code-block:: bash

      TOKEN=$(scaleout client create-enrollment-token --name batch-1 --expires-in 7d)

- **scaleout client enroll** — Register a new edge client using an enrollment token. Stores the resulting credentials locally and optionally writes a YAML config file.

  Options:

  - ``--enrollment-token`` (required) — The enrollment JWT issued by an admin.
  - ``-u`` / ``--api-url`` (required) — API server URL.
  - ``-n`` / ``--name`` — Optional human-readable label stored on the server.
  - ``--client-id`` — Stable identifier for the client (e.g. device serial number), used as the primary handle for the client: it names the generated config file (``<client_id>.yaml``) and is used for long-term traceability and idempotent re-enrollment. If omitted, the server assigns a UUID.
  - ``-p`` / ``--config-dir`` — Directory to write a YAML config file (named ``<client_id>.yaml``). Defaults to the current directory; ignored if ``--no-config`` is set.
  - ``--no-config`` — Skip writing the YAML config file (the local token cache is still updated).

  ``enroll`` always prints only the ``client_id`` to stdout (status messages go to stderr), so it can be captured with ``$(...)`` regardless of whether a config file was written — e.g. ``CLIENT_ID=$(scaleout client enroll ... --config-dir ./configs)`` followed by ``scaleout client start --init ./configs/$CLIENT_ID.yaml``.

  Example:

  .. code-block:: bash

      scaleout client enroll \
          --enrollment-token $TOKEN \
          --api-url <url-to-instance> \
          --name pi-unit-42 \
          --client-id rpi-serial-abc123 \
          --config-dir ./configs

- **scaleout client start** — Start a Scaleout Edge client, using a pre-enrolled ``--init`` config file or ``--client-id`` (recommended), or by self-enrolling inline with an enrollment token (see "Alternative: inline enrollment" above).

  Key options:

  - ``-u`` / ``--api-url`` — API server URL.
  - ``-i``/ ``--client-id``- Stable client id used to identify this client also used to look for refresh tokens on the local machine. When used with an enrollment token, requests this ID from the server.
  - ``--init`` — Path to a YAML config file produced by ``scaleout client enroll --config-dir``. Contains information such as api-url and client-id
  - ``--token`` — Authentication token. Normally a client refresh token (from ``enroll`` or a prior run); can also be an enrollment token, in which case the client self-enrolls on the fly (see the alternative workflow above). API keys are **not** accepted here — they are user-scoped credentials for SDK/CI use (see :doc:`apiclient`) and cannot be revoked per-client; use an enrollment token for edge clients.
  - ``-n`` / ``--name`` — Human-readable label for this client instance.
  - ``--remote-package`` — Use a remote compute package downloaded from the server rather than local-package in client.

  Examples:

  .. code-block:: bash

      # Recommended: from a pre-enrolled config file
      scaleout client start --init ./configs/rpi-serial-abc123.yaml

      # Alternative: inline enrollment (ephemeral/CI nodes)
      scaleout client start --api-url <url-to-instance> --token $TOKEN --name pi-unit-42

- **scaleout client list** - List all active Scaleout Edge clients in the network. Example:

.. code-block:: bash

    scaleout client list

- **scaleout client get** - Retrieve a specific client by ID. Example:

.. code-block:: bash

    scaleout client get --id <client-id>

**session**

The ``scaleout session`` commands allow users to start, stop and list sessions in Scaleout Edge.

- **scaleout session start** - Start a new federated learning session. Example:

.. code-block:: bash

    scaleout session start --name my-session --rounds 10 --aggregator fedavg

- **scaleout session stop** - Stop the currently running session. Example:

.. code-block:: bash

    scaleout session stop

- **scaleout session list** - List all sessions in your Scaleout Edge project. Example:

.. code-block:: bash

    scaleout session list

- **scaleout session get** - Retrieve a specific session by ID. Example:

.. code-block:: bash

    scaleout session get --id <session-id>

**round**

The ``scaleout round`` commands allow users to retrieve and list round information.

- **scaleout round list** - List all rounds in your Scaleout Edge project. Example:

.. code-block:: bash

    scaleout round list

- **scaleout round get** - Retrieve information for a specific round. Example:

.. code-block:: bash

    scaleout round get --id <round-id>

**combiner**

The ``scaleout combiner`` commands allow users to list and inspect combiners in the Scaleout Edge network.

- **scaleout combiner list** - List all combiners in your Scaleout Edge project. Example:

.. code-block:: bash

    scaleout combiner list

- **scaleout combiner get** - Retrieve a specific combiner by ID. Example:

.. code-block:: bash

    scaleout combiner get --id <combiner-id>

- **scaleout combiner set-public-hostname** - Set the public hostname for a combiner. Example:

.. code-block:: bash

    scaleout combiner set-public-hostname --id <combiner-id> <hostname>

**validation**

The ``scaleout validation`` commands allow users to retrieve and list validation results.

- **scaleout validation start** - Start a validation run for a given session and model. Example:

.. code-block:: bash

    scaleout validation start --session-id <session-id> --model-id <model-id>

- **scaleout validation list** - List all validation results for a project. Example:

.. code-block:: bash

    scaleout validation list

- **scaleout validation get** - Retrieve a specific validation result by ID. Example:

.. code-block:: bash

    scaleout validation get --id <validation-id>

**status**

The ``scaleout status`` commands allow users to check the status of the Scaleout Edge network.

- **scaleout status list** - List status entries for your Scaleout Edge project. Example:

.. code-block:: bash

    scaleout status list

- **scaleout status get** - Get a specific status entry by ID. Example:

.. code-block:: bash

    scaleout status get --id <status-id>

Scaleout core (Enterprise)
---------------------------

The ``scaleoutcore`` commands allow users to start and manage Scaleout Edge core services, such as the combiner, controller, API server, and Server Function hooks (requires Enterprise license or specific research related agreement).

Commands:

**combiner**

The ``scaleoutcore combiner`` commands allow users to start and manage (scale) the Scaleout Edge combiner service.

- **scaleoutcore combiner start** - Start the Scaleout Edge combiner service. Example:

.. code-block:: bash

     scaleoutcore combiner start

**controller**

The ``scaleoutcore controller`` commands allow users to start and manage the Scaleout Edge controller service.

- **scaleoutcore controller start** - Start the Scaleout Edge controller service. Example:

.. code-block:: bash

     scaleoutcore controller start

**api-server**

The ``scaleoutcore api-server`` commands allow users to start (scale) and manage the Scaleout Edge API server service.

- **scaleoutcore api-server start** - Start the Scaleout Edge API server service. Example:

.. code-block:: bash

     scaleoutcore api-server start

**hooks**

The ``scaleoutcore hooks`` commands allow users to start and manage (scale) the Scaleout Edge Server Function hooks service.

- **scaleoutcore hooks start** - Start the Scaleout Edge Server Function hooks service. Example:

.. code-block:: bash

     scaleoutcore hooks start

**migration**

The ``scaleoutcore migration`` commands allow users to manage the PostgreSQL database schema via Alembic migrations.

- **scaleoutcore migration generate** - Generate a new migration script based on model changes. Example:

.. code-block:: bash

     scaleoutcore migration generate --message "add new column"

- **scaleoutcore migration upgrade** - Apply pending migrations to bring the database up to date. Example:

.. code-block:: bash

     scaleoutcore migration upgrade

- **scaleoutcore migration verify** - Verify that the database schema matches the expected revision. Example:

.. code-block:: bash

     scaleoutcore migration verify

- **scaleoutcore migration current** - Print the current database schema revision. Example:

.. code-block:: bash

     scaleoutcore migration current

- **scaleoutcore migration check** - Check whether the database schema is up to date (exits non-zero if migrations are pending). Example:

.. code-block:: bash

     scaleoutcore migration check

- **scaleoutcore migration stamp** - Mark the database at a specific revision without running migrations. Example:

.. code-block:: bash

     scaleoutcore migration stamp --revision head